Head-up Cardboard - A Google Cardboard inspired 3d head-up display
DIY stereoscopic holography

OK Cardboard

By design, in VR you are not connected with the real world - you cannot see the real world and the VR world at the same time. I made Solar System for Google Cardboard to explore the possibilities applying AR (or some form of Mixed Reality) in Google Cardboard, mixing real world content with simulated content. This was partly successful, but with this (web) app, the camera video resolution is quite low, and the image of the real world has no perceived variation in depth (only one camera). This is not a bad thing if you want your primary focus to be the 3d simulated world, but if you want a good view of the real world then you would need better video resolution and two cameras. We already see the real world quite well without the use of camera and screen, so I thought about projecting 3d content directly into our view of the real world. Similar to what Google Glass does, except I wanted to use a phone and apply stereoscopic 3d.
The result is the Head-up Cardboard.

The idea is simple: just like Google Cardboard, except the stereoscopic images are reflected via semi-reflective / semi-transparent glass into your field of vision. You see the real world, mixed with holographic-like 3d content.

Work in progress

It's still work in progress and I am currently working on a new prototype. I will post the improvements as it progresses.

Tim Samshuijzen
13th December 2014
Making the Head-up Cardboard prototype 1

The purpose of this prototype is to simply confirm that a phone can be used for a stereoscopic head-up display, a proof of concept, as simple as possible.

Shown below are instructions for making a prototype 1:
  1. Get these items:
    • A piece of cardboard.
    • Two lenses.
          The lenses I use are from a ColorCross VR headset.
    • Some semi-reflective / semi-transparent material.
      Preferably very thin material, otherwise it creates a double ghost-image.
          I did not have such material. I used a CD-cover instead (remember them?).
          It's not perfect, but good enough for proof-of-concept.

    • Cut up the cardboard and make a hollow box (using Sellotape).
    • Cut holes in the cardboard and position the lenses. The distance between the centers of the holes should be something like 6 cm.
    • The height of the box depends on the focal length of the lenses. For my lenses it is 4 cm. To determine the box height for your lenses, simply position the lens above some text and measure the distance between the lenses and the text when the text appears most clear.

    • Cut up the semi-reflective / semi-transparent material into pieces as shown in the photo.
    • I used a CD-cover, and cut it up using box cutters (be careful!!).

    • Assemble the pieces as shown in the photo (using Sellotape). Apply enough tape to ensure a rigid structure.
    • The angle of the 'window' should be 45 degrees.
    • Note: the photo below shows the setup placed upside down. While using, it is meant to be held with the box above your eyes and the reflective glass/plastic in front of your eyes, in such a way that the edge of the glass/plastic touches your nose.
    • Note: the photo shows the lenses inserted into the other (wrong) side. This was while experimenting. The best fit for my setup was to have it inserted on the "window" side.

14th December 2014
Testing prototype 1

To test the proof-of-concept prototype I made some stereoscopic scenes and images. You can use this to test your own Head-up Cardboard. Note that the scenes and images are flipped, reflected through the horizontal axis. This is necessary to correct for the resulting reflected image.

Here is a web app showing a simple 3d scene (some orbiting planets) to test your Head-up Cardboard with your phone:

Looking through the Head-up Cardboard, with the app running on your phone, you should see something like this:

If your phone does not support WebGL, you can use this test image to test your prototype (click and maximize):

Looking through the Head-up Cardboard, with the test-image full screen on your phone's display, you should see something like this:

Through the Head-up Cardboard you will see the time and text to appear at slightly different distances, giving a 3d impression.

14th December 2014
Evaluation of prototype 1 - the proof of concept

My findings so far:
  • It sort of works! Using Three.js I made some orbiting objects, placed it in the center 20% of the screen. When aligned and focussed correctly, I got the stereoscopic 3d experience, while still being able to see the real world. I did notice that I would have to change focus and switch attention between the two worlds. The worlds did not merge into one world.
  • The lens choice is not quite ideal. I am looking into using different types of lenses / curved surfaces.
  • Compared to Google Cardboard, the FOV of the Head-up Cardboard is much narrower. This makes it difficult to find objects in 3d space when looking around. Fixed positioned content (independent of orientation or movement of the phone) seems to work well. I am looking into different ways of increasing the FOV, possibly a curved reflective surface.
  • The CD-cover that was used (see below) is not ideal. First of all it is not quite reflective enough for a good experience. You need a dark background to see the content. A more suitable material would have a slightly higher degree of reflectivity. Furthermore, the internal reflection creates a second slightly fainter ghost image.
  • A light floor or ground will make the image fade, because light from the bottom reflects back off the phone into sight. A dark casing on the bottom of the device will solve this.
  • The image seen through the device is a flipped version of what is on the phone's screen. So the app needs to flip the intended image. This is one of the disadvantages of this prototype, because it is not compatible with Google Cardboard apps. I am also looking into possibilities of creating a double mirror setup such that the image is not a mirror image, so that the apps for this device can also be viewed with standard Google Cardboard.
  • The orientation of the device is different from Google Cardboard. So the app needs to interpret the phone's movement differently. The ideal orientation would be to have the same orientation as with Google Cardboard.
  • The projected 3d objects give a sensation of depth due to the stereoscopic effect, but misses the differences in focal depth. The objects therefore do not properly mix well into the real world. For a true AR experience we will have to wait for Magic Leap's promising new technology.

17th December 2014
Listing improvements and variations

The main needs for improvement seem to be: a wider FOV, compatible with Google Cardboard, and better image quality.

  • Increasing FOV
    3 possible ways to increase the FOV:
    1. Get our eyes closer to the lens.
      We cannot get our eyes much closer to the lens, because the lens itself needs to be out of our direct line of sight.
    2. Increase the size of the lens.
      It will be a challenge to find the right size and shape of lens.
    3. Eliminate the need for a lens.
      For the eyes to comfortably be able to focus on the phone's screen without the use of lenses, there are two possibilities:
      1. Increase the optical distance between eyes and phone.
        Increasing the optical distance will result in a decrease of FOV.
      2. Use a curved reflective surface.
        This is functionally equivalent to a lens, so this would seem a good choice, although probably a challenge to make.

  • Compatible with Google Cardboard
    The only way I could think of to remain compatible with Google Cardboard is to use two reflectors, put together like a periscope. The bottom mirror would have to be semi-reflective, and the other fully reflective. (Note: in terms of beam-splitter technology, the bottom mirror is called the "combiner".)

  • Better image quality
    Through prototype 1 you see double (ghost) image of the stereoscopic scene, due to the external and internal reflections of the CD-Cover. To eliminate this ghost image we would need to use very thin glass, or a beam splitter cube.
    Furthermore, if we are to introduce a second mirror, then we would have to use a coated mirror, so the light is directly reflected without having to pass through glass.

Variations of the Head-up Cardboard
There are several variations of the Head-up Cardboard, some of them are listed here. Each has their own set of advantages and disadvantages.

Concept 1

Concept 1 is the original prototype.
Concept 2

Concept 2 is constructed like a periscope, using a semi-reflective mirror and a fully-reflective mirror.
Compatible with Google Cardboard.
Concept 3

Concept 3 is a reversed version of concept 2, places the FOV of the real world above the normal line of sight.
Compatible with Google Cardboard.
Concept 4

Concept 4 uses a curved semi-transparent mirror, doubling as a lens, possibly eliminating the need for a lens.

The next step
At this moment it is too early to commit to one particular concept. All alternatives seem to have their own advantages and areas for application, so I want to experiment with all three types. The next step is to get the necessary components the test the concepts and work towards a new prototype.

20th December 2014
Constructing the mirrors

The next step was to create two mirrors: the semi-reflective mirror and the fully-reflective mirror. I am very fortunate to know Tatjana J. van Vark, who is (among many other subjects) an expert on the subject of optics and vacuum coating. She has a vacuum coating machine (actually fully built by herself). Tatjana constructed the two mirrors I needed:
  • A 100% fully reflective vacuum coated mirror.
    A thin glass plate vacuum coated with Aluminium.
  • The combiner: a semi-reflective mirror (a.k.a. beam-splitter).
    A thin glass plate, vacuum-coated with an extreme thin layer of Aluminium for the semi-reflective side, and a thin layer of Magnesium Fluoride for the non-reflective side (anti-reflective coating, destructive thin-film interference).
    Goal: 50% transmission
    Result: 44% transmission (a very good result!)
Thank you so much Tatjana, without your help and inspiration this would not have been possible. This has become an educational journey.

Here are some photos of the process, and of the wonderful vacuum coating machine made by Tatjana J. van Vark:

20th December 2014
The mirrors

The resulting mirrors:

The semi-reflective mirror
Also known as a beam splitter, or the "combiner". A thin glass plate, vacuum-coated with an extreme thin layer of Aluminium for the semi-reflective side, and a thin layer of Magnesium Fluoride for the non-reflective side (anti-reflective coating, destructive thin-film interference). Result: 44% transmission.

The fully reflective vacuum coated mirror.
Thin glass plate vacuum coated with Aluminium.
21st December 2014
Testing the mirrors

The mirrors were tested in a simple proof-of-concept Lego model. I decided to primarily focus on Concept 2 (described earlier). The mirrors work great, they blend the real image with the augmented image very well. It became clear that a lens was needed, there was too much difference in focal distance between the images. Unfortunately I do not have any suitable lenses. The lenses need to be quite wide, possibly a Fresnel lens (sheet magnifier).

Project paused
In the meantime other projects have gotten my attention. Due to limited free time this project is now paused. I hope to pick this project up again sometime in the near future.