With the current structure hardware(which is the guts of an early kinect), we have a hardware limit to what we can get out of the depth camera – the depth camera hardware is limited to 640x480 pixels @ 30fps @ ~11 bit resolution.
When we made the holoflix app, we said, “let’s set everything to the resolution we’re getting out of the depth camera – that’s simple and everything will line up well.” For the most part, the app works as advertised, but, yeah, once you’re done getting wowed by the depth video, the next question is, “can the resolution get better?”
There’s one thing we could do to make a big improvement in the resolution:
You actually don’t need as much depth information as you do color information. Our brains are really good at seeing tiny changes or flaws in a 2D texture, but we’re nowhere near as good at seeing tiny changes in a 3D contour.
Take this 3D model as an example:
This was made by the structure sensor in the amazing itSeez3D scanning app. If you have a structure sensor, this app is free and worth playing with. The app creates a model with fairly coarse 3D contour information and fairly high-res 2D texture information. The mesh has 25,000 vertices, which look like this;
and a ~2700x2500 pixel (6.7 million total pixels) texture that wraps around that mesh, which looks (disturbingly) like this:
The key idea there is that there is wayyy more information in the 2D surface texture than in the 3D mesh. To put this in holoflix terms, it’s as if the depth map, which defines the overall shape, was a 109x109 pixel image:
and the color map, which has all of the surface features, was a whopping 1800x1800 pixel image that gets deformed around that shape:
That crazy ratio, of ~270 times more pixels in the texture information than in the contour information, is really common in models. Almost nobody in 3D modeling does the vertex coloring approach that holoflix uses, where there is a vertex for every pixel in the texture. It’s much more common to have a coarse 3D mesh and a high-res texture that goes on top.
That’s the easiest and greatest resolution improvement that we could make in holoflix – we have a camera capable of shooting 1920x1080 pixel movies, and we’re scaling that alll the way down to 640x480 pixels to match the depth camera’s resolution. The tricky part is getting perfect alignment of the high-res image to the mesh, and I’d need to experiment with that a bit before I could tell you how it’ll turn out.
In other areas, we’re running against hardware limitations of the depth camera. First, we’re stuck at the framerate set by the depth camera. The options that structure offers are 640x480 @ 30fps or 320x240 @ 60fps. We originally went for the higher resolution, lower framerate option, and I think that is probably the place to stay. For video, 320x240 is really low, even for just the contour information.
Next, we can talk about the depth resolution. The structure camera gives us a maximum of 11 bits of information, and we cut that down to 8 bits so we could store the depth information in a convenient grayscale format. We could cram those extra three bits in there, at the expense of human viewability. I think it’s a pretty small advantage, and I’m not super excited about it, but mentioning it for completeness.
We are interested in working with better depth-sensing cameras, as they become available. The most promising one is currently the iPhone 7 Plus with stereo cams running the forthcoming iOS 11, due for release sometime this month. Apple says it’ll open up Apple’s depth API for developers with iOS 11. They’re not saying yet if their API will support video or just stills, or any hard numbers on the resolution of the color/depth image – I’m waiting to see it once it comes out.
I’m curious to hear – if you had the capability to record higher-quality holoflix, what would you want to do with it? What kind of quality do you need?