3D Volumetric Video for Microscopy Applications

I’m interested in using a Looking Glass display for a cobbled together stereo microscope.

As I understand it, just using two webcams wont cut it for displaying proper video on the Looking Glass (unless one holds ones head in fixed position). Alternatively one can get a few dozen cameras strapped together in parallel but that’s tricky for microscopy applications.

Specifically:

  1. Can I use two Lytro Illium light field cameras to stream stereoscopic video to the Looking Glass?
  2. Is there software out there which can stream realtime video to the Looking Glass or does all video have to be pre-rendered.

Money isn’t a concern but viability is. Alternatively are there different ways to approach this problem to get 3D video streamed onto the Looking Glass for soldering/surgery applications?

Hmm, this is a tricky problem and @alex is the most knowledgeable about this. We may be able to figure out a solution, however, using some of our new view interpolation logic. Would you be able to output two RGB streams with depth information? We could maybe do some work on that, though certainly we don’t have anything in the product stage.

1 Like

Hey pmg,

Another Looking Glass user, Dany Qiumseh, made a really nice DIY microscope for the Looking Glass that captures lightfield imagery by moving one of those cheap USB dissection microscopes in an arc over the subject and capturing a set of images from different positions.

This technique makes really nice still images, but the trick for you is getting the image to refresh at video rates – doing that requires multiple cameras or a very special camera capable of filming microscopes.

My hunch is that the lytro cameras won’t work without modification for capturing microscopic subjects, and I’m also skeptical of the stereo camera -> depth approach – If you have multiple 2D cameras, you can calculate depth info, but getting consistent depth scale from one frame to another is challenging, and doing it in realtime is also quite hard. Almost all stereo -> depth conversion will have a number of artifacts. Reflective elements like shiny solder can also throw artifacts into the depth conversion.

I was just googling around for a microscopic Time of Flight sensor, which is probably your best bet – Texas Instruments put together a nice intro on their ToF sensors here. From a video point of view, they work like a kinect or realsense – they give you a depth map of the scene in front of the sensor. Unlike a realsense or kinect, they don’t use a structured light projector, and can probably get much closer to the subject. The sensors work with a variety of lenses, and I would bet that somebody has put a microscope lens onto one by now (although a quick google didn’t turn up anything, but maybe there’s a special name for this type of microscope that I don’t know).

I think that’s your easiest option – there’s probably an off the shelf sensor that does what you want, especially for large-ish subjects like a circuit board. You can use the camera feed to reconstruct a point cloud or mesh in a 3D environment like unity or three.js and then get that 3D scene into a Looking Glass. Bada bing bada boom.

The other option, which is super DIY-heavy, is to make an array of a large number of 2D cameras, aligning the output of the 2D cameras to make a quilt image, and then passing that to the post-process shader to get an image into the Looking Glass. It’s not easy to get more than a few live video feeds in to a single computer – you can only plug in so many USB cameras before you overwhelm the bandwidth of a USB port. I designed a system to get around that by having a bunch of tiny computers, each of which drives a single camera, aligns the image and sends just the cropped, aligned portion over the network to a central computer. This is quite a process, and the end result looks like this:

The images from this camera are really nice, though, and they don’t have any artifacts from reflectivity. Here’s a writeup I did of some lightfield filming experiments with this rig. I haven’t tried putting a macro lens on the cameras, although I know they do exist.

I don’t really recommend this for you – it’s a ton of work, and even though the end results are wonderful, you’ll spend a lot of your time making cameras rather than soldering. If you can find an off-the-shelf microscopic depth sensor, I think that’s the fastest train to solder town.

Let me know how it goes!
–alex

2 Likes

Hi, I’m currently doing work with the Looking Glass and Microscopy, focusing mainly on TEM tomography stuff. Here’s my quick take on this:

I’m an amateur programmer (Especially with the API), but I think the hope for streaming realtime video is a bit too difficult at the moment since it takes a bit of time to actually construct and shade the quilt. Instead, I think having some sort of program ready so that at the press of a button, a quilt-creation operation will occur then spit to the Looking Glass, is a bit more reasonable.

As far as collecting the images, I think there are several methods (some of which alex covered!) worth thinking about

  1. Dany Qiumesh’s method of the arcing camera, with an issue being toe-in
  2. A camera set on a track with a motor for rotation, with an issue would be adjusting focus as this occurs
  3. A set-up of numerous cameras on a bar, with an issue being how to best gather that data
  4. A single fixed camera with a stage capable of moving xyz and rotationally (kinda like some SEMs use), with issues being actually making that thing and coding focus adjustment
  5. A simple motorized vertical panel that the object can adhere to and rotate on, with an issue being toe-in
  6. System of mirrors to redirect light from a certain angle back into the camera, with the issue being figuring out how that would work.

Just some quick brainstorming! It would be pretty DIY, though

1 Like

PMG, This isn’t exactly what you are looking for but might be of interest. I developed a VR application for viewing confocal microscope images, and morphed it onto the LG when it came out. If you look at the ConfocalVR page on the www.immsci.com site you can get an idea. Drop me a note if you want to know more. tskillman at immsci dot com

How close is this to what Lytro is doing?

IMO, the gold standard would be to use a lightfield camera for a lightfield display. I know that sounds somewhat simplistic, but I think especially microscopy of biological subjects would have translucent structures that wouldn’t handle ToF or other depth-reconstruction techniques very well.

To what extent can large GPUs be thrown at the problem? IMO, it seems quite amenable.

Hm, honestly not sure and probably didn’t think it over as much as I should have before replying. To consider realtime rendering, we would need to consider a couple things:

  1. How do we get the images?
  2. What’s the fastest we can run a series of images into a quilt into the display?

For 1, I’m not quite sure what the limitations would be as far as image collection goes. Alex brought up the data input situation, and while I think his rig is a good idea, how close could we reasonably place the cameras/lens so that they don’t block eachother but still focus at the correct point efficiently enough to make a good microscope image? If we decided a solitary camera with a motorized stage, how quick and precise could we get the motors to work? Not saying that wouldn’t be possible, but I certainly don’t know off the top of my head.

For 2, I imagine this is pretty testable. I actually don’t think I’ve been using a GPU in my quilt creation since I create them via MatLab and haven’t been using the Parallel Computing Toolbox (so my time expectancy could be off), but as I stated before, my programming abilities are pretty questionable in efficiency. Maybe someone has tested this on the forums already? Thinking about it now, I’d be surprised if something like a Titan couldn’t handle it with some sensible code to back it up.

Sooooo… potentially possible, but maybe something folks with more experience than me have to play around with? I think the weak link here is an effective solution to 1, but I’d be glad to be proven wrong!

Machine vision cameras seem to be using USB 3 and 10 Gigabit Ethernet, these days.

I still think the Lytro/microlens array solution is best. You get the effect of a camera array, without the bulk. AFAIK, it’s equivalent to what the Looking Glass display is doing, in reverse.

BTW, speaking of Lytro, this is news to me:

It’s a real shame. I think lightfield cameras and displays need each other.

Thank you all for the constructive feedback, especially @alex

Some casual research suggests that “Lightfield Microscope piped to Looking Glass Display” is beyond the scope of what I can accomplish within a few weekends and a few thousand dollars with tools presently available. Specifically some napkin math (optimistically) suggests I can get 1 Lytro Illum frame per minute on the Looking Glass display with present inefficiencies in the rendering pipeline with consumer graphics cards. As I understand it, the Lytro Immerge 1.0 came with rolling server stuffed with ~48 graphics cards. And even that didn’t allow for realtime display. The unreleased Immerge 2.0 was even bigger.

In contrast, throwing two usb microscopes in a VR display is trivial. So I’ll do that on a lark and see how it goes.

Not to contradict myself, but intuitively, I don’t think this is even a GPU constrained problem. I think one could get ‘good enough’ performance with an FPGA and low resolution plenoptic camera on the order of the original 1.2 megapixel Lytro (the square tube thing). GPU’s make some tradeoffs in order to output one or two hdmi’s signals that an Xilinx or Altera would not need to work around. I wont be pursuing this further this year given how quickly depth mapping is advancing with smartphones but if I were I would follow up with builds made by labs connecting plenoptic cameras to fpgas and pipe that to the Looking Glass.

It does look like a couple of commercial solutions are popping up in Germany and Taiwan. Specifically Raytrix: Products – Raytrix and HVI Global: 3D AOI Wire bonding system- High View Innovation

That was not just one lightfield camera, but a large enough array to create an entire 6-DoF surround-video!