Did you know that your cell volume data could look like this?
Old: Traditional volume rendering
|
New: Path trace rendering
|
Did you know that same data could also look like this?
Old: Traditional volume rendering
|
Old: Traditional volume rendering
|
No, these are not illustrated interpretations of the data or some enhancements coming from Hollywood special effects, but they do apply modern GPU intensive rendering technologies learned from billions of dollars of rendering research to your volume cell data to make the spatial relationships more easy to interpret.
Try the viewer here in your browserNOTE to Dan: I think we need HTTPS for http://dev-aics-dtp-001/webclient_ec2/ This version of the photorealistic renderer runs in the cloud. The renderer requires a powerful computer with a high-end consumer level graphics card. This version can run in your web browser on just about any computer or even some mobile devices, because it simply provides a window that allows you to control a powerful computer we are running for you up in "the cloud". |
See the Methods section for a more detailed description of the lighting technology and the streaming rendering approaches.
Visualization Advances for Enhancing Spatial Relationships… Even Online
Simply looking at cells in 3D can be technically challenging, and visually analyzing 3D volumetric datasets can be mentally taxing. Current advances in computing power and techniques can help cell biologists access, analyze, and share this type of data more easily and more effectively.
Most volumetric visualization tools that cell biologists use have evolved from 2D image analysis predecessors. They provide powerful analysis capabilities, especially for looking at one slice of a 3D volume or a projection, but they have limited 3D visualization functionality. Transforming a stack of 2D slices into a 3D volume and then interactively rotating it makes complex spatial relationships easier to see and understand.
The most common 3D volume rendering methods make the stacked image slices transparent so the observer can see through the entire volume. However, this can compromise the appearance and detail of the signal (Fig. 1a). Users can adjust the transparency to make denser voxels (the 3D analog to pixels) stand out, but this rendering approach is far removed from how we view and intuitively interact with everyday objects, lacking, for example, any concept of a directional light source, adjustable highlights, or shadows that can help us intuit complex topologies and distance relationships. The volumes can require longer interactions and more experience to interpret, and it can be impossible to correctly interpret depth relationships if the user cannot interactively rotate the volume, for example if it is displayed on a printed page.
Displaying 3D volume data in this manner is computationally expensive and has been impractical for widespread interactive use until recently. The growing ubiquity of cloud computing, fast internet connections, and sophisticated personal computing devices with GPU acceleration is allowing us to share and explore large 3D datasets in new ways. Volumetric viewers can now run directly in a Web browser using the WebGL graphics library, which is supported on all modern devices [Dan, I suggest we point to our own viewer here, but also reference Bisque and maybe a couple of other webGL viewers, e.g. the offsprings of ParaView/VTK etc. if that is still alive], or by streaming more computationally expensive and sophisticated renderings directly to the browser window from more powerful computers running in the cloud.
New approaches to rendering volumes are also being adapted and developed for use in cell biology to help us literally see the cells in a more familiar and thus intuitive manner. For example, by adopting approaches such as “cinematic rendering” techniques to cellular volumetric data, one can see fine details become more visible and spatial relationships more interpretable, even in a 2D static printed image (Fig. 1b). This makes exploring cellular data more analogous to dissecting a cadaver rather than looking only at x-ray projections when learning human anatomy. These types of approaches should become an important addition to the arsenal of visual analysis tools needed to study and see a cell from every possible perspective and in every possible way. This cinematic approach can also be adjusted to emulate the method shown in (a) when transparency is useful.
Most volumetric visualization tools that cell biologists use have evolved from 2D image analysis predecessors. They provide powerful analysis capabilities, especially for looking at one slice of a 3D volume or a projection, but they have limited 3D visualization functionality. Transforming a stack of 2D slices into a 3D volume and then interactively rotating it makes complex spatial relationships easier to see and understand.
The most common 3D volume rendering methods make the stacked image slices transparent so the observer can see through the entire volume. However, this can compromise the appearance and detail of the signal (Fig. 1a). Users can adjust the transparency to make denser voxels (the 3D analog to pixels) stand out, but this rendering approach is far removed from how we view and intuitively interact with everyday objects, lacking, for example, any concept of a directional light source, adjustable highlights, or shadows that can help us intuit complex topologies and distance relationships. The volumes can require longer interactions and more experience to interpret, and it can be impossible to correctly interpret depth relationships if the user cannot interactively rotate the volume, for example if it is displayed on a printed page.
Displaying 3D volume data in this manner is computationally expensive and has been impractical for widespread interactive use until recently. The growing ubiquity of cloud computing, fast internet connections, and sophisticated personal computing devices with GPU acceleration is allowing us to share and explore large 3D datasets in new ways. Volumetric viewers can now run directly in a Web browser using the WebGL graphics library, which is supported on all modern devices [Dan, I suggest we point to our own viewer here, but also reference Bisque and maybe a couple of other webGL viewers, e.g. the offsprings of ParaView/VTK etc. if that is still alive], or by streaming more computationally expensive and sophisticated renderings directly to the browser window from more powerful computers running in the cloud.
New approaches to rendering volumes are also being adapted and developed for use in cell biology to help us literally see the cells in a more familiar and thus intuitive manner. For example, by adopting approaches such as “cinematic rendering” techniques to cellular volumetric data, one can see fine details become more visible and spatial relationships more interpretable, even in a 2D static printed image (Fig. 1b). This makes exploring cellular data more analogous to dissecting a cadaver rather than looking only at x-ray projections when learning human anatomy. These types of approaches should become an important addition to the arsenal of visual analysis tools needed to study and see a cell from every possible perspective and in every possible way. This cinematic approach can also be adjusted to emulate the method shown in (a) when transparency is useful.
Figure 1: Live hiPS cells in a monolayer are labeled for the cell boundary (magenta), DNA (cyan), and microtubules (white). Cells are imaged in 3D using spinning disk confocal microscopy and the image is displayed with: a) the most common type of volume rendering method as in this example from www.allencell.org/3d-cell-viewer.html; and b) newly adopted “cinematic lighting” techniques to volume rendering in development at the Allen Institute for Cell Science. NOTE replace with 20-20 slider!
Results: Example renderings of each cell line available in the Allen Cell Collection
A section showing one static example of every cell-line we have available in the 3DCV
Try to find a horizontal image gallery carousel to keep the height reasonable for ~20 images that it will contain.
Try to find a horizontal image gallery carousel to keep the height reasonable for ~20 images that it will contain.
Technical description of the volume rendering approaches used
The preferred method for displaying 3D volume data is ray-marched path tracing. We make use of the fact that tracing light from light source to camera is mathematically equivalent to tracing light from camera back to light source. (not sure that last sentence is understandable or in the right place yet)
The simplest ray marching volume renderer can be described as follows. We calculate camera rays through each pixel on the screen and trace them through the volume. When a ray enters the volume, it is “marched” by some incremental amount through the volume until it exits. The rays accumulate color at each marching step. Once the ray has exited the volume, we can finalize the color for that pixel.
The simplest ray marching volume renderer can be described as follows. We calculate camera rays through each pixel on the screen and trace them through the volume. When a ray enters the volume, it is “marched” by some incremental amount through the volume until it exits. The rays accumulate color at each marching step. Once the ray has exited the volume, we can finalize the color for that pixel.
Improvements can be made by carefully tuning the step size as the ray moves through the volume. We can use a theoretical approximation of light scattering through an isotropic medium to compute the step size and determine where scattering interactions occur. We allow rays that interact with the medium to bounce (“scatter”). If at every step where we have a scattering event, we have a change of direction of the light ray, then we can trace that ray’s path all the way from camera to light source. In this way we can use artificial light sources to illuminate the volume data set.
Some scattered paths will scatter again within the volume before reaching a light source. Light travelling along that path will experience even more attenuation. As a first order approximation, we can eliminate those paths from contributing to the final image. At a cost of computational performance, we could include multiple scattering for higher accuracy in our light scattering simulation. As of this writing, multiple scattering is not sufficiently optimized to allow interactivity in the viewer application. The approaches used in the images presented here are single scattering only.
Some scattered paths will scatter again within the volume before reaching a light source. Light travelling along that path will experience even more attenuation. As a first order approximation, we can eliminate those paths from contributing to the final image. At a cost of computational performance, we could include multiple scattering for higher accuracy in our light scattering simulation. As of this writing, multiple scattering is not sufficiently optimized to allow interactivity in the viewer application. The approaches used in the images presented here are single scattering only.
The final image is composed by averaging the contributions of many randomized ray paths per pixel.
There are many tuneable parameters that help determine the final appearance of the image. In particular, we can modify the ray step sizes so that scattering events are more or less likely to happen, we can change positions and intensities and colors of light sources, and we can change the scattering density characteristics of the volume data itself. We can do transformations on the volume data intensities themselves. We also provide a simple physically-based material model that allows for shiny reflections at scattering locations within the volume. (This material model treats the scattering locations just like any hard surface for shading and reflection calculations.) All of these knobs can be turned to bring out various features when examining the volume data in an interactive session.
There are many tuneable parameters that help determine the final appearance of the image. In particular, we can modify the ray step sizes so that scattering events are more or less likely to happen, we can change positions and intensities and colors of light sources, and we can change the scattering density characteristics of the volume data itself. We can do transformations on the volume data intensities themselves. We also provide a simple physically-based material model that allows for shiny reflections at scattering locations within the volume. (This material model treats the scattering locations just like any hard surface for shading and reflection calculations.) All of these knobs can be turned to bring out various features when examining the volume data in an interactive session.