Server-based Approach to Web Visualization of Integrated Three-dimensional Brain Imaging Data

Background

The Human Brain Project (HBP),10 a national effort to organize and share data about the human brain, provides many examples of systems for integrating and visualizing 3-D brain imaging data.1 As in other 3-D medical imaging applications, most of these systems are implemented in stand-alone software that must be downloaded and installed on the local user workstation. However, these and other research efforts are increasingly providing some form of Web-based access to their existing stand-alone applications. For example, BrainWeb11 and ICBM View12 provide Web-based slice viewers that allow a user to navigate through three orthogonal slices of a 3-D image volume. Similar slice viewers have been developed for the Visible Human13 and various interactive brain atlases.14 15 16 Several sites also provide some sort of Web-based visualization of 3-D models. Examples include the SPL Anatomy Browser,17 Cortical Flat Mapping,18 and the UCLA Laboratory of NeuroImaging.19 However, most of these applications resort to pregenerated Web pages and noninteractive or quasi-interactive techniques like “canned” 3-D scenes and movies.

Client-server systems have also been developed for 3-D visualization in teleradiology, although most of these systems have been experimental. Examples include client-side VRML renderers6 7 8 and custom client-server applications that are optimized for rendering speed.20 21 22 Although the performance of these systems can be excellent for applications such as surgical planning,22 the specialized hardware and software required currently limit their widespread use.

Some applications we are aware of that have a similar approach to our own are Webcortex,23 a Web front end to a precursor of the FreeSurfer brain surface reconfiguration program24; WebCaret,25 a Web front end to a different surface reconfiguration program called Caret26 and a general purpose Web front end to SGI's OpenInventor; and Cosmo3D applications.27 In all these applications, the server, which is a modified version of the corresponding stand-alone application, renders a 3-D scene, sends a 2-D snapshot of the rendered scene to a standard Web browser client, and manipulates (rotates, moves) the scene in response to mouse clicks initiated at the browser. The primary difference among these applications is that the underlying server application is different; therefore, the corresponding Web application has different functionality, and the client-server communication protocol takes a different form. In particular, the system described in this report uses a novel Lisp-based application that is highly suited to the server-based approach, providing a network API that permits multiple client applications to be easily written and integrated with database management systems. In addition, the system has been in active use for more than three years by collaborating neuroscientists.

Design Objectives

The ideal 3-D visualization system would be as integral to client-side Web browsers as 2-D image rendering is to current Web browsers. Internet connections would have sufficient bandwidth to deliver complex scenes to the desktop, where they would be rendered at interactive speeds using standardized software. Such an integrated system is not yet widely available because of low-speed Internet connections, inadequate client-side hardware, and nonstandard visualization software. Although advances in computer hardware, Internet 2, and standards such as X3d28 promise to alleviate these bottlenecks, the system we describe is designed to be applicable to the hardware and software environment currently available to most neuroscience researchers. Thus, the system should meet the following requirements:

1. Provide 3-D visualization of integrated brain image data.

2. Provide the ability to interact with the 3-D visualization to rotate, move, or zoom the camera; to add or delete structural and functional data sets; and to render the scene in different styles.

3. Work for a broad range of users, even those with relatively low-powered desktop hardware.

4. Require a relatively low-bandwidth Internet connection.

5. Require little or no specialized software installation at the client workstation other than a standard Web browser and the lowest common denominator version of Java.

6. Provide the ability to be integrated in larger information systems.

7. Provide patient confidentiality since our application, as well as many other potential applications of this approach, deals with patient imaging data.

8. Provide fast response time.

9. Provide, at the server end, the ability to accept multiple 3-D images and derived 3-D model formats.

10. Provide, at the server end, the ability to run on multiple hardware platforms.

In the following sections we demonstrate that the system we have designed satisfies requirements 1 through 7, and, to a limited extent, requirements 8 through 10. Because of this, the system has proven very useful in our local neuroscience research. As noted in the discussion, our current work is aimed at more completely satisfying the remaining requirements.

System Description

Our basic approach is to modify an existing stand-alone application so that it can function as a visualization server. We therefore first describe the stand-alone application before describing how we converted it to a visualization server.

Stand-alone Application

The primary driving neuroscience application for the applications we have developed is processing, integrating, and visualizing functional brain imaging data to understand how language is organized in the brain.29 A premise of this and related brain imaging work is that because each imaging modality provides different information, the integration of multiple modalities will provide more insight into the underlying biological processes than could a single modality alone.

In our studies, the primary data are obtained from patients undergoing neurosurgery for intractable epilepsy. Before surgery, structural MRI and fMRI image volumes are collected. The fMRI image volumes depict areas of the brain that are activated during language tasks. During surgery a procedure called cortical stimulation mapping (CSM) is used to locate areas of language on the cortex that need to be avoided during surgery.30 A primary goal is to integrate and visualize these functional language data (CSM, fMRI) in terms of the underlying anatomy of the brain obtained from structural MRI, since anatomy (or structure) provides a common substrate on which to map function. The epilepsy cases present a unique opportunity because the CSM data provide a “gold standard” against which fMRI and other noninvasive methods can be compared.

Like many other HBP efforts, the software tools we have developed run on high-end Linux or Silicon Graphics computers. These tools allow (1) reconstruction of 3-D models of the cortical surface, veins, and arteries from MRI scans; (2) reconstruction of the 3-D location of the CSM sites with respect to the 3-D models; (3) integration of fMRI image volumes with the 3-D structural models; (4) quantitative correlation between fMRI and CSM language areas; and (5) visualization of the integrated data. These capabilities are implemented in two stand-alone applications, the Visual Brain Mapper31 and the Brain Visualizer.32 These stand-alone applications are limited to computer facilities within the University of Washington Structural Informatics Group. Downloading and installing the applications on a local desktop requires high-performance hardware and massive data storage measured in hundreds of gigabytes and involves maintaining and updating the software. These reasons, plus the fact that the tools were designed originally with the ability to function as remote servers, created an opportunity to develop a Web-based approach.

Overview of the Client-server System

Figure 1 shows the architecture of our server-side visualization approach. The Graphics Server, which is the key component of this architecture, awaits commands over a dedicated port that is accessed by Java applet or Common Gateway Interface (CGI)33 client programs. In response to commands that make up the server API, the server loads MRI and fMRI image volumes, 3-D surface models, and CSM sites; renders the resulting 3-D scenes; and saves the rendered images as 2-D snapshots that are accessed by the client over the Web. User interaction at the Web browser or Java applet causes additional commands to be sent to the server, which processes those commands, renders the image, and returns a new snapshot to the client. The server can also be used to perform tasks that do not necessarily result in a rendered 3-D scene, e.g., for computing various analyses of the integrated data.

The Java and CGI clients can run as stand-alone Web applications or can be a part of more complex applications. In particular, we have started to integrate Web-based visualization into our Web-based Language Map Experiment Management System (EMS),34 35 which is used to manage metadata about the 3-D volumes and models (e.g., file locations, relationships), as well as other aspects of the study, such as patient demographics, transcripts of experimental trials, and surgical photographs.

Server Side

The brain map Graphics Server is a modified version of our stand-alone Brain Map Visualizer and Visual Brain Mapper applications, which, in turn, are implemented in our Skandha4 graphics toolkit.36 37 Skandha4 combines a subset of Common Lisp—useful for fast interactive programming and prototyping—with the ability to add precompiled C-based primitive functions that significantly accelerate computationally demanding routines. One class of such functions is a set of drivers for IRIX GL and OpenGL, which are libraries that provide access to 3-D rendering, either in hardware or software. Although other languages such as perl implement similar language-specific wrappers over GL, and at least one product, AutoCad, provides a Lisp-based scripting language for graphics (AutoLisp),38 Skandha4 provides extensive additional C-based functionality, as well as the ability to easily run in server mode.

For example, in Skandha4, graphical objects are defined at the Lisp level as lists of “graphical relations,” where a graphical relation is itself a list of parallel arrays of binary data defined at the C level, each of which defines one dimension of the object. In a 3-D surface mesh the vertices are described by a graphical relation consisting of three parallel arrays defining the X,Y,Z coordinates, and the triangular facets are described by a graphical relation consisting of three parallel arrays specifying the indices into the vertex arrays that define each triangle. The advantage of this uniform representation is that a single C-based drawing function may be written to iterate through an object, changing the drawing style depending on the presence or absence of particular graphical relations in the list defining the object. For example, a bit array may be added in parallel to the facet graphical relation to specify which triangles should be set to invisible. When the drawing function sees this bit array it only draws triangles whose bits are set. If the array is not present it draws all the triangles. This capability is used in Figure 2 to set to invisible all triangles defining the surface of the brain intersected by two cutting planes. The only changes to the code are the addition of the bit array defined by the position of the cutting planes.

This basic functionality is augmented by plug-in modules containing C-based Lisp primitives that may be included or excluded during compilation. We have developed several such modules for processing MRI data; reconstructing 3-D models of the cortical surface, veins, and arteries; importing and exporting MRI volumes, images, and 3-D models in various file formats such as GE Genesis format, Analyze 7.5, and MINC image volumes, tiff, png, and VRML; and visualizing and analyzing integrated data.

Skandha4 also includes a graphical user interface (GUI) module that provides a local event-driven interface to the Lisp functions. By separating the processing functions from the GUI it is possible to create different front-ends for the same functionality. In particular, server mode is implemented by bypassing the GUI functions and calling the processing functions directly.

In server mode Skandha4 implements a preforking server model. That is, on initializing and (optionally) loading data and/or code from a specified file, the parent process forks several child processes in advance, so that each of them can handle a new connection from a client. The server then binds to a specified port, and a connecting client need only implement a simple ASCII networking protocol to access all the functionality of the server. When the client disconnects, the child process terminates, and the parent process forks a new child, which is then available to serve a new connection from another client.

In server mode, scene rendering is typically performed off screen using the Mesa software implementation of OpenGL.39 In this mode the scene as rendered in the image buffer does not appear on screen, but instead is saved in an image file which can then be returned to the client program.

All these features make Skandha4 well suited as a backend graphics server to support interactive visualization on the Web. In addition to the Web interfaces described in this report, Skandha4 in server mode is also used for interactive knowledge-based 3-D scene generation by a Web-based application that dynamically constructs 3-D anatomical scenes from model primitives.40 41 The server functionality for the 3-D scene generator is achieved simply by starting the server process with a different set of Lisp files than those used for the brain map Graphics Server. More information about Skandha4, including downloadable source code, can be found on the project Web site.42

Server API

Any client connecting to the brain map Graphics Server on the specified port has access to the entire augmented Lisp programming environment of Skandha4. However, implementing a Web-based visualization client typically requires sending only a limited number of commands corresponding to those sent by the GUI controls in the standalone Brain Visualizer.32 For example, the client that we developed for visualizing an individual patient's data (client Example 1 described below and shown in Figures 2 and 3) only utilizes commands from a small set of wrapper functions. In general the name of each Lisp function in a module is prefixed by the name of the module that contains it.

Selection and Loading of Data

(xmri-subjects)—returns a list of identifiers of all subjects available in the EMS

This list is read on startup from a file that is generated by the EMS and that contains an identifier for each subject as well as the file paths of image, graphics, and map files associated with that subject.

(xsr-select-subject identifier)—select the current subject

(xsr-load-dataset)—load structural MRI image volumes and derived surfaces as well as the list of functional volumes available for the selected subject. The specific filenames for these objects are specified in the file generated by the EMS.

Brain Properties

(xsr-mask-quadrant-web ′(sagittal axial coronal))— create a cut-away view by masking a portion of a 3-D brain surface model, texture mapping the associated MR image volume on the sides of the masked quadrant. The masked triangular facets are set to be invisible by the addition of a bit array to the graphical relation defining the brain surface.

(xsr-unmask-web)—restore the unmasked volume.

(xsr-change-window-level window level)— MRI visualization parameters.

Viewport Properties

(xsr-set-camera distance azimuth altitude) —set camera location and viewing angles.

(take-snapshot)—render the scene and save a snapshot as a 2-D image.

Functional Properties

(xsr-add-fmri identifier)—add an available fMRI volume to the loaded dataset.

(xsr-change-fmri-threshold-gain threshold gain) —set visualization parameters for functional data.

(xsr-load-map-sites)—add CSM sites for the loaded dataset. CSM sites are given by a file defining a set of points in the coordinate system of the 3-D surface model, each of which may also have a label defining it to be a location on the brain surface where language function was found.

(xsr-remove-map-sites)—remove CSM sites

Image (click operation)

(xsr-fake-downclick-get-structure x y)—perform a 3-D pick operation at the (x, y) screen coordinate specified by a mouse click, and query the label assigned to the picked 3-D structure. The label is either the anatomical name of a structural component or the number of a stimulation site.

Client Side

Any of the above commands may be executed by connecting to the server machine on a specified port and simply typing the commands at the Lisp prompt. The scene is then rendered by the (take-snapshot) command. However, to provide a user-friendly client-side application it is necessary to create graphical user interfaces that send these commands in response to mouse clicks. We have developed several such client implementations that connect to the brain map Graphics Server. These applications are implemented as Java applets or perl CGI pages, but other approaches, e.g., Java Server Pages (JSP), can be also used. The applications we have developed include a Java applet for visualization of an individual patient's dataset, a CGI client for visualizing multiple patient CSM datasets in standard space, and various CGI clients for analyzing the relationships between fMRI and CSM measures of language.

Example 1: Visualization Applet for Individual Patient Datasets

Our Java-based Web interface is implemented in Java 1.1 (supported by both Netscape 4.x and IE 5.x and higher), with the Swing GUI. This Brain Visualizer applet provides nearly the same functionality as that of our stand-alone Brain Visualizer application,32 which was developed to visualize integrated data from structural MRI, functional MRI, and surgical stimulation sites. Functional activation can be studied both on the cortical surface and in depth of the brain structures. Source code for the Java applet is available on the project Web page.42

Figure 2 is a screen shot of the entire desktop with the running applet. The applet is configured to echo in the Java console all the commands sent to and received from the server. When started via a URL without any parameters, the applet connects to the server, issues the (xmri-subjects) command to obtain the list of subject identifiers, displays this list in the applet, and sends the (xsr-select-subject identifier) command in response to the user selection. The subject identifier can also be appended to the URL at startup, in which case these first two steps are bypassed. Once the subject has been selected, the server loads the associated MRI 3-D image volumes for anatomy, veins, and arteries as well as corresponding 3-D models (xsr-load-dataset). The user can then control the server using the GUI shown on the right-hand side of the applet, which includes tabs corresponding to the grouping of API commands described above. User interaction with the GUI is translated into a sequence of commands that are sent to the server.

The controls are organized into three panels. The “Brain Properties” panel shown in Figure 2 allows the user to show the cut-away view and control the appearance of the MRI slices. For example, the sliders labeled “Axial,” “Sagittal,” and “Coronal” are used to select corresponding cutting planes. Pressing the Mask Wedge button sends the (xsr-mask-quadrant-web ′(sagittal axial coronal)) command to mask out a segment of the 3-D brain surface model and render a new image. This command causes the server to set to invisible those surface facets that are within the segment to be masked then texture-map the structural MRI intensity values onto the three cutting planes and render the image. With the controls shown in the figure the user could then click the “Unmask Wedge” button, which would send the (xsr-unmask-web) command to restore the masked surface. The user could also click the “Set Window/Level” button, which would send the (xsr-change-window-level window level) command to change the MRI slice appearance.

The user can click one of the other two tabs to access additional controls. The “Viewport Controls” tab provides camera controls (Fig. 3A), which allows viewing the scene from any angle and magnification. The “Functional Properties” tab brings up controls for selecting one of the functional volumes available for this patient, manipulating its visualization properties, and loading/removing CSM sites (Fig. 3B).

Visualizing functional activation

Functional activation from fMRI is typically observed below the surface of the brain and often occurs in the deeper structures, e.g., the bottom of the sulci or the medial wall of the temporal lobe. For this reason, fMRI data cannot be readily compared with the stimulation map on the surface of the brain or even shown on a 3-D view along with the anatomical features of the brain surface reconstruction. We have implemented two alternative methods for visualizing the relationship between fMRI and stimulation data. With the first method, we highlight the surface areas that are located in the immediate vicinity of the fMRI activation sites. This technique, however, does not reveal activation in deep structures. As an alternative to this method, we implemented a method in which activation anywhere in the brain is projected onto the surface, using the point between the anterior and posterior commissures as a projection point. The user can adjust the visualization by setting the Threshold and Gain parameters (Fig. 3B). These parameters are passed to a weight function, which reflects a desired representation of fMRI characteristics for every triangle of the 3-D model of the brain. This function is then used to modify the color characteristics of the surface triangles. These methods permit direct comparison of the fMRI results and the stimulation map.

Deep activation is typically visualized by superimposing color-coded fMRI intensities onto individual structural MR slices. To achieve better visualization of individual slices in the context of the brain, we combined the 3-D models with individual slice images and implemented a cut-away view with functional activation color-mapped onto the grayscale MRI slices. The color map function is controlled by the same Gain and Threshold parameters (Fig. 3B).

Integration with the experiment management system

In parallel with this project we continue to develop our Web-based EMS.34 35 This application dynamically generates HTML Web pages based on searches of an underlying relational database. It organizes and manages the experimental data, including patient demographics, surgical photographs, transcripts of the CSM trials, functional and structural image volumes, 3-D models, and language maps—the metadata needed by the brain map Graphics Server.

We have begun to integrate the EMS with Web-based visualization by allowing the user to select a patient in the EMS, then click on a link to invoke the visualization applet with that patient preselected. Further development and tighter integration will enable results of complex queries (e.g., particular language error types) to be sent to the brain map Graphics Server for visualization.

Example 2: CGI Interface for Visualizing Multiple CSM Maps on the Average Brain

Figure 4 shows an example of another Web interface we developed, in this case to present pooled CSM data from a selected group of patients in a common frame of reference. The frame of reference is defined by the average structural MRI volume of 305 brains of normal patients.43 To map the CSM sites to the average brain volume we used the MNI_AutoReg software44 to find the best linear transform between the patient-specific coordinate system and the average brain coordinate system. This operation was performed offline, and the resulting transforms were stored where they could be loaded into the Graphics Server and used to transform each set of CSM sites to the common frame of reference. To provide the context for this visualization, we extracted a surface from the average brain volume, using our image segmentation tools.31

The Web interface shown in Fig. 4A permits the user to select a subset of patients for loading and then display the transformed CSM sites for all the selected patients on the average brain (Fig. 4B). The sites identified as language essential are distinguished by size and color. Rotation and zoom controls allow the user to adjust the rendering of the 3-D scene. The Web interface is implemented as a CGI script that utilizes the CGI perl module.33 The script accesses a separate running instantiation of the Graphics Server that is initialized by loading a set of Lisp functions that adds new functionality. In particular, the 3-D model derived from the average of 305 brains is preloaded at startup to improve response time, which, because of server memory limitations, is not possible to do for all 100+ individual patient datasets that can be visualized by the applet described in Example 1. The CGI approach is adequate in the case of the average brain because once the average brain model is preloaded, only a small amount of data need to be reloaded for each mouse click, namely, the individual CSM sites. However, it would be problematic to implement a more complex Web interface, like the one shown in Example 1, using this simple CGI approach, because of the need to preserve considerably more state information.

Status Report

Usage and Neuroscience Results

The Web-based visualization and analysis tools that we describe in this report are used routinely by our collaborators in the departments of Psychology, Speech & Hearing Science, Neurological Surgery, and Radiology at the University of Washington. These tools have greatly facilitated collaboration among a distributed and diverse group of researchers working on various aspects of the UW Human Brain Project. Individual researchers use these Web interfaces to visualize and analyze integrated data from several modalities without having to come to the SIG laboratory or needing to understand how to install or use the underlying applications.

More recently, the visualization applet is being used by one of us in planning language studies during neurosurgery. Before neurosurgery, fMRI activations are mapped onto the MRI-based structural models and then loaded into the database where they are available for viewing in the applet. The rendered image of fMRI activation projected onto the surface is then used to help determine during neurosurgery where to perform CSM and where to insert microelectrodes for recording of the responses of single neurons to language tasks. The goal of this study is to determine whether neurons within the fMRI activation regions show firing rates that are different from neurons that are not in the fMRI activation regions. The ability of the neurosurgeon to do this planning from his or her own office, without coming to our laboratory, has greatly facilitated this process.

The Web-based tools have facilitated scientific findings pertaining to language function of the brain that have resulted in several neuroscience publications: Corina et al.45 46 compared functional language maps derived from CSM and fMRI. Our Web-based data integration techniques were used to examine qualitatively and quantitatively the spatial proximity between sites of naming disruption observed during CSM and fMRI activation in the same patients. The data from a common task of visual object naming indicate only moderate correspondence between these two measures.

Visualization of the distribution of the CSM sites from multiple patients superimposed on the average brain was used by Martin et al.47 They generated multipatient maps, which summarize essential language sites for various patient groupings. Comparisons were made between language sites of patients with right-brain dominant language and those with left-brain language dominance. Other groups compared were patients with low versus high verbal IQs (VIQs) and males with low VIQs versus females with low VIQs. Serafini et al.48 used our Web-based tools to visualize and compare several functional modalities, including fMRI, proton echo-planar spectroscopic imaging (PEPSI), and event-related potentials (ERP) during a visual object naming task.

Finally, Kinbar et al.49 used the Web interface to analyze distributions of essential language sites with respect to the type of error made during the CSM procedure. They tabulated the frequency, location, and nature of error types, finding that semantic paraphasia was most common, followed by phonological reductions, phonological paraphasia, neologisms, and semantic/phonological blends. Semantic paraphasias were seen across wide regions of the cortex. Phonological paraphasias were more limited in extent and found with stimulation to the superior and middle temporal gyri (STG, MTG), supramarginal gyrus, and the inferior frontal gyrus. Phonological reductions were the most regionally specific and associated with posterior STG, MTG, and ITG, with a surprising lack of frontal lobe involvement.

Performance

Our collaborators typically experience response times of about 5 seconds per generated image (longer when updating the scene involves complex computational tasks). This response is composed of the following chain of events: (1) the server receives a command from the client and modifies the 3-D scene, (2) the server renders the new scene and saves it as a compressed image, (3) the client retrieves the image, and (4) the client decompresses and displays the image. Step one—modifying the scene—is highly variable. Most of the tasks (e.g., changing camera position or adjusting MRI window and level) require no more than a few milliseconds, whereas a few other commands, e.g., loading the dataset or adding functional volume, may take several seconds. Step two—rendering the 3-D scene—depends on scene complexity (i.e., the number of triangles in all 3-D models) and the image size. For example, the scene shown in Fig. 2 (about 1,000,000 triangles, image size 512 × 512 pixels) takes about 6 seconds on a Pentium IV 1.5 GHz computer. Steps three and four are largely beyond our control because they depend entirely on the client's computer and network capabilities. Step three— retrieving the image—depends on both the image size (769 K uncompressed and about 200 K compressed) and network throughput. Although image retrieval could be a bottleneck on a very slow network (e.g., about 40 seconds on a 50-kbs dial-up connection), it should not take more than a second on a fast network. The last step—decompressing and displaying the image—depends on the client's Java VM performance but should be well under 1 second. Therefore, rendering the scene in software (estimated as about 3 seconds for the example shown in Fig. 2) currently constitutes the bulk of the response time and presents a bottleneck that we are addressing on the server in our current work (see Discussion).

Discussion

In this report, we described a server-side approach to implementing Web-based interactive visualization and analysis tools for large 3-D neuroimage datasets. During the approximately three years that the system has been in active use, the primary lesson we have learned is that this approach works very well for the types of users with whom we work. Although the server-side approach does not permit as much interaction as stand-alone or client-side applications, and although the response is not real-time, these disadvantages are far outweighed by the convenience of having complex 3-D visualizations of patient data available directly at the user desktop on low-end machines without the necessity of installing special software and without the need to come to our central facility.

Our primary users are working biologists who only want to use the computer as a tool in the same way they use office applications or gene databases. Our experience suggests that these types of users are far more prevalent than the more computationally sophisticated users who have the knowledge and hardware to install complex client-side programs. In some ways our greatest success comes about when the systems we build are so simple to use that the users wonder why informatics is so difficult as to warrant an entire field of study.

Although hardware and software are rapidly improving, we believe that the server-based approach makes sense now because client-side rendering technologies are still too complex and slow for most users. In addition, neuroimaging datasets are typically very large (in our case, approximately a gigabyte per subject), and Internet bandwidth is not yet high enough to support massive transfer of such data. Even at the client end neuroimaging data may exceed the memory capacity of a typical user workstation, resulting in low performance or even program crashes. With the server-side approach, all these issues of compatibility, data storage, and performance need only be addressed once—on the server—which makes this approach appealing for neuroimaging applications that involve visualizing large image volumes and 3-D models on workstations in the laboratories.

The system we have built meets objectives 1 through 7 described in the design objectives. It provides advanced visualization capabilities and the ability to interact with the scene (objectives 1 and 2). Because no 3-D rendering is performed on the desktop, the user application (client) can run on low-cost hardware (objective 3), and because only 2-D images are sent from the server, relatively low bandwidth network connections need be used (objective 4). Because the client is either a standard Web browser for the CGI applications or a simple 2-D Java applet, little or no software installation is required beyond that which is available on most desktops (objective 5). We believe that this feature is an important reason for the success of our approach in our collaborative research. Because the client can be accessed via a simple URL that may have the subject identifier appended to the URL, the client can easily be included in a system that can generate the proper URL, as we have done with our EMS (objective 6).

The server-side approach also makes it much easier to ensure patient confidentiality (objective 7) compared with a client-side approach in which all imaging data are sent to the client. In the latter case the image files may include header information that could be used to identify the patient if not scrubbed. More insidiously, structural MRI is of such high resolution that it is possible to reconstruct the face of the subject from an image volume. In the system we have built, only 2-D image snapshots are sent without any header information. In addition, the 2-D snapshots only show image data within the confines of the brain (e.g., Fig. 2), so the face cannot be reconstructed. Thus, the server-based approach could be instrumental in giving users access to the analysis and visualization of essential aspects of patient imaging data while eliminating concerns about patient confidentiality.

Although the system we have built partially meets design objectives 8 through 10, there is room for improvement in these areas. One desired improvement is to decrease the 5-second response time (objective 8) by reducing the rendering time at the server (although we have heard no complaints from the users about this response time, most likely because they have never had anything like this capability in the past). We are investigating both hardware rendering and parallel data processing.

As noted previously, rendering is currently performed using the off-screen rendering functionality provided by the Mesa3D graphics library.39 The drawback of using this library is that it performs software rendering, which is computationally demanding and not nearly as fast as hardware rendering. Three-dimensional graphics hardware, driven by the PC gaming industry, has become not only highly capable, but also very affordable, and Web-based neuroimaging applications should capitalize on this technology.

It is feasible to take advantage of 3-D graphics hardware by rendering on screen and saving the images, i.e., by using screen grabbers, but this approach would make the server more cumbersome and would prevent the screen from being used for other purposes. However, the latest releases of hardware-specific OpenGL implementations support hardware-based off-screen rendering capabilities, which potentially makes them ideally suited for a server-based approach to 3-D medical imaging visualization. Off-screen rendering is generally not a part of Open GL core functionality but may be provided by extensions like GLX 1.3 and others. We are currently experimenting with this approach and have achieved promising preliminary results.

Another advantage of the server-based approach is that the server can run on high-performance hardware that makes possible parallel data processing methods. Even inexpensive modern servers typically implement a Symmetric MultiProcessor (SMP) architecture with two, four, or more CPUs, while advanced servers may implement massively parallel architectures. In our group, pilot studies were conducted to explore several different approaches to parallel rendering in software using the Mesa3D library, including (1) the Parallel Virtual Machine (PVM), (2) lightweight threads, and (3) System V Interprocess Communication (IPC).50 Lightweight threads and system V IPC approaches permit the utilization of several CPUs available on the same server, whereas the PVM approach can be used to distribute work among many computers on the network. Our preliminary studies confirm that rendering of 3-D scenes in software could be accelerated dramatically using parallel techniques.

A third approach to reducing rendering time is to take advantage of techniques that have been developed for decimating the number of triangles that need to be displayed without sacrificing image quality.51

Although parallel processing and decimation are promising approaches, our experience suggests that the best speedup will be obtained by using graphics cards specifically optimized for hardware rendering. On the other hand, computationally intensive tasks that do not involve surface rendering (such as spatial queries or volume rendering) may benefit from other approaches. In future work we will investigate the tradeoffs among these alternative approaches. The advantage of the server-based approach is that these enhancements will be transparent to the end user except in the sense that response time will decrease.

The server we have built can currently only accept image volumes and 3-D models in a few formats (Minc52 and Analyze 7.5 for image volumes, and an in-house Skandha4 format for 3-D models). The server also only runs on SGI and Linux computers but not Windows. Because Skandha4 is not widely supported (like many other brain imaging toolkits), and Lisp is not the favorite of most developers, we currently are porting our stand-alone and server-based applications to a Java3D environment, which will allow us to take advantage of Java3D hardware acceleration (objective 8), compatibility with many image and 3-D model formats (objective 9), and cross-platform portability (objective 10).53 The stand-alone application is being designed so that it can be run easily as a graphics server, using the basic techniques described in this report. In the meantime, the current code may be of interest to Lisp programmers developing similar applications.42

Although this report presents a specific application in brain mapping, the techniques should be applicable to other areas that involve visualization of large 3-D image datasets. Three-dimensional imaging is playing an important role in biomedicine ranging from the molecular level to macroscopic anatomy and is becoming more important in medicine for diagnosis and treatment planning. Current teleradiology systems primarily deal with 2-D images. Remote interactive visualization of 3-D models obtained from patient-specific image datasets has the potential to greatly advance such fields as telemedicine and teleradiology,6 7 8 in which additional functionality is often desired beyond basic image transfer and management. When accessed via a Web-based medical record system such as the University of Washington's Mindscape,54 a 3-D visualization server could bring advanced imaging tools directly to the primary care provider. By requiring only standard desktop tools such as a Web browser and by dealing with patient security issues at the server, such tools could become a valuable adjunct in clinical medicine as well as research.