GIS, Virtual Reality and Real-Time Simulation

Barry Bitters, Senior Software Engineer, Lockheed Martin Information Systems, P.O. Box 9340, Hurlburt Field, FL 32544; ph. [850] 936-1060; FAX [850] 936-1060; barry.bitters@lmco.com.

ABSTRACT: For many years, the U.S. military has used advanced, real-time, simulation technology to create and interact with virtual environments in an effort to develop realistic military training situations.Geographic information systems (GIS) technology has played an instrumental part in the real-time simulation arena. As a tool for creating realistic environments composed of both man-made and natural features, GIS and image processing technology has been used to generate real-time simulator databases for regions all over the world. Databases for real-time simulation systems store real-world, correlated information on elevation, man-made and natural features, and aerial photography for a region. This information is used as a source of data for developing both visual and non-visual (radar, infrared, night vision, etc) displays used in a variety of simulation systems. Traditionally, database generation for real-time simulators has been performed using expensive and specially designed hardware and software. Over the last year, procedures for generating real-time simulator databases using inexpensive commercial-off-the-shelf (COTS) hardware and software have been designed. As a part of this study, rapid database generation techniques have also been explored. This paper reports the results of this research into a low-cost COTS hardware and software solution that can be used in simulator database generation.

Introduction

For many years, the U.S. military has used advanced, real-time, simulation technology to create and interact with virtual environments in an effort to develop realistic military training situations. To provide truly realistic training, the U.S. military has always strived to employ real-time simulation technology rather than depend on the less realistic "canned" simulation sequences. In either type of simulation scenario, geographic information systems (GIS) technology has played an instrumental part in the real-time simulation process. Since the use of simulation technology first requires the development of a virtual environment, it would only be logical to exploit GIS technology. The inherent geo-spatial capability of GIS software to accurately map feature position is an essential part of the virtual environment development process.


Figure 1. A high-end, real-time, simulation system used in the U.S. Air Force
AC-130U program.

As a tool for creating realistic environments composed of both man-made and natural features, ARC/INFO has been used for many years to generate real-time simulator databases for regions all over the world. Primarily as a tool to capture, manipulate, and store locational and attribute information on man-made and natural features on the ground, ARC/INFO has shown itself to be an invaluable asset to the database generation process. However, the annual maintenance, training, and hardware costs that are required to sustain a multi-seat, UNIX workstation environment are excessive. This paper reports on a research and development effort to implement a more efficient and low-cost hardware and software solution to existing real-time simulation database generation capabilities.

Real-Time Simulation vs. Hollywood Style Simulation

Simulated environments or virtual worlds, in the form of movies, videotape, and all types of simulation hardware and software, are in fact created from a sequence of individual pictures recorded one-at-a-time onto film, magnetic medium, or displayed on a monitor. As the medium is played back, the eye and the brain think that they perceive motion between each displayed frame. When the film or videotape is played back at 30 or more frames per second, we see smooth, animated motion.

Today, if very high detail and smooth motions are required, two options exist for creating a simulation from a virtual environment: the Hollywood simulation process or an advanced hardware based simulation system. In the Hollywood simulation process an animation sequence is generated and stored for future viewing. Emphasis is on high detail rather than processing time, therefore rendering of individual frames can take some time. What kind of times are we talking about?? Hundreds of hours to create a virtual database and sometimes hours of processing per frame of film are required. Once each frame has been rendered and stored, the animation sequence is transferred to another medium, usually videotape one frame at a time until the whole sequence has been recorded. Keeping in mind that each second of video animation is a minimum of 30 to 60 pictures, and that 30 minutes of video at 30 frames per second is 54,000 individual images, this process can take weeks or even months to complete. The result is however, a very realistic, detailed, and smooth motion picture at a sustained rate of 30 or more frames per second.

In real-time simulation however, the emphasis is not on the creation and storage of a sequence of images to view as a movie. No, the emphasis is on the ability to fly-through, move about, or interact with a virtual environment at the same time the "animation" sequence is being generated. To do this, a visual database is created by storing locational and attribute information on all relevant man-made and natural features. Then advanced hardware and software are used to perform repetitive mathematical computations involved in the development of the scene content and scene rendering process. This processing is performed in the split-second just before the display is viewed. In real-time simulation, special hardware/software systems called image generators are used to convert raw geographic information into realistic graphic displays. These displays are generated in real-time as one moves about the virtual environment.  By generating visual displays in real-time, we hope that a realistic view of a virtual environment is produced.

However, limitations will soon arise due to the size of computer memory and processor speed. As the detail to be displayed increases, the amount of data to be processed increases; as the required resolution of the scene to be displayed increases, the size of the computer files also increase; as the number of displayed colors increases, the file sizes also increase. All these increases in file size ultimately result in a distinct reduction in display refresh rate. This means that the motion sequence slows down on the screen. Soon the "jerkiness" and hesitations of the display becomes noticeable, and can be attributed to more data than the processor can handle. By the time the highest visual quality of each scene is achieved, the animation has slowed down to the rate of a slide show.

Source Materials

As a minimum, there are four different types of data that are essential to create a complete real-time simulator database: elevation data, ground imagery, feature data, and a library of three-dimensional models.

Elevation Data

Elevation data, sometimes called the "terrain skin", is a numeric representation that accurately portrays the character of the landscape. This data is an essential element of any database used to develop an accurate virtual environment. However, a virtual database can be generated on a perfectly flat terrain skin. Elevation data can be in a gridded raster format or preferably in either triangulated irregular networks (TIN) or triangulated regular networks (TRN). TINs and TRNs provide a more computational and storage space efficient format for elevation data.


Figure 2. The elevation data, in triangulated
irregular network (TIN) format, used in this project.

For this project, USGS SDTS 7.5' digital elevation model (DEM) data was used. In ArcView and the Spatial Analyst extension, the original SDTS DEM data was converted to a standard Arc/Info GRID upon import. It was then converted to a TIN using the 3D Analyst Extension to ArcView 3.2. This TIN was used as the elevation data set for this project. Other available sources of elevation data are:

Ground Imagery Data

Aerial photography or some other form of image data is also essential. It serves as the spatial reference plane for all feature information that is to be stored and then subsequently displayed. If of high enough resolution, image data can also serve as the ground texture that is displayed during the finished simulation. The imagery should be ortho-rectified to insure that all distortion has been removed from the original image.

In this database, a single 9"x9" USGS panchromatic image was scanned at 12 micron and then registered and ortho-rectified. The final ortho-rectified image was saved at a one-meter ground sample distance in TIFF format.


Figure 3. The ortho-rectified image used in this project.

Other sources of imagery that can be used in simulation systems are:

Feature Data

Feature information is used in the real-time simulation process to provide discrete locations for the placement of 3D models in the final display. Each record of information includes a precise position on the terrain for the instancing of a 3D model. In addition to locational information, each record contains attribute information concerning feature classification, model type, model orientation, model size, model scale factor, feature material code, and 3D model name.

Accurate and detailed feature data is the most difficult and most time-consuming information requirement in any simulation project. Because of the level of detail requirement for this project, all feature information was extracted directly from the ortho-rectified aerial photograph. Pertinent features were interactively identified on imagery in a heads-up digitizing environment using ArcView. For each feature identified on imagery, location and attribute information was stored in a shapefile record.


Figure 4. The feature data generated for this project.

Three-Dimensional Models

A library containing a wide variety of man-made features stored as three-dimensional models in standard OpenFlight format is needed to allow the realistic display of feature data. Figure 2 illustrates the quality, the detail, and the variety of models in a 3D model library.

For this project, a library of over 300 models has been assembled. Models from many different sources have been used in this model library, and of course each source used a different set of modeling rules. As a consequence, an on-going project has been undertaken to recreate each model using a standard set of modeling rules.

There are many vendors of 3D models. However, most 3D models available on the open market are not optimized for use in real-time simulation. For this reason, there are very few commercial sources of 3D models that can be used in real-time simulation systems. Significant effort, time, and technical and artistic expertise are required to transform most 3D models into a form that is suitable for use in real-time simulation systems.


Figure 5. Examples of 3D models used in this project.

Real-Time Simulator Database Creation

Figure 6 shows the generalized processing that is necessary for generating and viewing a real-time simulator database. The process involves four phases:


Figure 6. Generalized process for creating simulator databases.

Import/Reformat Phase

During the import/reformat phase, all raw source data is imported into ArcView. Also, all data is transformed to a single coordinate frame, datum and projection. This insures that a close correlation between each data set is maintained. Additionally, elevation data is reformatted from its native format to an ARC/INFO triangulated irregular network (TIN) format.

Preprocess/Augmentation Phase

The preprocess/augmentation phase is the most time consuming phase of processing. In this phase, detailed feature information is collected and stored in shapefile format. Feature information is collected through direct interpretation from high-resolution aerial photography. Features are classified using a unique, three-tiered, hierarchical classification scheme - a classification scheme based upon those features normally seen in man-made and natural environments. This classification scheme and a unique set of shapefiles/themes and their associated attribute sets were developed for this project. The shapefile types and their associated set of attributes were developed, taking into account those features that are normally seen in man-made and natural environments, and modeling characteristic necessary to realistically portray these features in the virtual environment. Other steps that are performed during this phase are:

All processing during this phase can be performed in the ArcView, version 3.2 environment. However, with minimal additional software development time, this phase of processing could also be performed in the ArcView 8.1 environment.

Data Preview Phase

The Data Preview Phase is the first chance to view the virtual environment in an integrated state. Using the SiteBuilder3DTM Extension to ArcView 3.2 developed by MultiGen-Paradigm, Inc, all data, the elevation TIN, feature data, ground texture data and 3D models are integrated into one virtual environment. An elevation surface is created and covered with ground texture. At the same time, 3D models are placed on the terrain surface based upon location information that was stored in the feature shapefiles. The final result is an interactive viewer containing a virtual environment that allows an analyst to freely move about the virtual environment and perform quality control checks from unlimited vantage points.


Figure 7. A screen snapshot of SiteBuilder3DTM session. Note that this is a
still image extracted from a live fly-through.

Because the SiteBuilder3D extension was developed within the Avenue environment, currently the data preview phase must be performed using ArcView 3.2. Future releases of this software will be extensions to version 8.1 of ArcView.

Integration Phase

During this phase, the image data, elevation data, and the 3D model that were instanced in the database are transformed into an integrated database in OpenFlightTM format. This integrated databse is then converted into a data structure called a run-time database. A run-time database is a data structure that has been optimized for display in a real-time image generation environment.

Currently, the SiteBuilder3D extension to ArcView can export only elevation and feature data in standard OpenFlight format. It cannot create an integrated OpenFlight data set containing imagery, elevation, and feature data. At this time, the interium OpenFlight data generated by SiteBuilder3D must be post-processed into a run-time format using other software products.

Other Visualization Environments

In the past, viewing a real-time simulation database was only possible on large, expensive, and specially designed hardware systems. However, with recent advances in PC graphics hardware and software, it is now possible to move through real-time simulation databases on off-the-shelf PCs with Pentium III processors. For optimum performance and to obtain screen refresh rates of 30 frames per second or better, OpenGL compliant graphics cards are essential when viewing real-time simulation databases.


Figure 8. ArcView version 3.2 and the SiteBuilder3D viewer showing feature data at the beginning of the
quality control check.

While creating any real-time simulator database in the ArcView environment, it is essential to render and view a database. MultiGen-Paradigm, Inc. markets an ArcView Version 3.2 extension - SiteBuilder3D. This extension allows one to rapidly create a limited use run-time database and then interactively move through and view the data from an unlimited number of vantage points.

Special viewing software is essential to allow free movement throughout a real-time simulation database. One freeware viewer and several commercial products are currently available to view OpenFlight databases in real-time.

TerraSim, Inc. has developed and markets TerraToolsTM an easy-to-use interface to its powerful terrain and feature generation tools. This software enables a user to rapidly construct complex virtual worlds without tedious manual modeling. In many cases, GIS source data can be freely downloaded from the Internet and converted to a realistic 3D environment in a matter of hours. TerraTools supports the integration of custom GIS source data, CAD models, and design and architectural models, allowing the production of highly accurate 3D environments for urban, suburban, and natural environments. TerraTools is supported on Windows NT/2000 and SGI IRIX workstations.

MetaVR, Inc produces a low-cost PC visualization tool: the Virtual Reality Scene Generator (VRSG)TM. The VRSG viewer can render images at 30 to 60 Hz on a PC system using Intel Pentium III technology, Microsoft Windows, and a NVIDIA GeForce3 based graphics card. The VRSG application uses the Microsoft DirectX API to provide textured, real-time visualization of dense multi-player virtual worlds. Advanced terrain and texture paging technology enables VRSG to depict ultra realistic virtual worlds derived from both small and very large, detailed imagery feature and elevation databases.


Figure 9. A screen snapshot of a VRSGTM database
of Bicycle Lake, CA.

CG2, Inc. distributes a free PC visualization tool for real-time simulation databases: Audition. Although it is not a advertized as a full-function, real-time, simulation database viewer, Audition allows one to move freely about an integrated OpenFlight database. Audition was designed as a viewer for 3D models. However, the Audition software will also display and allow movement through integrated OpenFlight databases at very impressive speeds. This software was originally developed to exploit OpenGL graphics accelerator technology. The current version of Audition can display real-time images in a reduced resolution rendered format or in a wire-frame format on PCs not equipped with OpenGL compliant graphic hardware.

Terrain Experts Inc. (TERREX) produces a high-end real-time simulation database development tool: TerraVista. This software runs on Windows® 95, 98 2000 or NT, and can produce 3D terrain databases in either OpenFlight or TerraPage format. The TerraVista terrain viewer is a high performance real-time visualization capability that can be freely distributed. It includes the following capabilities:

Potential Uses of Real-time Simulation

Traditionally, because of the high cost of hardware and software, real-time simulation has been almost exclusively used by the military, the airline industry, and the space program. Today, as a direct result of low-cost, but very rigorous PC computing capabilities, this technology no longer has to be restricted to the military and the space program. Inexpensive hardware and software is now available that allows the generation and display of realistic virtual worlds in real-time. This new technology has widespread applications in the GIS arena and because of its low-cost it can now be used in a wide variety of applications.

What about using real-time simulation to display basic GIS data? The average layperson often cannot visualize and comprehend the abstract display of geographic information system (GIS) data. Those flat maps of multiple GIS layers we generate often mean nothing to the casual observer. And those lengthy legends included in GIS output can add an additional confusion factor. Transforming GIS data into a cartographically accurate three-dimensional visualization would allow both experienced and inexperienced viewers to quickly understand the structure and features of an area through an interactive exploration of a virtual environment created from the GIS data.

The urban planning field is concerned with accurately understanding and depicting the relationships between urban development and its surroundings. Often, site design problems are not readily apparent from engineering drawings. By using real-time, three-dimensional visualization during the early planning stage, many design problems can be discovered. This type of visualization allows the analysis of a site and its surroundings from an infinite number of viewpoints, not just from those viewpoints generated by an architect. Once a formal site design has been created and the approval process begins, a real-time simulation can provides clients, governmental authorities, and interested citizens a better understanding of the appearance and impact of the project. A highly detailed visualization that incorporates local landmarks and features and clearly portrays the relationship of the site to its surroundings also gives credibility to the design and the developer.

Conclusions

In conclusion, with the advent of powerful, low-cost PC computer technology, it is not only possible to create very detailed real-time, three-dimensional, simulation databases but it is also possible to interactively fly through these databases in near real-time.

GIS databases created using ArcView and ArcGIS, can be easily transformed into realistic virtual environments. Image-based feature and terrain data can be developed in ArcView, in both Version 3.X and 8.1 and then transformed in to OpenFlight format using the SiteBuilder3D Extension to ArcView 3.2. In future releases of SiteBuilder3D, this entire process will be available in the ArcView version 8.1 environment.

Recent advances in graphics display technology will have major impact on the real-time simulation arena. As these advanced graphic display technologies becomes more integrated with the current PC graphic display technology, a significant increase in feature detail and display polygon count will result. These new graphics capabilities have the potential to increase real-time simulation display rates by 10 to 100 fold. The resulting data content will allow the generation databases that will appear to be realistic renditions of the real world.

Acknowlegements

Without the direct support of Lockheed Martin Information Systems, and the 19th Special Operations Squadron this effort would not have been possible. To MetaVR, Inc., MultiGen-Paradigm, Inc., TerraSim, Inc., and Terrex Corporation we would like to express ours sincere gratitude for their assistance, understanding, and concern. Without their help this project would have gone nowhere!