To really see all of a location, up close, panoramic video lets you look all around as you move through streets, buildings, or grounds. When linked to GIS, one window shows a map with “you are here” updated as you move through the space in the adjacent panoramic video window. This capability is valuable where satellite imagery is limited by shadows or blocked by tall buildings, or when you simply must have a complete street-level view, such as in reconnaissance, planning, and emergency preparedness applications. This talk demonstrates these and other applications.
iMove has been developing spherical imagery for more than seven years, beginning with photographic and computer-generated (CG) spherical still panoramas, and evolving to spherical video in both mediums. Its spherical technology has been used in computer games, such as Broderbund’s Journeyman series. Adobe has adopted iMove panoramic and seaming technology in their popular photo editing series, Photo Deluxe, and Autodesk has done the same for CG panoramic stills and video in their 3D Studio VIZ computer modeling and rendering product.
iMove’s current focus is on government markets for spherical video. It supplies proprietary imaging hardware and software products, customized if needed to meet customer requirements.
Spherical imagery is digital imagery that captures a wide area of a scene, up to a full spherical view. It can be still images or video, real (photographic) or computer-generated (such as by a CAD package used by an architect), viewed live or offline. A number of companies produce tools for panoramic imagery, defined as 360º horizontally and up to 180º vertically. A panorama that contains less than a 180º vertical view is referred to as cylindrical (e.g. Apple QuickTime VR’s original product), while one with a full 180º vertical view is spherical. Imagery that is viewed offline can be post-produced to enhance the material, have links added to related information (which iMove calls “hotspots” and are generalized Web hyperlinks), interfaced to geographic information systems (GIS), etc. More about that below.
Single-lens panoramic cameras typically operate by gathering imagery from a parabolic mirror positioned near the lens. Hence they capture cylindrical panoramas, perhaps 30 to 45 degrees above and below the horizon. Their more costly cousins, multi-lens panoramic cameras, require a separate seaming step once the video has been captured, but can capture the entire sphere and at much higher resolution, depending on the number of lenses employed.
The power of full spherical imagery comes from its ability to see everything in a scene from a single vantage point. This new capability is crucial in such environments as unmanned vehicles and high security. And it delivers a convincing “you are there” experience for those who need to get a sense of an area or facility prior to or instead of actually visiting it. The digital nature of spherical imagery allows it to be transmitted over networks including the Internet, and distributed on CDs and DVDs. It also allows spherical imagery to be easily connected to complementary processes such as motion detection or object recognition, resulting in an even more powerful capability.
In order to get a true feel and appreciation of spherical imagery, it is important to view some sample images, such as those found at the web sites of panoramic imagery technology developers iMoveInc.com, Apple.com, and BeHere.com.
Figure 1 shows an iMove six-lens spherical video camera system, consisting of camera, storage backpack, and wireless palm-top camera controller. The camera is about 10 inches tall and weighs about three pounds. The backpack holds a portable COTS (Commercial, Off-The-Shelf) computer with removable hard drive for video storage, weighs about fifteen pounds, and stores up to two hours of spherical video on a single 72GB drive. The camera is controlled using a COTS palm-top computer (shown here is a Compaq Pocket PC model), which wirelessly directs capturing, compression ratio, exposure mode, etc. A battery belt provides for an hour of continuous operation. The camera may also be operated by automobile power using a cigarette lighter connector. The camera can be handheld or mounted on a tripod, robotic vehicle, top of a conventional vehicle, etc.
Figure 1. iMove Spherical Video Camera System
The camera captures fifteen frames a second, each frame consisting of six 768x768 full-color images. The raw 50MB/second data rate is reduced within the camera to about 7MB/second, using real-time motion JPEG compression.
Once captured, spherical video can be edited, corrected, seamed, optionally augmented with hotspots, linked to maps, and further compressed.
Typical editing steps are trimming the head and tail of a clip, adding a title, and editing clips together. The six video streams from the iMove camera are then automatically corrected, seamed, and compressed into a single seamless panoramic video stream. Calibration correction compensates for any slight manufacturing variations in lens positioning within the camera head and lens/sensor alignment. Color correction ensures faithful color rendering across all streams.
Hotspots, a generalization of web-page hyperlinks, can be added to panoramic video. For example, one may want to provide information about a particular building in the scene. When the user clicks on the hotspot positioned over the building (the cursor changes shape as it moves across a hotspot), a text box, audio clip, or video clip might be provided explaining about the building. A hotspot can also be a jump to a website, or to another panoramic video. The latter is useful when exploring an area and having a choice of directions in which to proceed, such as at a street intersection or an entrance to an office from a hallway.
An iMove software viewer can view spherical video from a CD, DVD, hard disk, or over a network. Video can also be streamed or downloaded from the Internet using web browser plug-ins. Third-party application software can incorporate viewing of iMove imagery through the use of the iMove Software Development Kit. The iMove spherical camera architecture will also support real-time viewing of panoramic video, where the key post-production steps are done in real time, “just-in-time” prior to viewing.
As typically happens with a new and very different medium, applications of spherical imagery, and particularly spherical video, were originally very focused on a single need, but are now finding homes in an increasingly widening range of government agencies. Here are some of them.
Spherical video’s introduction in the government was in the secure community for reconnaissance purposes, and has recently been adopted by the Special Operation Command. In situations where there is only one opportunity to capture a scene or place, full spherical video is the only effective medium. Often times a satellite view is lacking in one or more ways. The resolution may be too low, the view may be limited by buildings or shadows, or there may be a requirement to see more of the scene than one can from above.
iMove cameras with integrated GPS receivers produce video with each frame location-stamped with GPS coordinates. The result can be used with a GIS package, such as Esri’s popular ArcView, to show an overhead or map view alongside the imagery. These side-by-side views – “bird’s eye” and “feet on the street” – form a very powerful tool in exploring or surveilling an unfamiliar area. The overhead view shows an updated “you are here” marker, as the user is “moving” through the space. The communication between the map and ground views works both ways. Should the user wish to move quickly to another spot on the map without “traveling” the streets to get there, clicking on that spot on the map will do the job. The linking software is supplied by IITRI (Illinois Institute of Technology Research Institute), an iMove development and marketing partner.
The U.S. Navy is beginning to use spherical video in several ways. For example, an iMove spherical video camera will be one of the sensors deployed on a mast of an experimental autonomous unmanned undersea vehicle (UUV) that can reconnoiter shoreline areas and other waterways that are too shallow for submarines. When deployed, such a UUV will exit from a submarine torpedo tube, travel to its programmed location, collect visual, radar, radio, and other data, and return to the submarine. While submarine personnel have the ability to communicate with the UUV during its mission, typically the UUV operates totally under its own control.
Obviously in a mission like this, where you don’t know just where to look, panoramic video is important. There is a problem with this type of surveillance, however. A UUV that operates almost at the water’s surface is subject to the roll and pitch of the ocean. A cylindrical panoramic camera is likely to miss key content at the extremes of the wave slap cycle. However, a fully spherical video camera, used in conjunction with an attitude sensor, handily saves the day. It captures the entire scene regardless of the attitude of the vehicle at any point in time, and attitude data enables software to reorient each frame to produce a continually stable view. The imagery can also be fed to on-board target recognition software, which can identify objects in the scene to be prosecuted or avoided.
Another recent innovation, the integration of a pan/tilt telephoto camera within a spherical camera, can be extremely valuable in this and many other panoramic applications. A remote human operator, or a computer, can direct the pan/tilt camera to objects in the spherical scene of potential interest, and the system will integrate the high-resolution image at the proper location within the spherical image. The result is the best of both worlds – a high-resolution view of objects of interest, presented in the context of their surroundings.
Vehicle situational awareness is another application of spherical video, whether or not the vehicle is manned. For example, the next-generation Canadian military armored vehicle program is using an iMove spherical video camera to give a tank commander a complete outside view, without having to open the hatch and be exposed to enemy fire. And since the entire spherical scene is sent in real time to a high-speed network within the tank, others in the tank, such as the driver, can view the surroundings independently of the commander. Also, “theater” data arriving at the tank from elsewhere in the battlefield can be overlaid on the spherical “background”. In the other direction, spherical views can be sent to command centers or other friendly vehicles.
While early applications have been in mission-critical military situations, spherical imagery is beginning to be used by other government agencies and even commercial users.
Security is an obvious application of spherical video in situations where an entire scene or area must be under continuous, 24-hour scrutiny. Spherical systems can feed motion detection, tracking, and recognition software, which can in turn raise alarms. Unlike human security personnel, these systems see everything all the time and never tire. When linked to motion detection software, they deliver what iMove calls 100 · 24 · 7ä – 100% area coverage 100% of the time.
iMove is in discussions with Homeland Security officials about the use of this type of system in airports. Imagine a series of cameras with overlapping views linked together. When a suspicious person is seen in any camera’s view, the operator would have the ability to back up the video – with the suspect in site the entire time, with the view being automatically handed from one camera to the preceding one, right back to where the suspect was dropped off curbside, where the vehicle license plate, cohorts, etc., are now in plain view of the operator.
Today architects universally use computer-aided-design (CAD) tools to design commercial buildings. Drawings from the computer models are supplied to the builders, inspectors, and customer. When “textures” (wall colors, carpets, outside window views, etc.) are added to these models, panoramic video can be automatically rendered to provide realistic walk-throughs that are fully under the viewer’s control.
NASA has used spherical video to familiarize astronauts with the training version of the space station that is located at Johnson Space Center, reducing contention for this one-of-a-kind facility.
Spherical video is being used in hospital emergency rooms, where the handling of a patient can be reviewed later, after the frenzy of the moment, to assess the flow of personnel, equipment, and material – what went right and what could be improved in the future. And patients who are about to undergo surgery can get a patients-eye view of entering and being in the operating room, looking around and getting information about different equipment that they’ll be encountering, all under their control, to help reassure them about the process.
U.S. federal and state web sites all describe and promote places of interest. The U.S. Park Service has experimented with spherical imagery of national parks to provide virtual access to these treasures for the handicapped and for those who would like to preview sites before visiting.
Panoramic imagery is even making its debut in the entertainment industry. Atom Films has made several short interactive panoramic videos, and James Cameron is using spherical video in a “Return to the Titanic” documentary. Anyone up for an interactive armchair trip to Mars?
In many government applications, it is proving very valuable to be able to see everything, and to travel within a site under viewer control. Spherical video is an excellent way to provide this capability. When integrated with complementary technologies, such as GIS map software, spherical imagery delivers even more powerful results.