Tammy Bearly

David Theobald

Tom Hobbs

James Zack

Disseminating Natural Diversity Information Using ArcView IMS: Design Issues and Technical Considerations

The goal of the Colorado Natural Diversity Information Source is to provide information to assist residents, decisions makers, planners, and professionals make informed decisions regarding the potential effects of residential development on wildlife and plant habitat in Colorado. The GIS-based application we have developed to meet this challenge has evolved from an AML-based program to its current deployment via ArcView Internet Map Server. We discuss in this paper some of the design considerations and technical problems we encountered, and solutions we developed, for disseminating biological information using Internet browsers and AV IMS.

The Colorado Natural Diversity Information Source (NDIS) is a project that aims to inform Colorado residents, decision makers, planners, and professionals about their wildlife and plant resources. This information is aimed to improve local land use planning and decision making. The core of the NDIS project is the System for Conservation Planning (SCoP) system. SCoP is a GIS-based decision support system that utilizes spatial data and analyses fine-tuned to provide a scientific, rationale basis for understanding the potential effects of residential development on wildlife habitat in Colorado (Hobbs et al. 1997). Typical queries of NDIS and SCoP users follow the two classic GIS questions--"what is here?" and "where do such-and-such conditions exist?" That is, our users want to know what wildlife is found in a particular location and where are the hot spots of biological richness. Since the data is fundamentally spatial in nature, maps are the principal format for delivery this information, but we also provide information in other forms as well: tables of statistics, lists of species, pictures, and documentation.

We felt that because local land use decisions are multiple and diffuse in space and time, traditional methods of offering information on biological conservation, e.g., through the Endangered Species Act process, were minimally effective. Instead, we sought to inject biological information directly into the local land use planning process in a timely, effective fashion. Therefore, the major task we faced was how to best disseminate this information.

We completed the prototype SCoP application using ArcInfo AML, in 1995, following a simple client-server architecture. We deployed the application to our users via a X-terminal installed in their office, connected via a 56K frame-relay circuit to provide a Telnet connection from the client X-term to the server UNIX workstation. The advantages of this configuration were that our users would not need an ArcInfo license, the interface and data updates--our application uses over 1GB of data--were centralized, and we could utilize the full set of functionality provided by ArcInfo. The disadvantages of this arrangement, which became apparent quickly, were the complexity of logging on (some X-term settings etc.), the requirement of X-term emulation, and the impediment that users had to learn how to navigate through a non-standard (for our users) windowing system and interface. Also, concurrent sessions were limited by the number of ArcInfo and GRID licenses available on the server side. We found ourselves in a situation we likened to "not being able to get the Cadillac out of the garage."

Then Internet browsers and the World Wide Web burst on the scene, and we had to decide when and how to jump aboard the fast-moving Internet train. We foresaw a number of advantages to disseminating our information via an Internet browser. These included: leveraging Internet resources and the rapidly developing browser technology (including efficient compression of large images), capitalizing on the common interface provided thus reducing the learning curve, maintaining a centralized application for data and interface updating, making GIS functionality available to anyone who had access to the internet, and maximizing user feedback so that the system and data improve with use.

Because we had previously built a prototype in AML, we had already identified a number of key design issues. First, we had very large data sets that required special algorithm development. The system needed to provide information for up to 500 species--we subsequently trimmed that number down to 225--especially a map of suitable habitat for each species. Recall that one of the two central questions our users ask is "what species occur in this area", and therefore a dynamic query was required. The GRID function COMBINE was ideal for this, but it balked at about 40 input grids, clearly below our needs. Eventually we developed a way to encode the unique combinations of all 225 species into a single grid by basically using a string of 1s and 0s (and ultimately programmed using Avenue bitMaps--fast!). This approach to our dynamic query required the ability to handle grid-based maps. In addition to providing information about many species, we needed to provide that information in many ways, resulting in a project containing 7 views, with up to 200 themes in some views.

One major decision was to choose between developing a MapObjects (MO)-, or ArcView Internet Map Server (IMS)-based application. A flash demo from ESRI-Denver quickly convinced us that most of the functionality we required was built into ArcView IMS. Furthermore, MO did not (does not?) have functions that support GRID-based analysis. AV IMS provided fewer opportunities and tools for user input, compared to our AML version, which initially presented a few problems. At the very least, this forced us to reduce the interface to core functionality.

A major difference in designing Internet applications is the necessity of a "stateless" server in the web client-server architecture. What this means, in contrast to traditional application development, is that the programmer cannot assume (because she won't know) "where" the user is. For instance, our users make a "what is here?" query by drawing a bounding rectangle, ArcView processes the request, and the result (a JPEG image) is sent back to the client. However, the user on the client-side will likely ask questions specific to their newly defined area of interest. Meanwhile, the server has impartially set itself back to a "stateless" mode. We describe below our technical solution to this problem.

Recall that one of the principle ways our users access information in the SCoP application is through a "what is here" query. We called this query process defining the area of interest (AOI). Frequently there are pre-defined AOIs that users want to use in querying, such as a planning basin, city or county boundary. Also, users want to define their own boundaries to find out "what is going on in their own backyard"-- a user defined AOI. In this section, we describe the technical problems we ran into, and the solutions we came up with. But first, we will briefly describe how AV IMS interacts with HTML.

Basically, AV IMS works as follows. A user clicks on the map and then a new map appears or some text is written out to a different window. Behind the scenes, the MapCafe applet receives the user input, then calls an IMS Avenue script, which in turn writes output to an HTML frame on the client web browser (Figure 1). Most applications using AV IMS that we have seen to date (especially early on) are fundamentally driven by MapCafe in this manner.

However, our application design required essentially the opposite as well (Figure 2). The user accesses information through standard HTML interface mechanisms (Info Frame), which drives a change in the map (Map Frame). Our user principally navigates through the hierarchical menu structure in the Info Frame to access the maps, statistics, pictures, and documentation (Figure 3). Therefore, we needed to have HTML drive the map as well as have the map drive HTML, which requires HTML to be able to make calls to IMS or MapCafe directly. For example, we wanted to allow the user to set an area of interest ("what is here") by defining a box or selecting from a list of pre-defined areas of interest. Also, we wanted to be able to dynamically change the view that was being displayed, through an HTML menu. We found that driving the map through HTML was difficult. Below we discuss several of the problems we faced and provide our solutions to these problems as well.

Users wanted to be able to select from a list of pre-defined areas of interest. Once the user selected a pre-defined AOI, the map was to zoom into that area, and all subsequent lists, analyses and statistics generated use the AOI as the analytical boundary (mask). We did not find an easy way to create a scrolling list inside the MapCafe window unless we did some low-level Java programming. We implemented a drop-down list of areas using an HTML Select tag. We store the map extent for each pre-defined area in the value field of the Select tag. The HTML code looks like this:

Once the user selects a pre-defined area, we call MapCafe to change its map extent via the external API call (Figure 4). The only tricky part of this call is that the extent must be comma delimited and quoted. The call looks like this:

We request this API call from a JavaScript function in our HTML page, which gets called when the user selects an area from the list. Here is the JavaScript function:

Our users also wanted to be able to define their own AOI by drawing a box or polygon on the map. The map would then zoom to this new map extent and subsequent queries and lists would use this new analytical boundary. We hired ESRI to help us develop a Java program to display the rectangle as it is defined (see Appendix 1 for code listing), zoom to the new map extent, create a unique user-defined AOI name, and pass the name and map extent to an IMS Avenue script, which then passed it on to HTML via a call to an IMS Avenue script. The MapCafe Applet starts the process when the user selects the user-defined area button from the MapCafe toolbar. It then calls a Java function to retrieve the extent of the user-defined area, then calls an Avenue script and specifies which HTML frame to write to, then Avenue writes HTML code to store the extent in the Title Frame (Figure 5). Note that the HTML code uses the HTML onLoad event to run a JavaScript function called userDefined to store the variables:

When this code is loaded no text is displayed, it simply calls the function. This frame should be cleared after this code is run, so that it is not run again if the user chooses the Refresh button on the browser.

The Java code was packaged into the MapCafe applet. Because the server is set to "stateless" mode after a request is made because it serves concurrent users, there was no easy way to store data particular to a user's current query on the server side. So, we store the extent and name of the area of interest on the client side, inside the HTML code in the Title Frame. We hide the variables inside the client browser using JavaScript global variables.

To create a user-defined AOI, the user clicks on a button on the MapCafe toolbar. This leads to an inconsistency in our interface design, because logically the user-defined AOI is just another type of AOI, and so we wanted to it to be another option on our drop down list of pre-defined areas of interest. However, because our AOI list is implemented in HTML, and HTML cannot talk directly to MapCafe, we had to add a new MapCafe button. In our next release we may try incorporating the AOI list into MapCafe -- we recently saw a ESRI demo project that looked promising in this regard.

We assume here that our user has set their area of interest (either pre-defined or user-defined), and now are interested in generating lists of species and statistics about them that are particular to their AOI. First, the user selects a list for a particular group of species (e.g., mammals). Next, we retrieve the AOI name using a JavaScript function from the client's Title Frame. Next, we call an AV IMS Avenue script, via the ESRIMap extension on the web server, to access ArcView tables and create an HTML page to display the list (Figure 6). We use the ESRIMap extension as the URL in the JavaScript function call, window.open(URL, frameName). This allows us to specify which frame to write output to. We call the Microsoft Web Server using this code:

The ESRIMap extension uses standard URL syntax. The "?" suffix indicates "key = value" pairs will follow, and each pair must be separated by "&". The first two parameters are required: the map name and the command. ArcView generates the map name when you create a web page for your view. It can be found in the MAPNAME parameter in the page’s APPLET tag. The command is the name of the Avenue script to run. It must have the prefix "AVINETMP.", but this is not included in this call. All of the value pairs above are shown as JavaScript variables for flexibility.

We had a difficult time with our initial design because we wanted to display 7 different views, with some views containing upwards of 200 themes. Originally, we had all of our themes in one huge view, and many themes were hidden. However, we quickly found that we could not serve a view this large. The limitations on the number of themes that can be served in a view stems not from the size of the themes, but the length of the theme names. MapCafe creates a URL with the name of each of the themes in the view, and some web servers have a limit to the length of a URL. Therefore, to get around this limit we had to break our data into 7 views and use shorter theme names.

Because we settled on multiple views, we wanted to be able to dynamically switch views, using HTML to specify the view to be displayed. Even more demanding, we wanted to be able to create an entirely new view containing specific themes. For example, we wanted to be able to create a specific view if the user was interested in a specific species such as American Elk - there are roughly 15 themes we could display for this species alone. We could not find a way to instruct MapCafe either to change views or create a new view from HTML. Because of project deadlines, we resorted to modifying an IMS Avenue routine that was used to generate a print preview page for HTML. Instead of displaying a new view, we simply generated a static image (i.e. JPEG image), turning off or on specific themes in one view with many themes. The drawbacks to this approach were that the images oftentimes are visually cluttered, and users cannot zoom, pan, or turn on or off themes, nor can they subtract or add new themes to a view. We have been tempted to change our design and invest effort into solving this technical problem even before we have released this application, especially after we saw a promising demo from ESRI that would allow a drop down menu of views from the MapCafe window.

We had a number of difficulties debugging our application that were particular to AV IMS, but some that stemmed from the client-server architecture. At first, we had a lot of trouble with the MapCafe errors: " Map server is busy" and "Map server no longer connected." Because a couple of our Avenue scripts were very time consuming (2 to 5 minute response time), we modified the AV IMS Avenue routine, AVINETMP.ServeMap and set the timeout to the maximum time limit. We continued to get the busy errors, and could not identify a pattern to their occurrence. After consulting with ESRI specialists, we discovered that we were eating up memory and never releasing it. ArcView does garbage collection, but only when it gets some time to do so. We added a call to av.PurgeObjects at the bottom of our scripts to force memory clean up, before AV went on to the next request. We found that the major memory hog was a large .dll file that was loaded into a variable frequently but never released. Instead, we made this a global variable and assigned it only once. These changes greatly stabilized our application and our sense of humor.

The possibilities and opportunities presented by the rapidly changing Internet technologies are not without their problems, but at least they are fun and challenging problems. The ability to disseminate clear, useful, and concise map-based information and analysis is truely powerful. Below we offer a few simple lessons learned.

1. Keep the application design simple, and don't let the GISer in you totally drive the design. Because we are familiar with maps and spatial information, we are predisposed to provide perhaps too many maps and too complicated technical analyses. Because users previously had very little information, they are apt to become overloaded with information quickly.
2. Resist the temptation to re-design before your product is released. Because technology is changing so rapidly, many of the technical problems and limitations we face today will become solved in 3 months. We tech-weanies are constantly tempted to invest "just a little more time" in trying out a new approach or gizmo. However, resist re-designing and changing the application before your users can provide feedback--otherwise, users will have nothing to give you feedback about!
3. Develop your application with the technological abilities that will exists 1 or 2 years down the road. It is well known that computer processing speeds essentially double every 1 to 2 years. Don't constrain your design or product based on current technological speeds and bandwidths, for they will surely improve in the near future!
4. Learn Java!

We wish to sincerely thank Dave Neufeld and Bart Killpack, of ESRI Denver, who expertly guided us through a number of tense times.

N.T. Hobbs, D.M. Theobald, J.A. Zack, T. Bearly, W.E. Riebsame, T. Shenk^. 1997. Forecasting Impacts of Land Use Change on Wildlife Habitat: Collaborative Development of an Interactive GIS for Conservation Planning. http://www.nrel.colostate.edu/scop/SCoPwww.html

Tammy Bearly

Scientific Programmer

Natural Resource Ecology Laboratory

Colorado State University

Fort Collins, CO 80521

Tammy@NREL.colostate.edu

David M. Theobald

Research Associate

Natural Resource Ecology Laboratory

Colorado State University

Fort Collins, CO 80521

Davet@NREL.colostate.edu

Tom Hobbs

Research Scientist

Natural Resource Ecology Laboratory

Colorado State University

Fort Collins, CO 80521

and

Life Scientist

Habitat Section

Colorado Division of Wildlife

Fort Collins, CO 80523

Nthobbs@NREL.colostate.edu

James Zack

Senior GIS Analyst

Geomega, Inc.

2995 Baseline Road, Suite 202

Boulder, CO 80303

Geomega@pcisys.net