Enterprise GIS: Web-Based Map Production with ArcIMS and ArcInfo AML

GIS staff can spend significant time making maps for internal and external customers. This can take time away from other important functions such as maintaining and enhancing data. Moreover, a backlog of mapmaking jobs can increase turnaround time to days or weeks. Jackson County, Oregon, and GeoNorth, LLC, will talk about the managerial and technical aspects of a Web-based system they developed to automate and streamline Jackson County's map production process. This solution enables non-technical users to drive AML scripts from a Web form. It utilizes ArcIMS or , ColdFusion, Perl, and a shared database model.

Project History and Problem Definition

Jackson County is the sixth most populous county in the State of Oregon. Jackson County covers more than 2,801 square miles ­ making it equal in size to the state of Delaware. The current population is 174,550.

While Jackson County GIS Services has served both residents and internal departments with several interactive mapping applications over the last few years at their registered domain web site “SmartMap.org”, this has not fulfilled the consistent demand both internally and externally for large, high-quality cartographic maps.

The cost of a cartographic map generation at the county is $75 for all services (media, staff time, etc.) It is the same price whether it’s an 8 ½” by 11” black and white vector map or a 35” by 48” color map with aerial photos.>Unlike the interactive mapping sites, the order fulfillment doesn’t happen in seconds or even minutes. The map request gets “queued” with a GIS technician for production at a later time or even a later date.

In most GIS shops, programmers have written Avenue, AML, or VB/VBA scripts to automate ArcView or ArcInfo map creation using page layout templates and pre-defined map or legend rendering. For some common map requests, a GIS technician may need to tie up their machine resources running a previously developed script and template that requires their interaction to run and print, but no extra interaction for page layout or cartographic development. The result is a hardcopy map that cannot be reproduced without re-running the script with the same user input or retaining the output in some re-usable format (graphics file, postscript file, etc.) This saved copy would only display the GIS data as it existed at that moment.

If the user wants a digital version, then typically the output format has to be converted to something they can use (like a graphics file, assuming they have ArcInfo, or a PDF, which might require each GIS workstation to have a conversion utility like Alchemy or some of the newer output options with ArcInfo 8.)The output is typically more than will fit on a floppy disk, so a CD-ROM writable media must be sacrificed. E-mail is an alternative, but the size can often be troublesome with mail server constraints.

Project Goal

The project goal for the “Maps on Demand” project was to

• Enable non-technical consumers to make their own maps through the Internet
• Allow for maps to be downloaded in the format of their choice OR printed on-site

The achievement of this goal required an easy-to-use, light-weight front-end interface that would work in Internet Explorer or Netscape. It also required an output format that could be viewed in the browser or in a document format that was suitable for printing. The output formats selected for the initial implementation were PNG and JPEG for the browser-viewable output format and PDF for the document format.

Project Solution

The solution chosen for this project was the existing smartmap.org front-end powered by MapOptix. MapOptix is a 3-year old IMS product that utilizes both MO/IMS and ArcIMS for Internet maps and a thin client front-end using Cold Fusion to link GIS and tabular Databases. The back-end used to produce printable cartographic maps was Workstation, and Image Alchemy to create PDF output. GeoNorth developed and deployed the architecture, a web-based front-end interface for submitting map requests, and a web-based interface for map job administration. All three of these will are detailed in order.

Architecture

The architecture used for the Maps on Demand project was a combination of Internet, GIS, and Database technologies. All communications and configurations between the back-end components are implemented using shared databases. There are two primary databases. One is the MapOptix “settings” database used for all of the map images (layer rendering, layer ordering/thresholds, etc.) and one is the Maps on Demand “jobs” database, used for all of the actual map requests.

The Maps on Demand “jobs” database is primarily a “queue” of jobs (optionally processed in a “batch”) that are fulfilled by one or more back-end processes on one or more back-end servers. The jobs are all based off of a pre-defined “process” definition, which constrains the input and the location of the software that is “executed” on the back-end. All jobs, their arguments, their current “state”, and all state changes are logged as the jobs is executed.

Below is an example of how this architecture is implemented as a job goes from creation to completion. The numbers at each step are detailed following the diagram.

1. The job request is submitted by the user
2. A back-end process polls the database for new jobs
3. The job is “locked” by a back-end process
4. The AML is created based on the job request and the settings database
5. The AML is run
6. The AML output is converted to the requested, final output
7. The final output is delivered to the user via. FTP or E-mail, OR it awaits approval for printing

Front-End Web Interface

Once the user has an extent, map scale, and layer selection that they are satisfied with, they click on the “Make Map” button to display the below interface.

The user has a preview of the map exactly as it will look on the back-end, and can enter their own title information, the page size they would like, the format of the output they would like, and the method of delivery. The below screenshots show the immediate response from the job submission (updated every few seconds with JavaScript to show the job state transitions) and the job submission output.

Job Management Web Interface

The job management web interface controls all aspects of the application, including:

• The users logins and the user general information
• The process, arguments, and argument types
• The history of all jobs by user

As you can see in the below screenshots and descriptions, this architecture could apply to any command-line based process that a user wanted to run from the web that resulted in printable or digital output.

User Management

User management includes username/password, general contact information (used to populate the “popup” map job submission form,) administrative permissions, and the methods of delivery the user can request. In the case of e-mail, there is a “cap” on the size of file that can be sent. On-site print requests can be disabled or enabled. Any immediate download can be HTTP or FTP, and can reside on remote servers.

Process Management

The definition of a process includes the process executable (in this case, a Perl script,) the name of the process, and the arguments to the process. Argument order is the order sent to the command-line on the back-end. Arguments include the argument type, description, “default” value, and a flag to specify if the argument is required or not.

Argument Type Management

Each argument has a “type”. This includes standard input field types (checkbox, text, text area) as well as the type of argument it is (integer, number.) There is a lookup definition interface for select boxes (used here for map size and layout.)

Job History

Each job keeps track of the arguments sent to it and its state transitions. If the MapOptix settings and job definition have not changed, a job can be “re-submit” for creation. The job detail and re-submission (with the process running in the foreground) are shown in the below screenshots.

Lessons Learned

There were some definite lessons learned during the course of developing this technology. Many of these lessons have probably been published by programmers that have dealt with mainframe or service-level software for the last 20 or even 30 years. They are listed below

1. Be conscious of scalability during design. Job locking across multiple processes and multiple hosts was something we didn’t account for when we started developing this architecture three years ago. It became necessary when scaling up to alter the back-end processing and the database design to first “lock” the local job processing using exclusive locks through the general interface, then to “lock” the job to the host itself. This avoided multiple processes on the same machine blowing over each other, and also avoided multiple hosts processing the same job.
2. Only use database-specific objects if it’s a necessity. I’m sure the ArcSDE software developers might agree with the opinion that developing cross-database solutions becomes trickier when you start to employ database objects like relationships, triggers, stored procedures, etc. While it can save you tons of application code in your software, it will cost you hours and hours of “porting” time when you start to realize that some databases just don’t support the same things. For example, SQL Server 7 employs triggers to perform cascading deletions instead of basic relationship definition, which when combined with the triggers in SQL Server 7 actually cause more headaches. Microsoft Access is another one of those trickier database interfaces. It actually requires the use of ADOX instead of basic SQL to modify the relationship properties such as cascading deletions. Over time, GeoNorth has found that the easiest databases to “port” to other databases are those that have no “special” requirements from the database server or drivers. Another somewhat database-specific object that always seems to cause trouble are the large objects (BLOB, CLOB, Memo) and their “supported” interfaces from platform to platform (Oracle/ADO programmers are probably familiar with the Long/CLOB/BLOB tradeoffs.)
3. Always provide a job “reset” button. With software this complex that utilizes so many different software components, the “reset” or “restart job” button was one of the more useful recovery tools implemented. While preventative measures could be taken in many places, there was always the possibility of an ArcInfo or ArcPress license being in use, or a job that had to be stopped due to disk space shortage, or an FTP delivery that didn’t go through for the user due to a network anomaly. Because all of the job request information was retained in the database, an administrator could re-submit any of the jobs at any time as long as the underlying process definition or AML template was not changed.
4. Log Everything. Many of us who have paid our dues as system administrators (or are still paying them) can appreciate the value of a meaningful log entry. The software not only logs to files on the back-end, but logs its progress to the job database for every state change and every detail of the process being run. This makes the trouble-shooting of exceptional conditions much easier on the operator, and also provides rich information as to who requested what when and how long it took. This logging could also be used for running statistics on the general usage of map request and whether another process or process server is needed.
5. Make all aesthetics dynamic. One of the primary advantages to this architecture is that the color and layout of both the front-end and back-end can be configured through some sort of template, argument, or configuration storage. This included even the size, layout, and location of the pop-up window, as well as all colors/fonts in both the front and back-end page layouts.

Author Information

Nick vanVeldhuisen
Portland Office Technical Manager
GeoNorth, LLC.
921 SW Washington
Suite 777
Portland, OR, 97205
P: (503) 827-0827
F: (503) 827-0735
E: nickv@geonorth.com
I: http://www.geonorth.com