Implementing ArcGIS 8.1 for Redistricting
Bob Andrew, GIS System Architect for DynCorp Information Systems on the
project
with Larry Boden, Manager of Technical Operations at Esri Implementation
Services
DynCorp Information Systems, LLC (as system integrator) and Environmental
Systems Resources Inc (Esri as developer) designed a major upgrade at the
Department of Justice (DOJ). This provides a richer, faster, more manageable
system for DOJ to perform its responsibilities under the Voting Rights
Act. The paper describes "lessons learned" in going from UNIX workstations
to Windows NT desktop, from ArcInfo 7 with coverages and shapefiles to
ArcInfo 8 and ArcSDE with Oracle database.
Disclaimer
The purpose of this paper is to illustrate improvements for end-users in
GIS technology that has moved from workstation to client-server and in
spatial data design from shapefiles and coverages to a geodatabase. The
project used to illustrate this at the time of publishing this paper was
still in the final stages of development, so functionality discussed and
illustrated may not be in the final product. The 2000 decennial census
was a major business driver for all providers of redistricting tools to
closely examine their processes and software - this is just one of three
stories you will hear in this afternoon's session entitled "Redistricting
Using Census Data".
Redistricting Review
Under Section 5 of the Voting Rights Act of 1965, jurisdictions in 17
16 states are required to obtain preclearance
for any voting change, including redistricting plans that they enact. One
option for obtaining such preclearance is to submit it to the Department
of Justice for its review. In those instances, DOJ is required to respond
within sixty days of receiving any submission. The geographic data
contained in the submissions usually include at least a paper map of the
proposed redistricting plan. In addition, the Department allows a jurisdiction
to submit an electronic list , or "table of equivalencies," of county,
census tract, block group, or block and the electoral district to which
each is assigned. DOJ Plan Preparation staff have a role to ensure a proposed
map and the "equivalency table" match up without geographic slivers or
numeric errors. Once the geographic assignments and resulting GIS map are
verified as being accurate, the proposed redistricting plan is analyzed
to determine whether it meets the standards of federal law.
Former GIS System
A decade ago, DOJ Civil Rights implemented Unix-based system to perform
most of the necessary mapping and statistical steps required to analyze
any proposed voting district boundary changes arising from the 1990 Census.
This used individual workstations and a single server to store all
plans for redistricting, plus Census attribute and geographic data. In
the mid-1990's the same application was upgraded with some faster Unix
workstations and additional servers.
Reasons to Change
The existing system was very slow to re-draw large plans, as well
as it required specialized GIS staff to maintain the software and hardware.
Even small changes to the application required substantial effort. The
hardware was also close to or beyond its life expectancy.
Moving to an interface running on a standard DOJ
desktop system would avoid users also needing specialized workstations,
saving in capital and maintenance. Note DynCorp
I.S. staff were already onsite working at DOJ implementing an upgrade of
their Office Automation Systems.
Following an internal pilot on the beta of ArcGIS 8.0, DOJ contracted
with DynCorp I.S. to work with Esri Implementation Services to develop
a user interface, this time using ArcGIS 8.1, which in 2000 was just going
into beta. DOJ
committing to develop with the core new technology enabled the development
staff to take advantage of new features planned for ArcGIS 8.1 and for
DOJ to prolong the shelf life of their tools.
Design Approach
DOJ's attorneys, analysts and onsite GIS staff had already determined a
core list of requirements for the interface design, based on review of
their existing system, plus functionality that was available in 1999 through
commercially available redistricting software. A key objective was to use
core technology wherever possible: so DOJ Civil Rights Districting
Extension (DX), a custom user interface to ArcGIS was developed.
What was not specified in a requirements list built from a user-interface
perspective were vital data modeling steps - these impacted both
schedule and the design. This project used the strict discipline of an
integrated geodatabase model, rather than separate shapefiles and coverage
models. This approach is more sustainable over time in the evolving process
of redistricting.
Project Schedule
Esri and DynCorp planned to divide these requirements into three phases:
a client-only release; a client-server release that worked with 1990 data;
and a release to work with Census 2000 data. In third quarter 2000, right after reviewing
a conceptual design prototype at the Esri International User Conference,
the original requirements were broken down into a detailed design matrix
spreadsheet of some 170 implementation tasks. This spreadsheet was used
monthly throughout to manage design progress against the tasks and phases.
System Platform
Prior to DynCorp taking on the system integration
responsibilities, DOJ had contracted with Dave
Peters, Manager of Esri Systems Integration, to conduct a strategic
plan for system architecture, using bench-mark performance data of Unix
and Windows/Intel platforms to aid in sizing servers for ArcSDE/Oracle.
DOJ chose to stay with the proven
performance
of Unix, and now have two mid-range Unix servers and
a variety of mass storage, backup and archiving devices all dedicated to
aspects of Civil Rights GIS (which is broader than just the Voting Rights
usage featured in this paper).
For client hardware, ArcGIS 8.1 is deployed on the same high-end Windows
NT desktops as the rest of the office automation software, thus eliminating
DOJ's need for any special GIS workstations Using Windows NT in a managed
network like DOJ's is a challenge, because of the complexity of Administrator
rights vs. local user rights, and because the Esri design team were already
all on Windows 2000. The impact is more extensive onsite testing required
within the office automation environment.
The Districting Extension (DX for short) is principally client-server.
Given the magnitude of the Census 2000 data modeling and client-server
ArcGIS/Oracle design effort, DynCorp recommended against going with a web
implementation at this stage, such as ArcIMS 3.x, but the extension does
produce XML metadata of redistricting plans and a JPEG of the plan, which
could be used within ArcCatalog and
a simple intranet web service. (In hindsight, given that ArcIMS 3.1 was
only just released June 2001, sticking solely with ArcGIS was still the
right choice for this project at this time).
Data Modeling
The commitment to geodatabase also required disciplined modeling of all the
data structures and processes required in districting. One good example that can
be visually demonstrated is the data modeling required to support the four
demographic levels of geographic resolution: county/tract/block group/block. The
DX has tools to be able to drill down to selected level of resolution. For
example, to select a district comprised of a number of tracts plus several block
groups. The tools can show in more detail e.g. just the block groups of a
selected tract, while leaving all the other geography in a districting plan at
the tract level. This has value for performance reasons (not having display
every tract at the block group level), and is less visually cluttered. The data
model also supports the "join" necessary to display the selected
district with a single boundary and display the associated statistics.
Database Design
Oracle 8i was the chosen platform for this project. The DX data model is
implemented as state-by-state data storage in Oracle tables, supporting
all key Plan Properties required of all submissions as published in the
Federal Register in compliance with OMB guidelines. Census 2000 geographic
and population data is also stored using this same data model, so although
content is identical to that published by Census, for performance reasons
it is structured to suit the purposes of DOJ. With the completion of this
Voting Rights application, on-site GIS staff will be able to take advantage
of the same core data for other Civil Rights applications.
Application Performance
Since GIS is inherently demanding on bandwidth, one part of the design
effort concentrated on removing the need to re-draw the screen with
changes in district boundaries, even minor ones.
Techniques were developed to highlight selected geography prior to committing
a boundary move. Similarly, to avoid unnecessary server processing, a preview
of statistical changes before the commit step. In an effort to further
reduce network traffic, an endeavor was made to do all DX analysis work
in the client-side personal geodatabase, but with several key steps requiring
server-side polygon joins, for performance reasons this decision was revisited
and all plans are now worked from the Oracle geodatabase on the server.
Given the uncertain fate of personal geodatabase in its present form as
ArcGIS looks ahead to Release 9, this again was probably a prudent move.
It is fair to say that the team devoted to this project had a major impact
on the functionality that made it into ArcGIS 8.1 and ArcSDE 8.1
User Interface
Considerable effort was applied to ensure the end-user had intuitive access
to the entire districting process and workflow. Techniques like "dockable
windows" ensure that windows plan layers, plan statistics and plan maps
are consistently tiled, with users able to set the relative size of each
window. Custom floating toolbars and custom ArcMap icons are used
for many of the most commonly repeated tasks. Use of icons even extends
to a customization of ArcCatalog where colored flags indicate the status
of a particular plan - checked out for review, stored on server, on local
hard drive etc.
In the years following the release of the decennial
census data DOJ expects to receives over 3000 redistrictings plans to analyze
under Section 5.. Managing the submissions can be almost as daunting of
a task as actually reviewing them. For this, ArcCatalog was extended to
support the redistricting plan management system.
The DX uses the power of ArcCatalog’s metadata
tool to track plans. ArcCatalog creates metadata for each plan in an Extensible
Markup Language (XML) file stored locally or on a Web server depending
on if the plan is being edited locally or resides on the server for the
user to retrieve for editing. The user also has the ability to create a
new version of a plan to edit. In ArcCatalog, users can
search for, preview,
and explore plans. Thumbnails of individual plans are created to provide
the user a quick overview of the plan.
Usability principles are still being applied in the final acceptance
stages to important details like color palette, line weight. The DX will
have the agreed set as defaults, with ability by on-site GIS staff to update
these as required, as well as for users to define and store some personal
preferences.
Modular Design
Core ArcGIS provides a rich set of ArcTools for use in such user interfaces.
The entire DX interface is written as 28 modules (executables and DLLs)
that enable parallel development and independent testing. Many of these
modules are devoted to ensuring consistent use by whomever is assigned
to do plan creation, editing analysis or review.
Concluding Remarks
The lessons learned are not new nor unique, and given that the interface
is still in development we cannot fully claim that these are good lessons,
but what we can say, so far, is that data modeling is of great significance,
that well-defined requirements are essential, that the modular approach
to development is important, and most of all, that methodical project planning
and testing is a required ingredient.
Acknowledgements
Thanks to U.S. DOJ Civil Rights for the honor and opportunity to develop
a new Redistricting application for the Voting Rights section.
Voting Rights: Bob Berman and Bob Kengle. Admin: Doug
Crain and Nancy Sweesy
Onsite GIS: Mario Lopez-Gomez with Jonathan Brooks, Lou Summers,
Sharon Thomas
Esri Account Manager: Todd Rogers, now Civilian Federal Manager
(& DOJ Account Rep).
Development: Larry Boden with Ryan Bae, Justin Fan, Craig Ferris,
Kevin Kelly, Mike Redpath
OnSite Support: Steve Ciampichini, Fatima Awad, Jim Anderson,
Josh Blue, Jim Correll, Spencer Crim, Sean Cononie
DynCorp Information Systems Management: Doug Cheek with Kevin
Bell, Soo-Ung Kim, Bill Pratt, Lucian Russell
Robert D (Bob) Andrew
(now GIS Team Leader
with ICF Consulting Inc)