James Heald
Developing a National GIS Data Layer for Agriculture
One County at a Time
The
USDA Farm Service Agency (FSA), along with partner agencies Rural Development
and the Natural Resource Conservation Service (NRCS), is in the process of implementing
desktop GIS at over 2500 field service center locations nationwide. Ultimately, the GIS resources for the Agency
will be managed in a distributed database environment.
The
most critical component in the successful implementation of GIS for FSA is
development of the Common Land Unit (CLU) data layer. The Common Land Unit layer will ultimately include all farm
fields, range land, and pasture land in the United States. Data is currently maintained at the service
centers with annotations on hardcopy aerial photography. In conjunction with digital imagery and
other data, FSA will use the Common Land Unit to manage Farm Service Programs,
monitor compliance, and respond to natural disasters, among other tasks.
Due to funding constraints and the need to build support for GIS within FSA, we have had to start slowly, building momentum and developing data and pilot applications in parallel to show results quickly. Phase One of the data development was the creation of 12 digitizing centers at field office locations in seven states. These offices are developing the CLU data layer with on-screen digitizing using Arcview in a Windows NT environment. Customized digitizing and quality control extensions were written for the project.
Pending
budget approval, future developments will include a program of outsourcing data
production, developing new digitizing centers, upgrading the software
environment, and distributing the completed data sets to field service centers
with training and tools to maintain the data.
This presentation will discuss the current progress for this effort,
methodology, standards, lessons learned, and future directions.
Background
The
USDA Farm Service Agency (FSA), along with partner agencies Rural Development
and the Natural Resource Conservation Service, is in the process of
implementing desktop GIS at over 2500 field service center locations
nationwide. Ultimately, the GIS
resources for the Agency will be managed in a distributed database environment.
As with many public agencies, “the majority of FSA’s business data is
geo-spatial in nature, or referenced to geographic locations (i.e., land
records, field locations, soil types, etc.).
The ability to manage this data efficiently and effectively as we move
into the next century is essential.”[i] A report done in 1996 estimated that roughly
one-third of FSA’s business directly involved using and maintaining maps and
geospatial information. In those areas,
it was estimated that FSA could save approximately 42 percent of its costs by
integrating GIS into business processes.
The
most critical component in the successful implementation of GIS for FSA is
development of the Common Land Unit (CLU) data layer. The Common Land Unit layer will ultimately include all farm
fields, range land, and pasture land in the United States. Data is currently maintained at the service
centers with annotations on hardcopy aerial photography. In conjunction with digital imagery and
other data, FSA will use the Common Land Unit to manage Farm Service Programs,
monitor compliance, and respond to natural disasters, among other tasks.
Due
to funding constraints and the need to build support for GIS within FSA, we
have had to start slowly, building momentum and developing data and pilot
applications in parallel to show results quickly. Phase One of the data development was the creation of 12
digitizing centers at field office locations in seven states. These offices are developing the CLU data
layer with on-screen digitizing using Arcview in a Windows NT environment. FSA wrote customized digitizing and quality
control extensions for the project.
Additional extensions to support FSA program needs are being developed
as needed.
Challenges
FSA
faces a number of challenges in developing the Common Land Unit layer. First is the sheer size of the task. FSA needs to map all of the farm fields in
the country, or at least all the fields involved in agency programs. Second, the data is constantly
changing. Therefore, it needs to be put
in a maintenance program as soon as initial digitizing is completed. Also, FSA needs to be able to maintain
tabular and geospatial time-series information. Lastly, we have security and privacy issues. Farm and disaster payments are directly tied
to acreage and the crops reported by producers.
The Farm Service Agency (FSA) currently maintains records for about 2 million Farms, comprised of about 16 million fields. Map data is maintained manually on hard copy aerial photographs at more than 2500 local service centers. Tabular data is currently maintained on a System 36 mini-computer at these same local offices. Some of the tabular data is uploaded to a central mainframe computer. These legacy systems employ thousands of lines of Cobol code and utilize hundreds of VSAM files.
Because map data is manually maintained at a large number of sites, the quality of source material available for transfer to GIS varies considerably. Historically, as new NAPP photography became available, new hard copy photography was prepared and printed by FSA’s Aerial Photography Field Office (APFO) in Salt Lake City and shipped to the service centers. Employees at the service center were then tasked with transferring their line work and annotations from the old maps to the new maps. Depending on the county, this has been a massive job. For example, Kings County, California has over 1150 map sheets. Given other work demands, the maps have not always been transferred to the latest photography.
Standards for maintaining the hard copy maps exist, but the standards are somewhat flexible. In part, this has probably been done to alleviate some of the burden of maintaining the maps. However, local variations in source material will make it more challenging to contract out the digitizing.
In addition, because the maps are critical to conducting business at the service centers, the staff is very reluctant to part with them, particularly for more than a couple of days. FSA is currently testing out a process for scanning the source documents at APFO. Scanning can be completed within several days and the original source materials can be returned to the service center. Scans would be made available to contractors or digitizing centers in either hard copy or digital format.
Farm and field boundaries change
constantly. Tree stands are cleared to
create new common land units. Farmland
is sold and subdivided for development and common land units disappear. Ownership and producer relationships
change. Depending on the growing
season, crop data and planting patterns change one or more times a year.
The
Minnesota State office has estimated that, in a typical county, there are
between 1200 and 2400 map changes per year.
Workload for these changes is not spread out conveniently throughout the
year, but is tied to program signup deadlines, annual acreage reporting, and
compliance verification cycles.
As a result, FSA cannot simply digitize the
field information for the County and let the data sit on a shelf somewhere
until computer hardware, GIS software, and applications are delivered to the
counties. Maintenance is crucial and
development of the data layer must coincide with delivery of hardware, software
and applications.
In order to properly administer farm
programs, FSA is required to maintain time-series information on both attribute
and geospatial data. Eligibility for
certain farm programs and disaster payments is dependent upon the historical
use of land parcels. For example, land
is not eligible to be enrolled in the Conservation Reserve Program unless it
has been farmed in 2 of the 5 most recent crop seasons. Disaster programs are sometimes initiated
for past years.
FSA is responsible for billions of dollars
in loans, farm payments, and disaster assistance. In 1999, Market Loss Assistance Payments totaled over $5.4
billion to over 1 million producers. In
1998, Crop Loss Disaster Assistance totaled over $1.9 billion. These payments
are directly tied to what happens on the land and acreage reported by
producers. Therefore, the GIS data
layer needs to have an acceptable level of accuracy. For many years, FSA has determined field acreage and producers
have accepted these measurements. There
is some discomfort within the agency that GIS acreages will deviate too much
from the historical acreage and cause problems with producers. One way to ease this problem is to work with
producers to make sure that they agree with where boundary lines are drawn. Currently, digitizing specifications require
all boundary lines to be within 3 pixels of a feature on the DOQ. In addition, any changes to the data need
to meet strict auditing requirements.
When the system is fully implemented, FSA needs to be able to track who
made changes, when they were made, and why they were made. FSA also needs to protect the
confidentiality of any sensitive tabular data joined to the GIS data layer.
Creating and maintaining a farm field layer
for the entire country is an expensive proposition. Estimates have ranged from $16 million to $45 million just to
create the initial data layer, excluding preparation and distribution costs for
managing DOQ imagery. Because this
particular data layer becomes obsolete quickly, software and tools need to be
deployed to a large number of sites to ensure maintenance. While there appears to be broad agreement
that this layer is critical to redesigning farm program delivery, very little
of this has been included in the FSA budget up to this point.
Creating the Common Land Unit Layer
FSA
and its partner agency NRCS have been struggling with implementing the Common
Land Unit layer for almost a decade.
The first pilot program to integrate GIS in the service center workflow
began in 1991. This and subsequent
pilot programs verified that implementing GIS in local service centers would
dramatically reduce workload and improve the quality and consistency of
delivered services. Finally, in 1998
nine pilot sites were equipped as part of a Service Center Implementation
program and Common Land Unit layers were developed for these sites. FSA contracted out most of the digitizing
for the nine sites to private contractors.
At the
end of FY 1998, funding was obtained to establish 12 digitizing centers in
seven states. The State Directors from
Kansas, Minnesota, Nebraska, and Missouri lobbied the Farm Program
Administrator to get GIS implemented in their states and proposed setting up
two digitizing centers in each state.
The Farm Program Administrator agreed to implement the program, but
added an eastern state, Virginia, and a tobacco state, Kentucky. Oklahoma was later added to the program
because of issues involving the Cherokee nation. The State offices selected the digitizing sites and most were
established in local service centers.
These
sites were equipped with Windows NT servers and between one and four additional
workstations. The GIS software chosen
for use at the centers was Arcview.
Hardware and software were ordered in December 1998. Hardware and software were installed starting in March 1999. Training began in March and the first site
was operational in April. Deployment
and training continued through May. A
thirteenth site (Callaway County, Missouri) was added in January 2000. Callaway County had been part of an earlier
GIS pilot program and had been doing digitizing and other GIS projects using
Arc/Info in a Unix environment.
The initial proposal to develop the Common
Land Unit data layer outlined a process to transfer data from the hard copy
maps using heads-up (on screen) digitizing on a Digital Orthophoto Quad
base. FSA’s Aerial Photography Field
Office is responsible for delivering the DOQ to the digitizing centers. FSA made a policy decision to mosaic the DOQ
to create seamless County images, because image quality and edge matching
problems made the raw DOQ unacceptable as a base map for this project. As a result, APFO checks every single DOQQ
(USGS only checks 10 percent). APFO has found that approximately 8 percent of
the quarter quads have problems: for example, they don’t meet National Map
Accuracy Standards, they are projected in the wrong UTM zone, or they contain
the wrong photograph. The mosaics are
tone balanced and most common DOQ edge-matching problems are eliminated. Seam lines are also adjusted so seams do not
divide farm fields. Currently, APFO is
delivering full resolution TIF files tiled by quad boundary and a County Mosaic
in MrSid compressed format. Digitizing
is only done using the full resolution images.
The MrSid files are used mostly for quick viewing of data.
With current staffing levels and hardware,
APFO is only able to mosaic about 200-250 counties per year. In order to meet development targets, this
production level will need to be at least doubled over the next couple of
years.
With over 2500 county offices nationwide,
FSA has had to establish data development priorities for each state. Each State Office, in cooperation with our
partner agency NRCS, developed a priority list. Factors considered by FSA in developing the list were; 1)
workload, 2) availability of DOQ, 3) agricultural land and crop production
levels, 4) expertise and adaptability of the county office staff, and 5)
availability of Digital Soil data. The
digitizing centers began by working on data for their own states. The next phase of the project, starting this
summer, will be to begin digitizing top priority counties from other states.
As of May 1, 2000, initial digitizing has
been completed for 106 counties. We
estimate the digitizing centers will complete another 8-10 counties every
month. At the present time, only about
20 counties, in addition to the digitizing centers, have the hardware and
software to maintain their land units.
Arcview,
out of the box, includes most of the functions needed for creating data. However, it was necessary to extend and
customize the interface to reduce manual setup time and ensure data and procedural
consistency between the sites. The
funding obtained for this project did not include a budget for software
development. FSA staff did the original
Avenue programming to customize tools for the digitizing project.
Features of the original digitizing
extension included the following features:
q
Floating tool bar
q
File creation with defined attributes
q
Scale limitation on editing (1:4800 or
smaller)
q
Feature editing with some topological
checking
q
Automatic acreage calculation when features
are edited
q
Tools for creating inclusions (polygon
subtraction) and to streamline the process of combining polygons
q
Attribute editing through dialog boxes
Subsequent updates to the digitizing tools
corrected problems, particularly to reduce problems with segmentation
violations and file corruption, and added new functionality, including:
q
Integrated Quality Control tools
q
Checks Overlaps, Voids, Slivers, Multi-part
polygons
q
Automated Map Layout creation
q
Multi-attribute polygon labeling
q
A simplified search routine
q
Farm, Tract, and Field number search
q
Public Land Survey Township, Range, Section
search
Again, most of this functionality was developed in-house. However, some of the functionality was
adapted from user scripts downloaded from the Esri site. In particular, the shape check and
correction program developed by Andrew Williamson has been a lifesaver on
numerous occasions.
Where the files are being maintained, most
sites are currently using FSA’s standard digitizing extension. A more user-friendly maintenance tool has
been developed and is undergoing final testing. This tool is very similar to the digitizing extension, but
includes stronger data validation and some additional tools. The maintenance tool also forces attribute
data entry when polygons are added or edited.
Ensuring the Transition to GIS
Over the last several months, it has become
increasingly apparent that the only way to ensure a successful transition to
GIS is to eliminate any need for the hard copy maps. Among several problems with initial data modeling efforts was the
failure to include all of the data elements in the CLU table that are currently
being captured on the hard copy maps.
Tools are currently in development to allow service centers to capture these
additional data items and link them to the basic CLU data file. The bottom line is that no matter how
inherently useful the data, applications for making use of the data are
necessary to ensure use.
One of the major hurdles to eliminating the
hard copy maps involves the farm program compliance process. Every growing season, producers report their
growing intentions (crops and acreage) to FSA at the field and sub-field
level. At the peak of the growing
season, most county offices have 35mm aerial color photography flown. They receive this photography as
slides. Computer programs on the System
36 legacy system generate a random sample of farms, which are to be checked for
compliance. The appropriate slides are
inserted into a projector attached to a planimeter and manually rectified to
the hard copy maps. Field and sub-field
boundaries are measured and compared to the reported acreage.
We have distributed Arcview Image Analysis
software to four of the digitizing centers and they are rectifying digital
copies of the slides to the DOQ. One
county in Minnesota did a test by redoing their compliance from 1999 using
rectified 35mm slides and the digitizing tools to measure compliance with farm
programs. Measurement differences
between GIS and the planimeter were within an acceptable tolerance (generally
less than 5 percent) and the Compliance process took about 15 days (including
slide rectification) instead of 30 days using current manual methods.
Minnesota is aggressively pursuing a
strategy of doing digital compliance in all counties where hardware, software
and data are available. That could be
as many as 20-30 counties in FY 2000, although funding delays may significantly
reduce that number. We are also
planning to pilot a program of contracting out the rectification and mosaicking
of 35mm slides to create countywide seamless images that match the county DOQ
mosaics created by APFO.
FSA is currently developing a more automated
process for compliance, which will include sharing data with the System 36 to
eliminate as much data entry as possible.
This process will take the Farm Sample list generated by the System 36
and automatically select Common Land Units, based on the farm number, and copy
them to a compliance layer. Where
necessary, sub-fields will be added to the layer based on the 35mm
photography. The acreage will be
generated for each field and sub-field automatically by the tools.
Another data set from the System 36 legacy
system, containing the reported acreage and crop information will be linked to
this compliance layer, allowing comparison of reported and actual acreage. Visual interpretation will be needed to
verify that the reported and actual crop types are the same. The generated acreage will then be fed back
into the System 36. Where farms are out
of compliance, a map will be generated using a predefined template and a
notification letter will be generated using existing software. FSA is also planning to cut out countywide
sections of Landsat TM data to send to our compliance sites during the growing
season to see how this might aid them.
Use of these new tools and digital datasets should further reduce the
amount of time taken by the compliance process, while improving consistency,
and creating a permanent digital record of compliance activities.
It is important to note here that the System
36 is a legacy environment and that any applications developed to tap into its
data resources are interim applications.
Recently, the manufacturer withdrew its maintenance support for the
platform, making it essential for FSA to move more quickly towards a permanent
solution. However, it is our belief
that it will take at least several years to replace these applications with
fully re-engineered database products.
Moving forward with interim solutions, which have the potential to
significantly reduce manual workload and improve customer service, appears to
be well worth the effort.
Lessons Learned
In addition to the data that has been
created, digitizing centers have been very valuable for increasing the GIS
literacy within FSA. The centers have
become a fertile laboratory for generating ideas on how to transfer FSA program
functions from manual and outdated automated technology to GIS. Staff from the centers also have developed
the capacity to train employees at nearby counties and can fulfill a “help
desk” function for common problems.
These side benefits could be a major factor in recommending expansion
for the program.
Larger digitizing centers appear to work
better than smaller ones. Sites with
four workstations are significantly more productive than sites with only two
digitizing stations. They also seem to
place fewer trouble calls and have less staffing turnover problems.
Support from the State office is also very
important. Sites with strong State
office involvement are more successful.
The four states that originally proposed the project have been more
successful than the other three states.
They were up and running faster and have had fewer overall problems.
Access to other sources of local support,
such as universities, also enhances the success of the digitizing centers. Universities can help as both GIS resources
and sources of temporary staff for the digitizing centers. At least half of the digitizing centers
have some kind of relationship with local universities and community
colleges. FSA is currently talking with
a number of universities and, if funding becomes available, will actively
pursue formal partnerships.
However, in-house digitizing centers are, at
best, only a part of the solution to developing the Common Land Unit. The job is too big and FSA is not really
interested in being in the digitizing business. Even if FSA doubles the number of digitizing centers it will not
finish the CLU layer for the entire country for about 10 years at the current
rate of production. It would be advantageous to accelerate the program by
contracting out digitizing for at least 500 counties over the next two
years. If we can get budget
commitments, we will be contracting out as much digitizing as we can manage
over the next several years.
On a regular basis, we become aware of other
groups – federal, state, and local – who are digitizing farm field boundaries
for their own purposes. FSA needs to
find ways to develop partnerships with these other groups to effectively pool
our resources to ensure the sharing of data and applications.
We are also aware of projects going on
within FSA and our partner Agencies to digitize field boundaries outside the
umbrella of this project. Our biggest
concern about these projects is that they will not meet the standards developed
for the national program. It would be a
shame if they could not take advantage of tools that we are developing, but I
can certainly understand their frustration with how slowly the wheels seem to
be turning at the national level.
As indicated previously, maintenance is
critical to the CLU development process.
By the end of FY 2000, FSA will need to deploy Arcview, maintenance tools,
and hardware upgrades (additional disk space is crucial for the imagery) to as
many as 160 sites to keep the data from becoming obsolete. If we increase production, either with
additional digitizing centers or by outsourcing, we increase the need to deploy
maintenance hardware and software. We
also put pressure on our operations at APFO to mosaic and distribute DOQ.
In order to keep up with production, FSA
will need to find a cheaper and easier way to deliver maintenance tools and
other related functionality. One way we
have looked at and will look at more closely in the coming months is using
ArcIMS to deliver applications. Limited
bandwidth probably precludes delivering data over the web to some of our
service centers, at least in the short-term, but we might be able to use the
Internet as a software delivery mechanism.
Maintaining this data as shape files also
presents challenges, but appears to be unavoidable in the short term. One problem that we have already had at
numerous sites is getting staff to understand that more than one person cannot
edit a shape file at the same time.
Another problem is that current data for the counties will not be
readily accessible at either the State or National level. One way that FSA can solve these problems is
by managing the CLU in a distributed database environment.
FSA and the partner agencies are currently
evaluating database software and hardware systems to replace the System 36 and
our other legacy systems. Using SDE and
an Enterprise Database System will allow us to develop seamless, integrated
applications utilizing GIS as the front-end.
Database versioning may be a solution to the problem of managing
time-series information such as our annual compliance layer. But this may be a very costly solution
implemented nationwide.
Future Developments
Obviously, FSA has a long way to go before all farm fields in the country are digitized and GIS is fully deployed at all of our Service centers nationwide. Without additional funding, we will have completed the digital layer for 180-200 counties by the end of the year. But FSA currently does not have the budget to deploy Arcview in all of those counties. If we do not find additional funds soon, the data will not be maintained and will eventually become obsolete.
For the current fiscal year, FSA has
requested a supplemental budget appropriation for a number of projects,
including the Common Land Unit project.
With regard to CLU, the following areas are covered in this budget
request:
q
Contract digitizing for about 200 additional
counties
q
Hardware and Software for Maintenance sites
q
Hardware and Software for 10 new Digitizing
Centers
q
Hardware and Software for Image Processing
at APFO
q
Software development and Research and Development
Future funding for a project of this
magnitude is always in jeopardy and FSA needs to develop a more coherent
strategy for meeting the funding needs of this project. As GIS practitioners, our first priority has
been getting GIS software and tools to those counties where we have completed
the basic data layer. Our second
priority, in concert with departmental reengineering efforts, has been
prototyping software to meet FSA program needs in the field. Both of these will allow us to better
showcase the benefits of GIS on field operations and highlight the time and
budget savings that GIS can generate.
James Heald
GIS Specialist
USDA Farm Service Agency
Information Technology Services Division
Software Development Branch, Remote Sensing Section
Washington, DC 20250
(202) 720-0787
Jim_Heald@wdc.usda.gov