Craig Perreault

Using USGS DOQ's (Digital Orthophoto Quadrangles) and DRG's (Digital Raster Graphics) with Arcview and ArcInfo

A digital orthophoto is a digital image of an aerial photograph with displacements caused by camera angle and terrain removed. A digital orthophoto quadrangle (DOQ) combines the image characteristics of a photograph with the geometric qualities of a map. A digital raster graphic (DRG) is a scanned image of a U.S. Geological Survey (USGS) standard series topographic map, including all map collar information. The image inside the map neatline is georeferenced and fit to the projection of the source map. Arcview and ArcInfo provide users with the necessary tools to work with this image data, but because of the large file sizes, data compression, file naming conventions, and storage techniques used by USGS, how to use these files is not immediately obvious to most users. The good news is we can take advantage of the storage standards to make data access easier and also to automate the process of mosaicing and clipping adjacent files to make one seamless image.
The purpose of this paper is to highlight:

• background information on DOQ's and DRG's
• the processes of creating world files
  
• file naming conventions
  
• reading DOQ header files
  
• creating image catalogs
  
• removing collars from DRG's
  
• managing DOQ and DRG data sets
  
• putting adjacent images together
  
• DOQ contrast or histogram stretch.

A DOQ is a scanned 9 by 9 inch photo, taken from an airplane at 64 representing the quadrangle as shown in the table below
ab = The quadrant, either ne, nw, sw, or se
c = either h, denoting it is the header file, or c, denoting it is the compressed image

In the example below, the most north easterly quad would have four quarter quad images, made up of eight files. They would all have the same eight character prefix and would be named: O4709308.neh, O4709308.nec, O4709308.nwh, O4709308.nwc, O4709308.swh, O4709308.swc, O4709308.seh, and O4709308.sec.

Creating a world file to view the DOQ's in Arcview or ArcInfo

Step #1 Rename the compressed image file to have a .JPG file extension (rename O4709308.sec O4709308se.JPG). If you are in an unix environment you could create a symbolic link to the file (ln -f O4709308.sec O4709308se.JPG).

Step #2 Create a world file to provide the information needed to project the image into the real world. The world file for .JPG images will have the extension .JGW and will be made up of six lines.

1. x scale
   
2. rotation information (use value zero)
   
3. rotation information (use value zero)
   
4. y scale (as a negative number)
   
5. the x coordinate for the center of the upper left pixel
   
6. the y coordinate for the center of the upper left pixel
   

1
0
0
-1
x
y

Using Avenue to create world files and to view the DOQ's using MatchRectangles

This Aveneue example shows two methods to view DOQ's using Arcview. The first method is by creating a world file and renaming the image with a .JPG extension as described above. The second method is by using Arcview's MatchRectangles request.

'*** This first section of code reads the header file used by both methods ***

number.setdefformat("d.dd")
HeaderFile = TextFile.Make("c:\paper\O4709308.seh".asFileName,#FILE_PERM_READ)
quadname = HeaderFile.read(38).trim.asString
quadrant = HeaderFile.read(2) 'valid values are: NE NW SE SW
skipdata = HeaderFile.read(104)
numbrows = HeaderFile.read(6).trim.asNumber
numbcols = HeaderFile.read(6).trim.asNumber
skipdata = HeaderFile.read(numbcols - 156) 'read the rest of header record 1
skipdata = HeaderFile.read(numbcols) 'read header record 2
skipdata = HeaderFile.read(288) 'move to position 289 in record 3
ulx = HeaderFile.read(24).trim.substitute("D","E").asnumber 'x coordinate of upper left corner
uly = HeaderFile.read(24).trim.substitute("D","E").asnumber 'y coordinate of upper left corner
lrx = (ulx + numbcols) 'calculate x coordinate of lower right corner
lry = (uly - numbrows) 'calculate y coordinate of lower right corner
HeaderFile.close

'*** Method 1: Create the world file and copy the image to a file with .JPG extension ***

WorldFile = LineFile.Make("c:\paper\O4709308se.jgw".asFileName,#FILE_PERM_WRITE)
WorldFile.WriteElt("1")
WorldFile.WriteElt("0")
WorldFile.WriteElt("0")
WorldFile.WriteElt("-1")
WorldFile.WriteElt((ulx+0.5).AsString)
WorldFile.WriteElt((uly-0.5).AsString)
WorldFile.close
file.Copy("c:\paper\O4709308.sec".AsFileName,"c:\paper\O4709308se.JPG".AsFileName)

'*** Method 2: Add the DOQ to the theme assigning coordinates using MatchRectangles ***

theView=av.GetActiveDoc
theView.AddTheme(Theme.Make(srcname.make("c:\paper\O4709308.sec")))
FromRectangle = rect.makexy(1,1,numbcols,numbrows)
ToRectangle = rect.makexy(ulx,uly,lrx,lry)
DOQTheme = theview.getthemes.get(0)
DOQTheme.getimgsrc.MatchRectangles(FromRectangle,ToRectangle)
DOQTheme.Invalidate(TRUE)
DOQTheme.setname(quadname++quadrant)

On the CD Rom there are several informational files that contain detailed documentation. The file named \document\doqtext\doqsti.txt has the technical instructions including the file format for the header file. The header file has four records in it and is of variable length. The length of each header record is the same number of bytes as the number of columns wide the image is. There are only 400 bytes of information in each header record and the rest is padded with blanks. In the above example, it shows how to read the header file in Avenue. In the example, the variable numbcols is the width of the image.

When using Arcview you must load the JPEG image extension. The compression ratio for DOQ images is about 10:1, meaning a 40 megabyte quarter-quad compresses to about 4 megabytes. When viewing DOQ's you may notice performance issues while uncompressing these files, "it can be slow". Also, there is some data lost when using JPEG compression.

A digital raster graphic (DRG) is a scanned image of a U.S. Geological Survey (USGS) standard series topographic map, including all map collar information. The image inside the map neatline is georeferenced and fit to the projection of the source map, which is most often the Universal Transverse Mercator projection. The map is scanned at a minimum resolution of 250 dots per inch. Please refer to the following link for the technical documentation: http://mcmcweb.er.usgs.gov/drg . You can access the following link to find out what DRG data is available for your area. http://mcmcweb.er.usgs.gov/status/drg_stat.html .

DRG File Naming Conventions

The data is stored in TIFF file format and is available on CD Rom. Each CD Rom typically has a one degree block of DRG's, which consists of 64 TIFF image files, one for each quadrangle. The file sizes are usually 3 to 9 megabytes in size. The files follow DOS 8.3 naming convention of 12345678.tif where:

1 = A letter that denotes the series, scale, and class of the DRG (described below)
23 = Latitude of south east corner of the quadrangle in degrees
456 = Longitude of south east corner of the quadrangle as a positive integer in degrees
7 = A letter from A to H representing the northing grid value as shown in the table below
8 = A number from 1 to 8 representing the westing grid value as shown in the table below
tif = The GeoTIFF file extension

The different series, scales, and classes of the DRG are:

R = 7.5' series, 1:20,000 scale, Topographic class
O = 7.5' series, 1:24,000 scale, Topographic class
P = 7.5' series, 1:24,000 scale, Orthophoto class
L = 7.5' series, 1:25,000 scale, Topographic class
J = 7.5' series, 1:30,000 scale, Topographic class
K = 7.5'X15' series, 1:25,000 scale, Topographic class
I = Alaska series, 1:63,360 scale, Topographic class
G = 30'x60' series, 1:100,000 scale, Planimetric class
F = 30'x60' series, 1:100,000 scale, Topographic class
C = 1 x 2 Degree series,1:250,000 scale, Topographic class

Viewing a Digital Raster Graphic (DRG)

Viewing the DRG's is much simpler than viewing DOQ's because DRG's already have a world file and the proper file extension. They are in TIFF format with the file extension of .tif for the image and .tfw for the world file. The user must be aware that the map collar data makes it difficult to display adjacent images. At the Minnesota Department of Natural Resources (DNR) we have removed these collars. The un-collared data will still have an area of null data around the outer edges because a raster image is a rectangle and the quadrangles are tilted.
You need to make sure these "offsite" areas have a null data value.

The following example shows the problem of displaying adjacent DRG quadrangles with the collars still on.

Both ArcInfo and Arcview support Image Catalogs. An Image Catalog is a DBASE or Info table where each record contains an image file name and the image's minimum and maximum coordinates. The coordinates are used to select which images will be displayed, a world file is still needed. This allows users to draw several DOQ's or DRG's with one theme in Arcview or one command in Arcplot. The colors and legend information is obtained from the first image found in the image catalog. The table contains five fields: IMAGE (a string that holds the full path name to an image), XMIN, YMIN, XMAX and YMAX (numbers that hold the extent of the image). ArcInfo has a command CREATECATALOG. It is best if there is some overlap between images so there are no gaps. This is the case with DOQ's but not with DRG's. You will find the gap size can be rather large between some adjacent DRG's.

For our setup at Minnesota DNR, we have four system environment variables used to locate the DOQ and DRG data available to our system. These variables are as follows:

DOQFIRST - Look first in this directory for DOQ's (LAN)
DOQSRC - If not found in the DOQFIRST then look in this directory (WAN)
DRGFIRST - Look first in this directory for DRG's (LAN)
DRGSRC - If not found in the DRGFIRST directory then look in this directory (WAN)

Each of our regional offices in the state stores their images on their local area network (LAN), and the system variables DOQFIRST and DRGFIRST point to that data. The system variables DOQSRC and DRGSRC point to the rest of the state's data over the wide area network (WAN). We have set up a site, on a unix machine, where all the regional data sets for the state are symbolically linked in one directory. This allows for access to all the DOQ's and DRG's in the state, as well as allowing our computer programs to look for data on the LAN first, for performance issues.

We have developed an Arcview extension to make the distribution of DOQ's and DRG's less time consuming to offices without internet connections. This tool allows the user to specify a set of quadrangles and then copy the associated DOQ and DRG data to a destination directory. The destination directory can be a portable disk drive (like a JAZ or Zip drive) or a temporary directory to be used to write to a CD Rom. This extension searches the directories pointed to by the system variables and creates a .dbf file that stores the file sizes for the available DOQ quarter-quads and DRG quads. Then a theme is added to the view with its legend classified as follows:

Just DOQ data was found
Just DRG data was found
Both DOQ and DRG data was found
Just Partial DOQ data found
Both Partial DOQ and DRG data found
No Data was found

Partial DOQ means that one, two, or three quarter-quads of DOQ data was found but not all four.

The user then selects the quads (polygons) they wish to copy DOQ and/or DRG data. Arcview has many ways to select polygons including using the Select Features tool or by using the THEME: Select by Theme option or by using the Query button. Once the user has made a selection, the copy tool, provided by this extension, gives the following dialog to show how much disk space is needed to copy the selected quads. This can be especially helpful if you have limited disk resources such as a 650 Mb CD Rom. You could Cancel and change the selected set of quads to optimize your available disk space.

Some other applications of this extension include:

• Check which DOQ quarter-quads are on a particular CD. Helpful if a county is on multiple CD's
  
• Verify data written to a CD before distribution
  
• Check the volume of data (in bytes) for a selected set of quads
  
• Copy a partial CD of DOQ data to the hard disk
  
• Check what DOQ and DRG data is available to the system

Another program, we at the Minnesota DNR have written, is a tool to put adjacent images together. This is especially useful for digitizing off of DOQ's and for working with a large extent. The program determines which images intersect with the selected shape file and then puts the images together into a single new image. This is done by simply filling out the following dialog:

For Clipping DOQ's

Pseudo code for the program:

• Buffer the shape file (optional)
  
• Make a list of the quarter-quads that intersect with the shape file
  
• Clip the quarter-quads to the extent of the shape file (if necessary)
  
• Do a histogram stretch (optional)
  
• Resample to new cell size (optional)
  
• Mosaic images
  
• Clip to the shape file (optional)
  
• Export as a new GeoTIFF or a Erdas (.GIS) image file

For Clipping DRG's

Pseudo code for the program :

• Buffer the shape file (optional)
  
• Make a list of the quads that intersect with the shape file
  
• Clip the quads to the extent of the shape file (if necessary)
  
• Mosaic images
  
• Clip to the shape file (optional)
  
• Export as a new GeoTIFF file

This program uses the EPPL7 raster analysis program.
For more information on EPPL7 and a copy of this extension (DOQ_DD.AVX) go to:
http://www.mnplan.state.mn.us/EPPL7/

A histogram stretch is an image processing command that improves contrast in monochrome images such as DOQ's. This stretch redistributes clustered data values, spreading them across the full range of grey tones. The following four histogram charts show some examples of the distribution of data values in DOQ's. The charts show the number of cells for the grey scale values from 0 (to the left) through 255 (to the right).

The stretch will reduce the peaks and clusters in the histogram. This process will cause some data loss, but may make it easier to see features, thus making it easier to digitize from. Histogram Stretches can also be useful when mosaicing images where one is brighter than the other, reducing the contrast at the edges where the files are joined.

I would like to thank Tim Loesch, a colleague and friend, as we have been working together to develop these tools and procedures. This has made using these digital basemaps easier for many users throughout Minnesota.

Craig Perreault
Information Technology Specialist III
Minnesota Department of Natural Resources
1201 East Hwy 2
Grand Rapids, MN 55744
Phone: (218) 327-4335
Fax: (218) 327-4263
E-mail craig.perreault@dnr.state.mn.us