Ronnie Buzzard
This paper will go into great detail on the processes, procedures, and steps used to create a nationwide GIS. Working with national (USDOT) and state agencies Michael Baker Corporation has created the NPMS. The NPMS is a GIS that tracks geospatial, metadata, and attribute data of all hazardous liquid and natural gas transmission pipelines located in the United States.
The process includes data collection, data processing, and creation of an Arc/IMS website to exhibit the NPMS data. The site also exhibits other related DOT data such as population, unusually sensitive areas, natural disaster, and commercially navigable waterways.
Michael Baker Corp was contracted by OPS to fulfill the requirements set by OPS to build the National Pipeline Mapping System. In October 1998 the National Repository and several State Repositories including California, Kansas, Kentucky, Louisiana, Minnesota, New Jersey, Oklahoma, and Texas were created. Upon creation each State repository is responsible for all LNG facilities, hazardous liquid, and Natural Gas Interstate and Intrastate pipelines within their state. The National Repository is responsible for all Interstate and Intrastate pipelines not contained within a state with a repository. The National Repository is also responsible for Interstate edge matching, and supervision of the Individual State repositories.
In March of 1999 to ensure accuracy and reliability of pipeline data a set of standards was developed by a joint government-industry Pipeline Mapping Quality Action Team (MQAT II). MQAT II was formed to work with the OPS on creating the digital pipeline location and attribute layer of the NPMS. The team was sponsored by the OPS, American Petroleum Institute (API), American Gas Association (AGA), and Interstate Natural Gas Association of America (INGAA), and included representatives from multiple Federal and State government agencies, and the natural gas and hazardous liquid pipeline industry.
The NPMS Standards for Pipeline and Liquefied Natural Gas Operator Submissions was created to guide operators through the submission process and to answer any questions that may arise. The Operator standards cover such issues as metadata, geospatial, and attribute data creation and format. It provides complete instructions with real world examples. It also includes regulatory and general requirements, and transmission instructions.
The National and State Repository Standards for the NPMS were created with input from Federal and State government agencies and the pipeline industry. They address and describe how the National Repository and State Repository collect, process, exchange, distribute, display and store data.
Each submission to the NPMS consists of three parts: Geospatial data, Attribute data, and Metadata. Each of these parts must adhere to certain rules and regulations to ensure seamless incorporation into the NPMS database. These rules and regulations are clearly outlined in the operator standards and are enforced by the National Repository.
The geospatial data submission to the NPMS is a geo-referenced line representing a hazardous liquid pipeline, NG pipeline or a point representing a LNG facility. These submissions can be made in any number of formats including but not limited to Paper, Arc/Info .E00, ArcView .Shp, AutoCADD .Dwg or .Dxf, MicroStation .Dgn, Intergraph .Mge, MapInfo .MIF and Generic.Txt. Operators can also submit their pipeline data in paper format. The Operator Standards provide complete submission instructions and examples for all digital and paper submission formats.
The Attribute data submission to the NPMS is a table that is linked to the Geospatial data describing the individual pipeline or LNG facility. It contains such information as operator number, operator name, system name, sub system name, pipeline ID, diameter, commodity(s) carried, commodity description, interstate designation, pipeline status, data quality, revision code, and metadata file name.
The Metadata submission to the NPMS is a table that contains many pieces of important information that is used by the NPMS. The Metadata files contains such information as submission information, geospatial data quality information, attribute data information, projection information, transmission information, hard-copy data information, operator contact information, and source information.
The National and State repositories are responsible for collecting the pipeline and LNG facility information, processing it, quality checking it, and incorporating it into the NPMS database. To determine what repository the submission is sent the operator must verify the pipeline's classification as interstate or intrastate. Intrastate pipelines are pipelines that do not cross state boundaries; these pipelines are to be submitted to the individual State repository or if none exist to the National Repository. For Interstate pipelines, the pipelines can be submitted as a whole to the National Repository or split up according to State boundaries and submitted to existing State repositories with the balance going to the National Repository. If this is the case, the National Repository is responsible for collecting the remainder of the data after it has been processed by the other repositories, edge match it, and send it back to the originating repository. This is done to create uniform pipeline data that seamlessly crosses State boundaries.
The attribute tables includes 19 total fields, 16 populated by the operator and three populated by the Repository. Of the 15 fields populated by the operator 10 are required and six are optional. The 10 required fields are as follows: 1) OPS_ID - this is a five-digit accounting number assigned by OPS to the company that physically operates the pipeline system. 2) OPER_NM - a person or firm who operates a pipeline system and engages in the transportation of gas or hazardous liquid. The operator may or may not also be the owner of the pipeline system. This name is linked to OPS_ID in the OPS database. 3) SYS_NM - this is the system name. It is defined by OPS as "All parts of a major natural gas transmission line or hazardous liquid trunkline through which gas or hazardous liquid is transported. By definition, only one firm can operate a pipeline system. Operators should assign unique names to each of their pipelines systems. A pipeline system may have an unlimited number of branches. Each pipeline system must be represented by one or more pipeline segments. 4) PLINE_ID - this is an identifier for a specific section of pipeline within a pipeline system. 5) COMMODITY1 - this is the primary commodity carried by the pipeline system. 6) INTERSTATE - this is a (Y)es or (N)o designator to identify if the pipeline system is an interstate pipeline. 7) STATUS_CD - this identifies the current status of the pipeline segment. 8) QUALITY_CD - this is the operators estimate of the positional accuracy of the submitted pipeline segment. 9) REVIS_CD - this identifies the pipeline segment as an (A)ddition, (M)odification, or (D)eletion. 10) METADATA_ID - this is the name of the metadata file associated with this data.
The six optional fields are as follows: 1) SUBSYS_NM - this is a unique name assigned by the operator for a smaller subsection of a pipeline. 2) DIAMETER - this is the diameter of the specific pipeline segment. 3) COMMODITY2 - this is the secondary commodity carried by the pipeline. 4) COMMODITY3 - this is the tertiary commodity carried by the pipeline system. 5) CMDTY_DESC - this is descriptive information of the commodities carried by the pipeline system. 6) OPER_LINK - this is a link between the geospatial elements and their respective attribute records.
The three fields populated by the Repository are as follows: 1) NPMS_SYS_ID - this is an eight- digit unique ID given to each pipeline system by the specific repository, each repository is given a range of numbers to prevent duplicates. 2) ALBERS_MILES - this is the mileage assigned to the specific pipeline segment. 3) SUBREPOS_CD - this is the repository where the submission originated.
To facilitate the attribute data creation process, the National Repository has created software that guides the operator through the attribute creation process. The attribute data creation software was created using Visual Basic and can be installed on the operators PC. The application creates an Access database that can in turn be appended to the geospatial data by the National or a State repository. This software can be downloaded from the NPMS website, or sent to the operator on CD upon request.
The Operator Metadata is descriptive information about data, such as the timeliness of the data, attribute sources, and accuracy of the data. The Operator Metadata includes three tables: Primary Metadata, Operator Contact, and Source Material. These tables include information about the operator, the pipeline, and the submittal. The Primary Metadata table includes information such as submittal dates, projection information, and other pieces of information that is useful to the repository. The Operator Contact table includes contact information such as contact name, phone number, address and e-mail. The Source Material table includes information on the source of the geospatial data, like the source of the submittal i.e. USGS quad sheets etc.
The National Repository has also created metadata creation software that guides the operator through the metadata creation process.
Once this information is compiled by the operator there are several ways to send it to the NPMS. The operator can deliver it by e-mail, FTP it via the NPMS website, or send it on disk by mail. When the data is received by the repository it is processed into proper NPMS Format,, and is put through a rigorous QA/QC process. This process includes Check-in, Process, Check-Plot, QA/QC and Database Incorporation.
The Check-in phase of submission processing includes assigning an NPMS-ID to the submission, and logging it onto the submission tracking web page. The ID is created by adding the repository Id; for example NR for National Repository, VA for Virginia Repository etc., then adding it's OPS Id and then the four-digit submission number. For example; company X with an OPS ID of 12345 making it's first submission to the National Repository the NPMS ID would be NR+12345+-0001 = NR12345-0001. Once this Id is generated the Operator is notified that the repository has received the submittal and is given the NPMS Id. Once this is accomplished the Processing stage can begin.
Although each submittal is different a standard procedure has been adopted by the NPMS to process the data. The first step is to convert the data to an Arc/INFO coverage format. With paper submissions this includes digitizing the maps, for all others the NPMS uses various techniques to translate the data. This includes using conversion techniques in ArcTool Box, and the use of a universal translator called FME. Once in Coverage format the repository is to define the submission's projection. This information can be found in the Primary Metadata table. Next the Geospatial data is appended to the Attribute data. This is done in ArcView using a script called Fix Join, that allows all joins to be made permanent. Once the Tables are appended, the next step is to remove all fields that are not required such as Oper_link. The next step is to run the submittal through an aml that was created by the National Repository. This aml does several things 1) Reprojects the pipeline data to Geographic, Nad83. 2) Adds nodes to the pipeline data at State boundaries (this aids in the interstate edge matching process). 3) Creates a new field called Albers_Miles. 4) Reprojects to Albers and the mileage of each segment is calculated and placed in the Albers_Miles field. 5) Reprojects the data back to Geographic. 6) Adds and calculates NPMS_SYS_ID, and SUBREPOS_CD fields. 7) Arranges all fields into proper NPMS format. Once this is complete the QA/QC stage can begin.
The QA/QC stage begins at the repository. The repository insures that all field names are in NPMS format, and required fields are populated. At this point all State repositories will submit their data to the National Repository. The National Repository will then turn over the data to an independent QC team. The team will be responsible for performing quality control checks on all NPMS data. The QC Team will not correct any errors that are found in the data set. All errors, regardless of their simplicity or frequency, will need to be corrected by the submitting repository. Following an error report, a new submission of the data set must be sent to the QC Team until all (QA/QC) tests are passed. The four steps that the QC Team will follow are Automated Screening Program, Visual Screening Process, Detailed Visual Screening, and Reporting.
The Automated Screening Program includes: 1. Check field attribute value, 2. Check attribute data rules, 3. Check attribute value against the allowable range, 4. Check field attributes that cross reference data tables, 5. Check fields that can be cross referenced to metadata, 6. Check topological Structure/connectivity of data set, 7. Develop statistical abstract of the submission, 8. Track Procedures.
The Visual Screening Process includes: 1. Verify overall integrity, 2. Thematically shade attributes.
The Detailed Visual Screening includes: 1. Interstate pipeline systems, 2. Inspect closely for localized data inconsistencies, 3. Overlay with standard boundary data to ensure general positional accuracy, 4. Check for unexplained inconsistencies, 5. Overlay selected sections with scanned quad data to check detailed position, 6. If there are multiple data tiles check for edge-matching, 7. Overlay with other repository data to check for conflicts.
Once the QC stage is complete and the report is made to the National Repository the Check-plots can be made to the operator. Upon receipt of the check-plot the operator is given 10 working days to review, make changes and return the changes to the originating repository. If not contacted by the operator within ten days the repository is to assume that the data is correct and can incorporate that submission into their database. If changes are necessary the changes are made and the submission is re-sent through the QA/QC process.
At this point National Repository appends the submittal to the database and the data is made assessable to the public. The NPMS accomplishes this by making all data downloadable from the internet at; www.npms.rspa.dot.gov/data/dot_data.htm, and by making a fully functional Arc IMS site at: 209.118.98.195/lower48/ this IMS site features not only pipeline data but also related layers affecting pipeline operators and the public. These layers can be selected and identified online, and viewed in relation to pipeline data to analyze any risk the pipelines may pose to geographic areas in need of protection.
The Arc IMS site was designed to enable the user to classify the liquid pipelines, query the data by operator and to search by geographic area, including state, county, and zip code. This was done by adding a series of tools to the bottom of the layers list using ASP. These tools allow the user to pick a liquid or gas pipeline from a dropdown list then pick the operator in question. Once the operator is picked the site automatically selects the pipeline, displays it in green and zooms to it's full extent. If the user wishes to query by geographic area the user would choose "zoom to", choose the area, and the site automatically zooms to that area.
The process of building the NPMS is ongoing and updates are constantly being made. Currently the NPMS has collected and processed over 133,000 miles of natural gas pipelines, and over 130,000 miles of liquid pipelines. As additional pipeline mileage jurisdictional to OPS is collected, the NPMS is becoming a more useful tool to government officials, operators, and the public.
Ronnie Buzzard
GIS Consultant
Michael Baker Corp.
3601 Eisenhower Ave. Suite 600
Alexandria, VA 22304
703-317-6205
fax: 703-960-9125
Rbuzzard@mbakercorp.com