This paper presents the findings on the effort to migrate existing data from a large Unix based Oracle/SDE 3.0 environment to a Windows 2000/Oracle/ArcSDE 8.1(Beta 2) environment.
It addresses the issues of the optimal solution selection, best implementation method, and some of the user feedbacks, including performance and improvements in graphical interfaces.
For this experiment, data from the Brazilian Amazon Basin was used, including geopolitical information, vegetation, hydrology, land use, deforestation, and areas of potential forest burning layers.
By the end, the paper presents the next step: migrating from the Avenue application.
The System for the Amazon Surveillance – SIVAM - is a Brazilian Government Project which goal is to provide applications that will help Brazilian Government in all levels to achieve the self-sustained development of the Brazilian Legal Amazon Rain Forest region (Figure 1).
SIVAM consists of equipment, software, and tools, which are combined to assist Brazilian public and private user agencies to support the objectives of the Government of Brazil for the Amazon region. Specific goals of the Government were defined for meeting the challenge of sustained development while preserving the environment:
• Environmental protection
• Control of land occupation and use
• Surveillance and control of borders
• Curtailment of illegal activities
• Prevention and control of disease
• Protection of indigenous populations
• Surveillance and control of air traffic
• Monitoring of river navigation
Figure 1 - Legal Amazon
The SIVAM project was originally conceived to aggregate all its products and data on three Regional Monitoring Centers, located at Manaus, Belém and Porto Velho, also having a General Coordination Center, at Brasília. It was also planned to deploy 940 satellite communication equipments with a SIVAM user kit. The idea was to provide SIVAM information to city and state based users.The current implementation of the, SIVAM, developed by Raytheon Systems Company and ATECH (Fundação de Tecnologias Críticas e Aplicadas), is based on the HP-UX operating system and its correlated PA-RISC platforms. It uses a large database hosted on enterprise level Servers, sharing a RAID 5 configuration system. The software implementation architecture is based in a standard client-server style using an Oracle8i database as backend, a SDE 3.0 connector, and ARCView 3.2 and ARCInfo 7.X as clients. In order to provide web-based geographic services, an ARCViewIMS server was implemented. Figure 2 shows the current hardware architecture of the Environmental Surveillance Subsector of SIVAM.
Figure 2 - SIVAM Environmental Surveillance Hardware Architecture
The set of applications developed to support daily-based activities were built using AVENUE and AML as macro-languages. Although it seems to be a very conservative implementation, it accomplished all the contractual requirements, and it reflects the current state of the technology at contract award timeframe.
SIVAM data is a collection of several sources of geographic and tabular information collected on many Brazilian Federal agencies, Amazon Region State agencies and other data providers. It comprehends of satellite and aircraft based images, existing raster and vector of GIS layers, CAD data, and social-economics tabular data from a large variety of existing databases, summarizing several terabytes.
It uses a base map provided by IBGE( Instituto Brasileiro de Geografia e Estatística) scaled in 1:250.000 of the entire area. This effort was accomplished in a cooperation agreement between SIVAM Coordination Agency and IBGE.
The internet demand increase and its spread-out across all nations, thus asking for high quality low bandwidth services, and the continuous research for system improvement lead the team to look for alternatives in order to reach those goals.
Also the team verified that the current implementation lacks the scalability, reliability, and performance that a true distributed object oriented environment can provide.
Although UNIX based system have known reliability, robustness, and open standards committee policy supported by all developers, it is also a fact the INTEL – MS Windows based systems are much cheaper to acquire, install, maintain, deploy and provide training, specially if one consider the client side (workstations).
It was proved that the UNIX based application is not as friendly as its Windows based counterpart. Most of the personnel capable to use the system was skilled in MS Windows application but with no or very limited UNIX training. It is good to mention that the Project is deployed in a region which human resources have limited computer operation skills.
The goal of this case study is to provide a road map to upgrade the system increasing its scalability, reliability, performance, and user friendliness, with minimum effort, using its own personnel, reducing overall implementation costs, without continuity solution (i.e. maintaining the daily-based operation).
Test data for the case study was selected from a subset of SIVAM vector layers, LandSat-5 images and set of IBGE tabular data from the Rondônia State in Brazil. Figure 3 provides a view of the area. A smaller region under the main test region was selected because deforestation and land use layers were available. Figure 4 shows the selected region. Figure 5 shows LandSat-5 images used within the selected region.
Figure 3 - Main Test Region
Figure 4 - Detailed Test Region.
Figure 5 - LandSat 5 Images used.
Several architectures are candidates for implementation. The first possible choice would be a simple upgrade on the equipments and software, that is, UNIX would be kept as the operating system for the workstations and servers, and UNIX-based versions of ARCView and ARC/Info would be installed on the client workstations. This would mean that features based on the capabilities of the Windows NT technology would not be available, such as: ArcObjects, ArcToolbox, ArcMap and ArcCatalog, all designed using COM/DCOM technology.
A second choice would be radically replace all computer hardware to Intel-based systems running Windows 2000/NT, including data and Internet map servers. This would allow all new features of Esri latest technology software for the clients, but all the investment on hardware and software for UNIX-based systems would be lost.
A more conservative but reasonable approach would be to accept current trends on software technology and use UNIX were it is best suitable, data servers using Oracle as DBMS and ArcSDE as spatial middleware. Workstations would use Windows 2000/NT, thus taking advantage of all distributed objects features of ArcGIS. This approach would also preserve all investments on the large already existent UNIX servers running HP-UX. Availability of ArcIMS for HP-UX is an issue, tough, and that may drive the Internet map services to be installed on Windows 2000 servers.
As it would seem logical, the last approach was selected as the correct path to follow, since it involves less risks, costs and training for the system administrators and DBAs. This solution would trigger a radical revolution on the clients, though, exponentially increasing the capabilities available for the users, using an environment which they are already familiar with.
ArcSDE will host and serve all spatial vector data. Tabular data received either from other SIVAM sites, state or federal agencies will also be stored on the Oracle/ArcSDE server. Raster data will also be stored preferentially on ArcSDE. In the case of performance issues, the file server or locally storage can be used to improve performance.
Ingestion of vector layers will be preferentially a direct export/import process from the SDE 3.0 servers on SIVAM Regional Centers current configuration to the ArcSDE servers on the new configuration, using the native compatibility among versions. Ingestion of data coming form other sources and other formats will be achieved by using ArcToolbox capabilities and MS Access database infrastructure when needed.
All spatial data manipulation will mainly use ArcGIS product family features, giving precedence to GUI based capabilities and maximizing the use of user-driven interfaces such as wizards. Use of ancillary ARC/INFO Workstation capabilities is restricted to those not covered by ArcGIS. Use of AML and Avenue is discontinued, giving ground to VBA customization or DCOM distributed objects architecture (ARC Objects).
There will be two ways of data publishing. The first one will be through the ArcGIS applications on each workstation. The user shall print them. Data will be published to the general public using ArcMap map production capabilities, but the dissemination of data over the intranet/Internet connections will be strongly encouraged, using ArcIMS.
Management of both ArcSDE and file based stored data will be accomplished by using ArcCatalog, including raster data stored on the file server or local file system. This will also include metadata.
In order to prove that the solution was feasible, practical and accepted by decision makers, a test bed was developed. The Test Bed configuration was composed by two dual Dell Pentium III 733 MHz with 512 MB RAM and 10,000-RPM SCSI RAID-5 storage units (36 available GB) as servers and a Dell dual Pentium III 1GHz with 512 MB RAM and a 10,000-RPM SCSI disk was chosen as workstation. A Catalyst 2912 XL CISCO switch, a SonicWall firewall and laptop with network analyzer software complemented the hardware specification as shown in Figure 6.
Figure 6 - Prototype hardware configuration.
The proof of concept was made using the sample data from the Amazon Region mentioned above, mainly the Rondônia State. It was composed of several layers and social-economic tabular data with the objective to produce thematic information of the region in the same format as of the current system.
The following figures present the results as described bellow:
a) Figure 7 - Rondônia State livestock information (IBGE);
b) Figure 8 - Rondônia State Population Information - Male and Female distribution (IBGE);
c) Figure 9 - Rondônia State Agricultural Production;
d) Figures 10, 11 and 12 - Rondônia State LandSat images of Rondônia combined with vector data (roads, railroads, political boundaries, localities and hydrology);
Figure 7 - Population of bovine, swine and poultry flocks from selected Rondônia Municípios.1
Figure 8 - Population of selected Rondônia Municípios.
Figure 9 - Agricultural Production of selected Rondônia Municípios.
Figure 10 - LandSat images of Rondônia combined with vector data (roads, railroads, political boundaries, localities and hydrology).
Figure 11 - Same images and data used on Figure 10 , zooming in.
Figure 12 - Another zoom on the data described on Figure 10 - showing city limits .
Migration from the HP-UX 11.0-based Oracle/SDE 3.0 vector layers to the ArcSDE 8.1 Server was smooth, using batch files with sdeexport and sdeimport commands. No imprecision or loss was observed on the migrated data. Ingestion of coverages and shapefiles from other sources was also easy, using ArcToolbox. Importing MS Access tables into ArcSDE, nevertheless, proved to be difficult, although ArcGIS applications could read Access tables directly. It would seem a good idea that Access jet engine format should be added to the standard export and import formats for tables, in addition to the existing DBase and Info formats. We had to use Oracle SQLLoader or Access to Oracle ODBC connection to circumvent this problem.
Concerning raster images, although ArcSDE was able to import them on the database, it was still not easy to manipulate Landsat 5 TIFF format images when imported to the ArcSDE geodatabase. ArcCatalog performed well with file system stored images, though, especially those on Erdas Imagine format.
Migrating UNIX-based applications to a Windows-based environment was extremely simple, as expected. Some results are shown on Figure 13 , which replicates a SIVAM deforestation classification (Figure 14). Figure 15 shows a map developed by ARCGIS and a current SIVAM map is shown on Figure 16.
Figure 13 - Using ArcMap to view classified deforestation regions on Rondônia.
Figure 14 - SIVAM current Deforastation Application.
Figure 15 - Draft map produced by ArcMap.
Figure 16 - SIVAM Deforestation Map (Raytheon Courtesy).
This paper has addressed the issue of migrating from a heavily-based Unix environment to a Client-Server environment where UNIX is used as the server side solution and MS Windows 2000 architecture is used at the client side.
To validate the approach, a test-bed was developed, which proved the correctness of the selected solution when considering cost, friendliness, training, visual effects, and application out-of-the-box features.
The next research path is the evaluation of all necessary steps for the complete migration from an Avenue based application within ArcView to an application using Esri ArcGIS COM/DCOM technologies.
Clements, Paul C. and Linda M. Northrop, Software Architecture: An Executive Overview. Technical Report CMU/SEI-96-TR-003 ESC-TR-96-003 , February 1996.
Environmental Systems Research Institute, System Design Strategies. Esri White Paper, June 2000.
Environmental Systems Research Institute, System Architecture Design for GIS. 20th Annual Esri International Users Conference Preconference Seminar, June 2000.
This study and report were funded by the Commision for Coordination of the System for the Surveillance of the Amazon Project (CCSIVAM). Our thanks go to Esri Defense Group that allowed us to work with ArcGIS 8.1 Beta versions, IBGE for providing tabular and vector data, and Raytheon Systems Company SIVAM Environmental Surveillance IPT, for providing us SIVAM SCC GIS vector and raster layers and Environmental Surveillance print screens .
Capt. Eduardo Quesado Filgueiras, MSc.
Database & Software Architect
Commission for Coordination of the System for the Surveillance of the Amazon
Project
Professor
Lutheran University of Brazil
Address:
Av. do Turismo, 1350
Tarumã - Manaus - AM Brasil
CEP: 69049-630
Phone # + 55 92 631 2421
Email: quesado@cc.sivam.gov.br
1 A "município" is a Brazilian political-administrative division, which encompasses a city, town or village and the territory associated to it, including all localities within that territory.