Bridging the Gap Between ArcView and Visual Basic

This paper presents creative solutions for facilitating the integration of ArcView and Visual Basic for large-scale application development. The application being modeled, one created for the U.S. Forest Service, tracks terrestrial wildlife observations and attribute data related to those observations. It integrates a custom ArcView extension with a Visual Basic interface and data access capabilities. The mating of ArcView and Visual Basic on such a large scale presented several hurdles, including passing information back and forth using DDE, using Visual Basic to control ArcView commands, and how to quickly and reliably access and update data.

Introduction

For years, application developers have been developing a wide variety of custom ArcView extensions using the Avenue programming language. While this has allowed for the development of a variety of highly functional applications, it has generally been difficult for developers to take advantage of the more robust and mainstream functionality of a Visual Basic application from within an ArcView extension. This paper presents some creative solutions that were developed for facilitating the integration of ArcView and Visual Basic on a large-scale application developed for the US Forest Service. The application integrates a custom ArcView extension with Visual Basic's interface and data access capabilities, allowing users to take advantage of some of the more advanced features available from Visual Basic while still essentially operating completely within an ArcView environment.

The mating of ArcView and Visual Basic on such a large scale presented several hurdles, including passing information back and forth using DDE, using Visual Basic to control ArcView commands, and how to quickly and reliably access and update data.

Background and History

The USDA Forest Service (USFS) has embarked on a national effort to develop a series of software applications that will provide consistent Ecosystem Management tools and capabilities to it's vast array of users across the country. This system, known as the Natural Resources Information System (NRIS), consists of several distinct modules that collectively provide users with tools to manage such issues as air, water and soil quality; vegetation; human dimensions; and terrestrial wildlife.

In 1999, the USFS contracted with DataLOGIC, Inc., to begin development of the NRIS terrestrial wildlife module, known as Fauna. The purpose of the Fauna module was to provide field biologists with a simple, fast, and consistent application that would allow them to standardize the collection of basic information, easily track threatened and endangered species sightings and surveys, and provide consistent evaluation and reporting tools to users across the country.

Working closely with the Forest Service staff, the Fauna development team identified the business needs and goals of the target users and prepared high-level designs of an application that would provide the field biologists with the tools they needed. Based on this evaluation, it became clear that the application should be a GIS based application with a significant spatial component, rather than a database application that simply allows storage, retrieval and reporting of wildlife data.

In order to meet those goals the team decided to use Esri's ArcView GIS software to provide the spatial component of the Fauna application. However, in an effort to bring more sophisticated data management capabilities to the application, it was decided that the team would attempt to integrate Visual Basic tools for data entry forms and data access with the ArcView extension to form the complete Fauna application.

In the fall of 1999, a prototype was developed as a proof of concept for the Fauna application. During the prototype phase, many of the specifics of the interaction between ArcView and Visual Basic were developed and tested, and have become the backbone of the Fauna architecture in subsequent release versions.

Version 1.0 of the Fauna application was released in the fall of 2000; this version served primarily as a data entry tool for users to begin entering their wildlife data into the Fauna database. Version 1.1 is being released in the early summer of 2001; this version provides a fully integrated client server application - the ArcView/Visual Basic interface interacts with a Microsoft Access database and ArcView shapefiles on the client side, and also provides full integration with an Oracle database and ArcInfo coverages on the server side for official "corporate" data storage.

Data Access - Philosophy & Architecture

The requirements set forth for the Fauna application specified that the application must be able to interact with an Oracle database over a Wide Area Network (WAN), but must also be able to operate in a standalone state, disconnected from any server. This requirement allows biologists to take the application into the field on a laptop to perform direct data entry, and still allow them to connect to the server later to upload their data to the master database. These requirements posed several major challenges to the Fauna development team, such as how to provide fast and consistent access to the server side database over inherently slow network connections, and how to provide reliable, stable, and secure data access through an ArcView interface without giving users direct access to the data tables, something that is difficult to do in ArcView and Avenue using traditional database access techniques.

To overcome these obstacles, a database architecture was developed that provided a mirror image of the corporate server side data on the client machine (Figure1). Rather than connecting directly to the corporate data in Oracle, a copy of the corporate data is created and maintained on the client in Microsoft Access through a process known as Data Currency. In the Data Currency process, each table in the Oracle database is compared to it's counterpart in the local Access copy of the database, to determine if the data in Oracle has been modified since the last time it was copied to the client.

If the Currency process determines that the local client data is not current with the Oracle data on the server, then a download process is initiated that copies the new data from the server to the client. This design ensures that the user has the most up to date data available on their client system, and allows them to disconnect from the server and operate in a standalone environment. This system architecture has provided an increase in performance on the client side that is orders of magnitude greater than what would occur if the ArcView application was constantly connected to the server database over the network.

To further maintain the integrity of the Fauna data, a second database is used on the client side to store data that is currently being entered by the user. This database, known as the Working database, provides a means in which new data could be added, deleted, or changed using specific Fauna tools added to the ArcView interface. Once this working copy of data is validated and approved, the user "commits" the data to the corporate database on the server. In this way the Oracle database and corporate GIS files only have to be accessed either at the time of copying down the most recent corporate data, or when "committing" approved working data to the corporate database. This system provides a significant improvement in performance, because the primary work of adding, deleting, viewing, or changing Fauna features occurs completely on the client side of the application, interacting with the local Access database and GIS files.

The DDE Communications Model

Another guiding principal of the Fauna application was that there would be no direct access to any of the application database files by the ArcView software itself. This was done for two reasons: first, having ArcView directly access the data results in a situation that is not secure - users would be able to modify data without going through the Fauna interface, which could result in data corruption. The second reason was to take advantage of the sophisticated data access functionality provided by Visual Basic. In order to meet these requirements, there needed to be a stable mechanism for communicating between ArcView/Avenue and Visual basic - this was accomplished through the use of Dynamic Data Exchange (DDE).

In the DDE architecture, the ArcView application and the Visual Basic application exist as two separate processes on the client system; DDE provides the means for passing data back and forth seamlessly between the two processes. The DDE communication was established as a two-way link between the two applications. Avenue scripts within the ArcView application were capable of calling various forms and functions from the Visual Basic (VB) portion of the application, and, likewise, the various VB forms and functions could call Avenue scripts whenever necessary.

While the DDE communication generally worked very well, there were, needless to say, several hurdles that had to be overcome before the system became stable enough to use on a consistent basis. The major difficulty was in making sure that the DDE connection was not only initiated at the proper time, but that it was also broken at the proper time. On both the Visual Basic and ArcView sides of the DDE link, there were several problems encountered in which the DDE link would either remain in a connected state even after it was no longer needed, which resulted in timeout errors, or the link would be broken prematurely, which resulted in a function or script continuing to process without having all of the information it needed to complete successfully.

The following sections describe in detail some of the issues that were involved in establishing a secure DDE connection between ArcView and Visual Basic, as well as some of the techniques that were developed to overcome these difficulties.

ArcView to Visual Basic Communication

In the Fauna DDE communication model, commands are sent from ArcView to Visual Basic through the use of a single form in the VB application that acts as a "listener" form. This form, which is loaded when the application first starts, is invisible to the user; it has one primary purpose in the application, and that is to sit in memory and wait until it receives a command from ArcView. The function of this form can be likened to a transfer station; once the form receives an ArcView command string, the string is passed to an interpreter function that determines exactly what it is that ArcView is requesting. From there, the interpreter function launches the appropriate activity, whether it be opening another form for user interaction or simply executing a function within the VB application.

An example of this type of communication is shown in Figure 2. In this example, the user has selected a Fauna tool within the ArcView interface that adds a new Fauna feature known as a Basic Observation. The ArcView tool captures the location that the user has clicked, and passes the coordinates of that point, along with several other pieces of data, to a script that initiates a DDE function call. That script uses DDE to pass the data string to the "listener" form in the Fauna VB application. The listener then interprets the command string, and determines that the user is requesting to add a new Basic Observation at the specified coordinates. The result is that the interpreter instructs the application to open the Basic Observation Data Entry form, pre-populated with the coordinates that the user clicked in the view.

Figure 2.

There were two significant hurdles that had to be overcome in order for this approach to work smoothly. The first was that since there is a limit on the number of characters that can be sent through the DDE execute command, it often became necessary to send additional information by instructing ArcView to create a text file containing a list of parameters that would be required by the VB function. The command sent to the Visual Basic interpreter would then instruct the application to read the text file created by ArcView to determine the values of the input parameters it needed to perform the requested activity. This same technique was also used frequently to pass long command strings in the reverse situation from Visual Basic to ArcView.

The second and more significant problem that occurred was in getting Visual Basic to release the DDE connection at the proper time. When VB receives a DDE command, it must complete the entire command before it releases the connection. In many instances, this was not a problem, because the command would complete successfully, the connection would be released, and VB would return to a state where it would be listening for the next command. However, in cases where there were complicated functions that were being initiated by the command string, there would often be times where the VB function would need to make a reverse call to ArcView somewhere in the middle of the original function call. If a call was made back to ArcView before the original function was complete, the application would hang until the DDE timeout value was reached, resulting in a DDE Timeout error.

The solution to this problem was to use a timer control within the listener form. In this scenario, whenever the listener received a call from ArcView, it would store the command string in a property variable, then initiate a call to the timer control. This effectively closed the DDE connection, since the only activity that was actually requested by the DDE command was to initiate the timer; once the timer was started, the DDE call had completed it's mission and closed itself. The only activity conducted by the timer control was to start the command interpreter, which would then read the command string from the stored variable and initiate the proper activity. The timer control interval was set to a value of one millisecond; effectively, this resulted in a situation where the requested command would initiate immediately as soon as the DDE connection was closed, so there was no performance penalty to the application using this approach. Since the connection was effectively broken, any VB function that had been called was free to initiate a new request back to ArcView without any fear of causing a timeout error.

Visual Basic to ArcView Communication

Communication from Visual Basic to ArcView was performed using a similar concept, although the technical aspects of the communication varied slightly. In this scenario, the ArcView application itself is functioning as the "listener" - whenever it received a command string from Visual Basic, it would simply execute the command. The command string could be a single Avenue command, a string of Avenue commands, or simply a call to run an Avenue script that exists within the current ArcView project.

Command strings were passed from Visual Basic to ArcView using either the LinkRequest command or the LinkExecute command on a text box object. The difference between the way in which these two commands are used is based on whether a value needs to be returned or not from ArcView. If a value needed to be returned from ArcView, then the LinkRequest command would be used. If the goal was to run a script or command that does not return a specific value, then the LinkExecute command could be used.

Continuing with the above example (as shown in Figure 2), when the user has clicked the Apply button on the Basic Observation form (Figure 3), the field values shown in the form are written to the appropriate tables in the Working database as attributes of a new record, and to a text file for transfer to ArcView. VB then signals ArcView to read the text file and add a new point to the observation shapefile. The new point is now a feature and the legend is updated to reflect the classification of the new observation.

Figure 3.

A similar type of problem occurred with this type of communication as occurred with the ArcView to VB communication path: a Visual Basic function that called an ArcView command will not proceed until the ArcView script that has been called has completed running. If the particular script that was called needed a significant amount of time to complete, then a DDE Timeout error would once again be likely to occur. In these cases, the solution was to use the Avenue AV.DelayedRun command to execute the script instead of simply calling the AV.Run command. Because the DDE command was telling ArcView to simply execute the DelayedRun command, as soon as that command was executed the DDE connection would be released. The time delay parameter on the DelayedRun command was set to the minimum value of zero seconds, which resulted in the immediate execution of the desired command, removing the possibility of a command timing out.

Future Development - Migrating to an ArcGIS Platform

The current version of the Fauna application has been developed to run on an ArcView 3.2x platform. However, the migration path of the application is to begin development of a version that will also run on the ArcView 8.x platform. The application has been designed with this migration in mind; although the ArcGIS version will not require DDE to communicate, the amount of work required to transition the application to the new platform will be kept to a minimum.

Even though the Fauna application is packaged as an ArcView extension and is driven by the ArcView interface, over 80% of the actual code has been written using Visual Basic. This was done intentionally, with the intent being that the remaining 20% written in Avenue will be the focus of the migration effort to ArcGIS.

Once the Avenue code has been converted to Visual Basic, the remaining significant task will be to re-write the communication on the VB side to remove the need for DDE. Because the application has been designed using the "listener" form approach, this will be a relatively straightforward process; all of the functions that currently are called by the "listener" form will be converted so that they are instead called directly by controls within the ArcView 8 interface. This design will minimize the effort required to perform the conversion, and will also minimize the impacts on the other portions of the application that do not directly communicate with ArcView.