Building GIS Web Services With .NET and ArcObjects

Esri UC2002 - Paper 191


Brian Flood, Spatial Data Logic

Forward

This paper presents a variety of techniques for incorporating distributed computing power into a GIS using Microsoft’s .NET technology. The techniques cover browser based clients, Esri’s ArcMap and the new .NET "Smart Clients" however it does not rule out other methodologies or frameworks. Due to limited space in covering three complex subjects, some sections include only cursory information. For more detailed information, please contact Spatial Data Logic directly.

The opinions expressed within are solely those of Spatial Data Logic and may not represent the opinions of either Microsoft or Esri. Additionally, all of the techniques covered require a special "web enabled" license from Esri. Please contact your local Esri representative for more information.

Overview


We are at the beginning of two product lines that may profoundly impact how large scale Geographical Information Systems (GIS) will be designed and deployed. Both lines bring critical pieces to the puzzle and both will help solve the underlying problem of the enterprise: How to build and deploy feature-rich user applications over a distributed network while lowering the total cost of ownership for the individual client machines.

The two products are Microsoft’s .NET initiative and Esri’s ArcObjects. The former will provide the user interface (UI), the network communications, the component framework and the basic glue to hold the entire system together. The latter will provide the three basic tenants of GIS: data storage, map rendering and spatial analysis. When coupled together, we have a very potent combination, one that will allow for massive multi-user systems that deploy as separate units but exists as a single, server based entity. A system that can be fully administered (installations and updates) from a central location yet still provides a comprehensive UI that end users have come to expect from traditional desktop applications.

This represents the long-term goals of distributed GIS, however there may be several steps in between before the ultimate system is fully incorporated into existing GIS systems. This paper outlines different scenarios where distributed GIS processing can be incorporated into either existing or future enterprise systems. The scenarios use both .NET and ArcObjects because it is believed that they represent the best tools for the job. However, it should be noted that other frameworks might be used in their place. The paper is separated into three sections, each representing a different method of incorporating distributed GIS processing techniques into the enterprise.

The first section covers GIS Web Services built using ArcObjects on the server and using XML for the remote communications. As an example, a simple TopologicalOperator Web Service is created. Any SOAP (Simple Object Access Protocol) aware clients can consume the service and perform spatial operators (Buffer, Intersect, Difference etc.) on their geometry data. Web Services are best suited for applications that must integrate with other outside applications that may have been created using completely different standards.

The second section covers .NET Remoting from within the ArcMap client. Remoting is a more powerful technology then just simple SOAP based Web Services because it allows for secure, two-way communications using complex object graphs. From an ArcMap perspective, it provides a very good mechanism for communicating with remote ArcGIS services, many of which will have more ArcGIS functionality then is present at the calling client. A sample "Remote Editor" is shown that allows ArcView 8.x users the ability to edit complex ArcSDE based Geodatabases without changing licenses. The .NET Remoting architecture dramatically decreased the time needed to create the sample.

The third section covers SDL’s GeoLogic.Net software. Unlike, the Remoting example of the second section, which shows only portions of the GIS being remoted, GeoLogic.Net goes the next step and remotes the entire GIS. While it is not the only goal of GeoLogic.Net, remoting the entire GIS paves the way for self-updating client code, which ultimately leads to reduced TCO (Total Cost of Ownership) for the enterprise. The .NET Smart Client technology that is used deploys an application like a traditional web page, without any client intervention. However, unlike typical browser based GIS applications, the end user is presented with a feature rich UI that is far more capable.

This paper will not cover the basic definitions of how Web Services work, there are far better resources on the web and at the UC2002 conference that describe in detail XML, SOAP, WSDL, UDDI or REST. This paper concentrates on specific examples of the technologies in action, examples of where distributed processing can help solve real problems. Although the commercial business model of web services may prove attractive in the future, all of the examples presented deal with solving problems from within your own system. It should also be noted that all of the scenarios described, be it a simple Web Service or something as complex as GeoLogic.Net, require a special license from Esri to "Web Enable" the ArcObjects components.

Section 1 - Web Services using ArcObjects

The concept behind a Web Service is nothing new and is really just a modern incarnation of old RPC (Remote Procedure Call) techniques. Computer A wants Computer B to do some chunk of work on its behalf because it does not have the necessary instructions or does not have access to data needed to complete the work. What is new in the process is an industry wide standard for exposing and consuming these services, a critical piece that was missing in older attempts at integrating computer systems. XML, via the SOAP protocol, has become the universal standard for the transfer of instructions, data and results across machine boundaries. Previous attempts at defining standards for remote calls like DCOM or CORBA suffered from rigidity, proprietary protocols, and overly complex standards. SOAP replaces these with a simple, open architecture that makes extensive use of existing Internet protocols. It is these universal protocols that help make the XML/SOAP combination so effective, it is built on accepted and proven technologies. For example, Web Services traditionally use HTTP as the transport mechanism, which allows it to move freely (relatively) through corporate firewalls. In previous systems, piercing the firewall represented a major stumbling block for both administrative and security reasons.

Figure 1- Any SOAP aware client can consume GIS functionality that is exposed by GIS Web Services

Figure 1- Any SOAP aware client can consume GIS functionality that is exposed by GIS Web Services

TopologicalOperator Web Service

.NET makes creating Web Services almost effortless. Using the Visual Studio.NET (VS.NET) IDE, you can create a simple Web Service in minutes, relying on the ASP.NET framework to publish and host the service. As an example, a simple GIS Web Service can be created that exposes basic ArcGIS spatial operations like Buffer or Union. We want to expose this functionality to other computers that do not have any GIS software installed on them but have a need for performing spatial operations. .NET allows us to wrap the GIS functionality and easily expose it to any client that can connect to the server using the HTTP/XML/SOAP combination.

The code in Figure 2 illustrates how little is needed to expose the functionality as a Web Service. A VS.NET Web Service solution is created and is used to host the single C# class. ArcGIS as well as the Esri ArcObjects.Net Interop assemblies must be installed on the same server where the Web Service will be hosted. Make sure to make the appropriate references to the ArcObjects.Net assemblies before compiling any code.

The single SDLTopologicalOperators class derives from the .NET WebService class, it is through inheriting all of this existing functionality that creating new web services becomes so easy. Add public methods that decorate themselves with the [WebMethod] attribute to expose the method signiture to the public. Compile the application and let VS.NET do all the publishing work for you.

public class SDLTopologicalOperators : System.Web.Services.WebService
{
public SDLTopologicalOperators()
{
InitializeComponent();
}
#region Component Designer generated code

[WebMethod]
public string Buffer(string featureXML , double distance)
{
ITopologicalOperator pTopo;
IGeometry pGeoOut;
IGeometry pGeo = GeometryHelper.GeoFromXML(featureXML);

pTopo = (ITopologicalOperator)pGeo;
pGeoOut = pTopo.Buffer(distance);

return GeometryHelper.XMLFromGeo(pGeoOut);
}


[WebMethod]
public string Union(string featureXML1, string featureXML2)
{
ITopologicalOperator pTopo;
IGeometry pGeo1 = GeometryHelper.GeoFromXML(featureXML1);
IGeometry pGeo2 = GeometryHelper.GeoFromXML(featureXML2);
IGeometry pGeoOut;

pTopo = (ITopologicalOperator)pGeo1;
pGeoOut = pTopo.Union(pGeo2);

return GeometryHelper.XMLFromGeo(pGeoOut);
}


}//SDLTopologicalOperators class
Figure 2 - Simple GIS Web Service to expose Topological Operators. Error handling and user validation have been removed for clarity.

To facilitate communication with another system that does not have ArcGIS installed, geometry data must be serialized to a format that both systems can understand. The proprietary ArcGIS format is obviously prohibitive so the next logical choice is ArcXML (or GML). The internal helper class GeometryHelper exposes static methods for transitioning between any IGeometry and their relative ArcXML representation. There are some IGeometry data types that cannot be represented in ArcXML but for this example, the basic geometry types are handled.

The calling application, be it javascript in a browser, a MapObjects application or another ArcIMS server, sends valid ArcXML geometry representations to the server by calling the appropriate method. If the calling application is built using .NET, you just need to reference the Web Service at design time. You can then use the methods as if they were any other local resource, the .NET framework will execute the method using the appropriate protocols on your behalf. You can also use SOAP toolkits provided by a variety of vendors or by using a lower level, send HTTP GET/POST requests directly at the server. ASP.NET, which is hosting your service, will interpret them and automatically hook up the request to the service.


http://www.gisasp.net/services/TopologicalOperators.asmx?Buffer&featureXML=&distance=10&
Figure 3 - Using HTTP GET to invoke a web service hosted in ASP.NET

The above example shows a relatively simple service that makes only cursory use of the ArcObjects components. In reality, more complex services will need to interact with a variety of components, some of which contain large object graphs themselves. A good example of this is the Map CoClass, which is primarily accessed through the IMap and IActiveView interfaces. From these interfaces, a developer has access to many different ancillary objects like the individual layers, the display transformation or the graphics container. It is this incredible array of classes that makes ArcObjects such an attractive framework, however some parts of this design (as it exists today) can be problematic in multi-threaded containers like ASP.NET. The foremost problem is that most of classes in ArcObjects are marked as STA (single threaded apartment) and worse off, many of the more compelling classes are also implemented as singletons. While the "singleton" terminology is sometimes used in different contexts, in this case, it means that an object will only be created once per Win32 process space. As such, a singleton marked as an STA will be pinned to its creating thread by the SCM (Service Control Manager). Access to that object from other threads will be facilitated by the SCM using marshalling, an inherently resource intensive operation. Additionally, in certain scenarios, interaction with the objects can be thread unsafe and will lead to deadlock failures. In short, using ArcObjects in a multithreaded container like ASP.NET can lead to both poor performance and hard to reproduce crashes when concurrent traffic load increases. Poor performance and sporadic crashes will require additional physical resources and software licenses to guarantee the defined level of service, both of which represent additional financial burdens.

To circumvent these problems, all interaction with any object that is a singleton or is spawned from a singleton should be moved out of process to ASP.NET. While this adds to the complexity of the design, it provides another level of reliability in the form of recyclable object pools, in addition to sidestepping issues of thread safety. Design of the out of process worker classes will be discussed in the next section. Do not underestimate this problem.

It is likely that in future versions of ArcIMS, Esri will expose a large variety of GIS Web Services. Instead of exposing low-level geometry services, you will build on the Esri services to create more specialized ones. For instance if you were a local municipality, instead of creating a generic Buffer service, you might create and expose a 200-foot abutters list service that accepts parcel identifiers or postal addresses. Other software that is not GIS enabled could simply use the service to query for a list of valid tax records. Unlike traditional html techniques, the Web Service provides a formalized interface (via SOAP) for the calling application to use; the call to the service is as natural as a call to a local resource. For the immediate future, using .NET and raw ArcObjects provides a way to quickly expose GIS functionality. Additionally, at this time, it is unclear how Esri will expose the bulk of ArcObjects in the context of ArcIMS. Some services may need the fine grain control that only raw ArcObjects exposes, in these cases, using ArcObjects directly may be the only solution to the problem.

Section 2 - .NET Remoting in ArcMap

.NET Remoting is a powerful technology that is sometimes overshadowed by the incredible wave of industry hype around Web Services. On a conceptual level, Web Services and Remoting are the same; a remote computer is doing some chunk of work for your local application. Remoting, however, provides a far more flexible set of services that includes robust security, two-way events and the ability to pass complex object graphs between both client and server.

Remoted objects can be hosted inside IIS (5.0 or greater) or reside as just another process running on the server. When hosted inside IIS, Remoting uses the HTTP Channel, allowing it to safely traverse the firewall, in addition to gaining the authentication services of IIS. Remoted objects outside of IIS can use the TCP Channel and as such, can have slightly better performance. Either way, by using the binary formatter, Remoted objects will always perform better then Web Services which need to serialize to XML, a decidedly more verbose format. The drawback to Remoting is that the .NET framework is needed on both the client and the server, however if this can be guaranteed, Remoting provides a better alternative to Web Services. It is unlikely that you would use Remoting to service thousands of unknown clients but it is very likely that you would use Remoting to service a well-known network, your intranet, extranet or local LAN for example.

Remoting, when used in the context of ArcMap client, allows for some very interesting scenarios. Some scenarios gain access to functionality that does not exist on the client machine while others will use superior server processing power as an adjunct to client processing. The former will be discussed next, in the "Remote Editor" example. The latter will be discussed in the next section, where the GeoLogic.Net software allows a single user to harness the processing power of large server farms.

RemoteEditor - .NET Remoting in ArcMap

A good example of using Remoting from within ArcMap can be seen with the RemoteEditor extension. The basic concept of the extension is to allow multiple ArcView 8.x clients the ability to edit ArcSDE Geodatabases using stateful transactions that can interact with the Geodatabase’s versioning and conflict detection engines. It accomplishes this by using client side resources to cache user edit information, which are then sent to the server for commitment to the database. If conflicts are detected on the server, the resulting information is sent back to the client for further user interaction. The .NET Remoting architecture greatly simplifies the communication process by seamlessly allowing entire object graphs to traverse the network and by allowing for remoted objects to be passed by value and by reference.

An interesting side benefit of using a remote editing workspace is the ability to work with ArcIMS Image Services on the client while still being able to edit the ArcSDE layers that makeup that service. By managing the editable workspace (IWorkspaceEdit) from behind the firewall, the server component can interact with the Geodatabase as it normally would on the secure server LAN, while the ArcMap client that is holding the remoted proxy may be connecting from across the Internet. The benefit being a single ArcIMS service that can be viewed and edited from multiple, geographically separate ArcMap (ArcView) instances. Unlike ArcIMS Feature Services, the Image Service could be shared by browser based clients as well.

The RemoteEditor consists of three major parts, a client side ArcMap Extension (REE) and a server side component (RES) that handles the editing by managing a pool of worker processes. The third part is set of general classes that are referenced by both the client and server, essentially providing a common bridge between the two. Naturally, EsriCore is referenced on both sides as well.

Figure 4 - ArcMap RemoteEditor extension allows multiple ArcView clients to edit versioned ArcSDE layers

Figure 4 - ArcMap RemoteEditor extension allows multiple ArcView clients to edit versioned ArcSDE layers

The REE is a normal ArcMap extension that implements IExtension and IExtensionConfig. It creates the remoted object during the Startup method by pushing the actual request to another thread. This avoids potential network errors or remoting problems from blocking the main ArcMap thread during loading. Background threads are used in a variety of other areas in the extension and as a result, the IDllThreadManager interface is also implemented. Although the .NET runtime is reasonably adept at cleaning up errant resources like threads, it is good practice to clean up everything you create that contains known critical resources. IDllThreadManager just informs your extension when the ArcMap client is exiting, giving you a chance to clean up your threads. If you are using the ThreadPool for all your threads then you don’t need to worry about anything. Figure 5 shows the startup code for the extension, notice how little is needed to connect to the remote object.

//IExtension Interface Implementation
public void Startup(ref object initializationData)
{

m_pApp = (IApplication)initializationData;
m_pDoc = (IMxDocument)m_pApp.Document;

//Register the extension as MultiThreaded
IMultiThreadedApplication pMTA = (IMultiThreadedApplication)m_pApp;
pMTA.RegisterThreadManager((IDllThreadManager)this);

//push the network connect to background thread
ThreadPool.QueueUserWorkItem(new WaitCallback(connectToServer));

}



private void connectToServer(object state)
{

try
{
m_EditSession = new REEditSession();
m_ress = RemoteEditServerStatus.Connecting;
HttpChannel channel = new HttpChannel();
ChannelServices.RegisterChannel(channel);

m_RemoteEditServer =(IRemoteEditServer)Activator.GetObject(
typeof(IRemoteEditServer),
"http://10.0.0.1:1234/RemoteEditServer.soap");

m_ress = RemoteEditServerStatus.Online;
}
catch (Exception exp)
{
m_ress = RemoteEditServerStatus.Error;
}


}
Figure 5 - REE code to connect to the Remote RES server

The m_RemoteEditServer member level variable holds the reference to the remote RES object so the extension can now safely work with the object as if it were any other local object. The location of the object’s soap file has been hardcoded for clarity but would most likely be kept in an XML configuration file on the client so that the server location could be quickly changed without compiling.

REE employs several ITool classes that use ArcMap geometry and graphics resources to sketch new features or edit/delete existing ones. Their actions are recorded within a RESession object, which is eventually sent to the RES when applying the changes. Figure 6 shows the process of applying edits to the server; notice the ease of calling methods and passing complex objects to the server.

public void ApplyEdits()
{
if ((m_RemoteEditServer!=null) &&
(m_ress==RemoteEditServerStatus.Online))
{
//move to another thread
ThreadPool.QueueUserWorkItem(new WaitCallback(DoApplyEdits));
}

}



private void DoApplyEdits(object state)
{
try
{
//send edit session to server for processing
//session includes serialized versions of geometry
RemoteEditServerResults e;
e = m_RemoteEditServer.ApplyEdits(m_EditSession);
switch (e)
{
case RemoteEditServerResults.Ok:
m_EditSession.Edits.Clear();
break;

case RemoteEditServerResults.Conflicts:
IEditConflicts ec = m_RemoteEditServer.GetConflicts();
//show Conflicts dialog
break;

case RemoteEditServerResults.Error:
m_ress = RemoteEditServerStatus.Error;
break;
}
}
catch (Exception exp)
{
m_ress = RemoteEditServerStatus.Error;
}
}

Figure 6 - REE extension applying edits to remote server

The m_EditSession variable holds a collection of individual edits, each of which contains a serialized version of the geometry. For simplicity, the IGeometry members are serialized to ArcXML and then rehydrated on the server using the same GeometryHelper class from Section 1. A better method would be to implement the ISerializable interface instead of the [Serializable] attribute. By manually filling the binary stream, COM’s IPersistStream interface could be used, essentially keeping the IGeometry variables intact and more importantly, providing a mechanism for conveying complex geometry types, types that cannot be expressed in XML. Currently, COM objects with inherited interfaces cannot be automatically serialized in the remoting process.

Figure 7 - Remote Editor ArcMap extension

Figure 7 - Remote Editor ArcMap extension

The RES is implemented as a well-known, remote singleton object. Its main job is to forward edit requests and communications to and from the worker processes. As discussed previously, COM threading limitations present in ArcObjects produce serious problems in multithreaded containers. The RES needs to accept concurrent connections from multiple clients and as such, needs to spawn additional threads so that any one request does not block all other requests. To circumvent the ArcObjects threading issues, the RES creates a pool of out-of-process workers, each of which connects to the ArcSDE database. The state of any one edit session will remain for the duration of the client side edit session, with the RES continually redirecting client traffic back to its originating worker process. When the edit session is complete, the RES will free the worker process for use with another client. Unlike the worker processes that render maps in GeoLogic.Net, worker processes whose only job is to commit edits require far less processing resources. As such, a greater amount of workers can be placed in the active pool, essentially allowing more concurrent connections to be made. It is in this managed pool of worker processes that the efficiencies of Remote Editing can be seen; resources are acquired, used and then released without specific intervention from the end user.

As mentioned, the RES holds a collection of proxies to the out of process workers, each of which contains an interface to the worker process (ISDLWorkerProcess) as well as status information about its current state. This LIPC (Local Interprocess Communications) based remoting uses a custom channel created using Win32 NamedPipes. It is not necessary to create a full-scale remoting channel, however it will make working between the processes far easier. Other IPC techniques such as memory mapped files or the standard .NET TCP Channel can also be used. The TCP Channel has the drawback of having to go through the loopback adapter in addition to exposing another open port. This is perfectly acceptable when a separate computer needs to connect but is somewhat restricting when used between local processes.

The main RES object creates and manages both the pipes and worker processes on the fly, facilitating easy startup and teardown of errant worker processes. It passes the newly created pipe name to the worker process using command line parameters and then waits for the process to contact it back through the pipe. The contact comes in the form of the worker process remoting the RES, all be it from a different direction then the REE (inside the ArcMap client). Once in place, the RES can safely hook up edit sessions coming from the REE to worker processes in the pool.

The RemoteEditor extension is an interesting application of .NET’s Remoting technology that helps solve certain editing scenarios. It was created as a “proof of concept” application specifically for this paper and has not been implemented in any real world systems. However, because the technology of using shared, remote object pools across clients is valid, with some more work, the entire ArcMap editing process could be remoted. Coupled with the reach of ArcIMS Image Services, remote editing may help to solve some of the trickier "centralized location" problems like sharing read/write data sources over municipal, county or state boundaries.

Section 3 - GeoLogic.Net

GeoLogic.Net is a blanket terminology used to describe a variety of technologies used by SDL in building an ASP (Application Service Provider) for use by local government entities. The service will use ASP.NET, ArcIMS and various SDL created software to provide a complete and seamless GIS for the client. The main benefits for the client is the outsourcing of many complex portions of enterprise GIS. Specifically, hardware and software purchasing, security and backup services and most importantly, the qualified staff to keep the system running.

The validity of the ASP business model is best discussed in another venue, the main purpose of this section is to discuss the technologies used and more importantly, how they relate to the concept of completely remoting the ArcObjects components on the server. The use of ArcIMS with ASP.NET to serve multiple read-only clients will not be discussed, yet makes up a significant portion of the overall design. The vast majority of clients will still access municipal data via a web browser, however there are many other users who will need the feature set and processing power of a traditional desktop application. Editing, printing and geoprocessing are just a few of the GIS tasks that are far better suited in richer user interfaces.

The two primary technological benefits that GeoLogic.Net seeks to accomplish are automatically updated clients and true distributed computing power for the end user.

Auto Update Client

In the real world, nothing is ever perfect the first time it is created. Software flaws or changes in business logic are a constant in a world where exact specifications are the exception to the rule. No matter how well you plan for the future, you will eventually find yourself in a situation where the instant rollout of a software patch is necessary. Browser based applications are extremely adept at handling this type of scenario, change it in one location on the server and the fix is propagated to all of the desktops. This ease of administration allows for quick changes in reports, input forms or even how the measure tool works over a map. Where the browser model falls short is in creating rich user interfaces such as the editing tools in ArcMap or the dynamic cell arrays in Excel. Mixture of DHTML and script can go a long way but in the end, they do not provide the necessary base functionality to be productive. DHTML was the original design of SDL’s distributed rich client, Figure 8 shows the browser based DHTML client connecting to a series of web services to communicate with the GIS functionality on the server.

Figure 8 - Early DHTML client shown making edits against the remote ArcSDE database

Figure 8 - Early DHTML client shown making edits against the remote ArcSDE database

The system worked well in many situations, however, in some cases, the majority of development time was spent recreating Windows like UI controls. This type of functionality was not only available for free within .NET but was also vastly superior. In the end, .NET Smart Client deployment coupled with Remoting made for a far more attractive design; the deployment model of a web page with the UI power of a traditional desktop application. Figure 9 shows the new SDL distributed client, one built entirely in .NET. The tremendous benefits of working entirely with the .NET framework cannot be understated, however it should be noted that with this added flexibility came added complexity. Adding components for geometry, symbology and individual map layers increased the complexity of the design when compared to the DHTML client, however it added a new level of functionality. The best part being that as changes or fixes to any of the underlying components are made, they are simply posted to the SDL’s web server for propagation to the client. It is this "on-the-fly" software patching that brings to traditional desktop applications the deployment powers of web pages.

Figure 9 - GeoLogic.Net 'Smart Client'

Figure 9 - GeoLogic.Net "Smart Client"

It should be noted that Sun’s Java is also a valid target framework for this type of application, in fact, Java has been doing this for some time. However, there are two major stumbling blocks when using Java to build this type of application. On the client, Java does not take advantage of any native OS resources. Had Java targeted native resources in Windows, by far the dominant desktop OS, it may have been widely adopted as the premier way to build desktop applications. On the server, ArcObjects is a COM technology and as such, it meshes with .NET far better then it does with Java. Esri will most likely provide XML gateways in future versions of ArcIMS, however it is unclear to what degree the fine grain classes of ArcObjects will be exposed. Even if they are exposed via XML, going through an XML bridge for every method call could cause serious bottlenecks. In the end, it was decided that .NET is the better technology for the job.

Distributed Computing Power for the End User

In Section 2, .NET Remoting was used to harness server based editing sessions, in effect, giving the desktop end user efficient access to resources that it otherwise would not have been able to use. To a lesser extent, they also gained from the processing power of the server handling the editing request, freeing local processing power for other tasks or at the very least, keeping the UI responsive. What if nearly all time consuming tasks could be remotely executed, leaving UI responsive tasks like graphics manipulation to be processed on the client? It may be possible to push nearly all intensive GIS processing to the server.

For instance, when a map requests a full redraw, each layer of the map could be rendered using a different worker process on the server. Unlike the linear rendering methods that are traditionally used, the map would essentially be rendered in parallel with the resulting rasterized layer caches being returned to the client. As separate caches, the client can reorder, toggle visibility and change transparency of each layer locally, without a round trip to the server. Additionally, subsequent changes to individual layers, like symbology, can be retrieved for just the single layer and not the entire map.

This decoupling of the rasterized layer caches from their geometry backing stores is a concept that is used heavily in GeoLogic.Net to achieve the illusion of local, stackable layers. The layers are in reality, loosely held proxies to actual ArcObjects layers residing somewhere in the server farm. The MapRouter will attempt to direct subsequent requests back to their originating context, however it makes guarded decisions based on cluster statistics generated during heartbeats. It will always return a request to its origin IMap if it is marked as affined, as is the case during versioned edit sessions.

Figure 10 - GeoLogic.Net uses both local graphical resources and pooled server based ones

Figure 10 - GeoLogic.Net uses both local graphical resources and pooled server based ones

There are two different types of users in a distributed computing scenario. The first is one that is quickly pushed through server operations, the main goal being one of efficiency. By sharing server resources over many end users, benefits resulting from economies of scale are realized. This is the predominant design goal of many distributed systems, move as many end users in and out of the server as fast as possible. This is certainly a necessity for portions of the design but tends to overlook the second type of end user.

The second end user type is one that would benefit from the additional processing power of the server farm. Instead of allocating just enough resources to satisfy a typical "desktop" request, the end user actually receives far more server resources. Instead of drawing one map’s layers over the server farm, they are drawing several concurrent maps over the server farm. Examples where this may be helpful are side-by-side comparison maps showing different renderers or several maps showing varying temporal views of the same data. With their extents connected, a scale change in one map triggers a change on all of the other maps, with the requests executing concurrently over the server farm. Obviously, the total amount of processing time used during this request is far more then the average request but it accomplishes something that can be far more important: greater execution speed. Cheap, high performance Intel CPUs used in large rendering clusters afford their processing power to the single end user for any one increment of work, without blocking the main thread of execution for the UI. End users can simply work on a separate map while the other is being remotely rendered.

Taking advantage of server based processing is nothing new and several trends over the years have seen the balance of power shift between the client and server. With .NET WinForms and Remoting, the balance is certainly still shifted to the server, as is the case with browser applications, however a great deal of client side processing is also utilized. GeoLogic.Net seeks a balance of efficient server side processing while still allowing certain "power users" far greater processing power then they could have accomplished locally.

General client and server design

The basic design of GeoLogic.Net calls for using an object model that takes full advantage of the .NET Remoting framework. Classes are designed for three main duties: client side interaction, efficient network transit and server side communication with ArcObjects. Many objects, such as those found in the Geometry and Symbology namespaces, can morph into their corresponding ArcObjects equivalent using special server side helpers. This design choice is a tremendous help when conveying client side intent to the server side operators.

Once the client had contacted the front-end servers, their requests fall in one of three different GIS categories: Simple, Complex, or Module. All of these request types are shuttled through the MapRouter, a component that intelligently balances requests over the server farm. Unlike the simple pool manager used in the RemoteEditor sample, the MapRouter component takes far more variables into account before dispatching a request. Although their overall jobs are equivalent, that of shuttling requests between remote clients and worker processes, the MapRouter uses several assessment algorithms in the process. By maintaining detailed statistics about the status of all worker processes in the cluster, intelligent decisions about where to send the request can be calculated. Unlike other load balancers, the MapRouter needs to deal with both affined and non-affined requests. When marked as affined, a request is sent back to its creating machine->process->thread. Otherwise, it is balanced over the rest of servers using the above algorithms.

Simple "round robin" dispatching only works efficiently when each increment of work is roughly equivalent, however it begins to fail when requests become variable, as is the case when worker processes are handling several diverse GIS tasks (Simple, Complex, Module). Any one request may be executing something as simple as spatial operators or as complex as rendering features, two actions that may take dramatically different time scales to complete. The worse case scenario being several long requests queued in one process with several short requests in another. The MapRouter is designed to avoid these "bunching" problems across the cluster.

Simple requests use ArcObjects but do not need either a fully loaded IMap stack or individual ILayer objects. The TopologicalOperator Web Service described in Section 1 would be considered a Simple request. Simple requests execute at the boundary of worker processes and never enter the main queues, thus avoiding additional overhead.

Complex requests require an IMap, IWorkspace or an ILayer, all of which are considered expensive resources on the server. As such, they are acquired only by sitting in the process queue, waiting for the critical resource to be available. The status of the queue is reported to the MapRouter during a heartbeat that fires across the farm two times a second. Drawing layers, searching featureclasses or simple editing are considered Complex requests.

Module requests are essentially Complex requests except they execute custom module logic instead of commonly shared logic like layer rendering. Anything that would be placed in an ArcMap IExtension would be considered a Module request. Because of their arbitrary nature, these type of requests are in part executed using .NET’s Reflection functionality. As is the case with all requests, context based session state is available for the server farm.

Finally, many requests are completely unrelated to GIS. Entry forms, reports or login components are examples of logic needed by the client that does not need ArcObjects. These request are executed in the front-end IIS servers. The vast majority of these other requests use the functionality of ASP.NET, however some return custom objects. For instance, ASP.NET pages are used to power most of the Info Pane requests, the functional equivalent of ArcMap’s Identify window.

GeoLogic.Net is a work in progress, while a large amount of the system is functional, more work still needs to accomplished. However, it does represent the core of how SDL’s ASP will operate. It is believed that through a combination of browser based clients and .NET Smart Clients, the majority of a municipalities GIS needs can be accommodated.

Conclusion

Web Services and .NET Remoting are two methods for linking both client and server together. The former will help link dissimilar systems together while the latter will be beneficial to closed systems. With the advent of inexpensive processing power, distributed computing can be extremely useful in solving many problems that exist in today’s enterprise GIS.

As is the case with nearly everything in .NET, the equivalent design can be accomplished using traditional C++ methodologies. With enough time, money and manpower the identical .NET tasks can be realized. What .NET brings to the table is the incredible productivity the languages and more importantly, the framework afford the individual developer. This is true whether designing standalone desktop applications or headless server processes and is especially true when designing truly distributed applications that make heavy use of remoted object models.

Both .NET and ArcObjects are remarkable technologies that can greatly simplify the process of building large-scale GIS applications. Changes in deployment techniques will ultimately lead to lower costs for desktops while still providing rich, end user interfaces. SDL hopes to leverage these techniques when building enterprise GIS systems in the future and capitalize on two product lines that are just now in their infancy.

References



Brian Flood
Spatial Data Logic Inc.
bFlood@spatialDataLogic.com
http://www.spatialDataLogic.com