Dr. Ghulum Bakiri
Abstract:
PBX-RMMS is a comprehensive system developed by MicroCenter (Bahrain) for the remote monitoring and maintenance of a large number of PABXs installed at geographically dispersed customer sites but owned and operated by the Bahrain Telecommunications Company (BATELCO).
The system is built around the client-server model and consists of a number of client modules (one for each client PABX that needs to be monitored) and a Center system - located at BATELCO’s headquarters at Hamala.
The Client modules are directly connected - each to a client PABX - capturing messages, warnings, and errors generated by each PABX in real time. Each client module can be customised to filter out the significant, serious and major errors and report these to the Center through a dial up telephone connection.
The Center system is extensive and handles numerous inter-related tasks. Its main function is to receive the errors reported by the various clients, alert the appropriate technician responsible for the particular client site and present a highly user-friendly interface for the visualization, interpretation and clearing of the reported errors.
This paper discusses the integration of the system with the ARCView2 desk-top GIS. Errors received at the Center are displayed in real time in GIS format, color coded by the severity of the error. ARCView2 allows users to focus at will on certain categories of errors, restrict attention to errors in a certain area or to selected important sites and provides an easy to use interface for the technicians to connect to any PABX to clear/rectify faults. Several other advantages of utilizing ARCView2 are discussed in the paper. The technology described can be applied to any type of system in which real time (or semi-real time) monitoring is required.
Telecommunication companies such as BATELCO (The Bahrain Telecommunications Company) normally provide telephone services to their larger clients (such as large corporations, banks, hospitals, government ministries, etc.) by installing sophisticated PABXs that handle hundreds or even thousands of lines directly at each customer location. Any failures in these PABXs (Private Automatic Branch Exchanges) will disrupt the telephone service in the entire organisation which can prove costly in terms of loss of productivity for businesses, or at times even dangerous (e.g. for hospitals). Hence, a modern PABX normally carries out extensive self tests continuously and reports the status of the system regularly on a continuous basis through messages and warnings that are displayed on a monitor attached to the customer PABX. The problem is that these warnings or error messages are left unnoticed since providing an experienced engineer at each customer site just to monitor and interpret these warnings and error messages is an expensive proposition.
As an example, loss of the mains power to the PABX (and the switch to the standby battery power) is a condition that is immediately detected and reported by the PABX. If that fault is noticed and attended to at once, the error can be usually corrected and power restored within hours --- possibly without any loss of service. However, the loss of mains power normally goes undetected until the backup batteries are fully discharged and service is lost - a panic and costly situation that could have been avoided.
The solution proposed and implemented in PBX-RMMS (Remote Monitoring and Maintenance System) is built around the Client/Server model and consists of a number of client modules (one for each client PABX that needs to be monitored) and a Center system - located at BATELCO’s headquarters at Hamala as shown in Figure 1. The complete system is developed under Microsoft Windows using C++ and Object-Oriented technology, coupled with class libraries for the database engines and the communication capabilities.
The Client modules run on standard PCs directly connected - each to a client PABX through an RS-232 serial connection (replacing the dumb terminals that were previously utilized to display messages and errors reported by the PABXs). The Client modules are designed to filter out error codes from normal (status reporting) messages, determine the category (severity) of the error through a table lookup in an error database, and report significant errors to the ‘Center’ through a dial-up telephone (modem) connection.
Additional details and features of the Client modules are presented in sections 4.1 and 5.
The Center system is extensive and handles numerous inter-related tasks. It is located at BATELCO’s head-quarters and runs as a set of inter-acting modules on a number of IBM-PC compatible machines connected in a local area network, as shown in Figure 1.
One of the main modules in the Center is responsible for answering the calls made by the clients, interact with them for handshaking and identification purposes, and log the errors reported in the ‘active faults’ database. Another module continuously scans this database and alerts the technicians in the Center through a voice module that sounds certain pre-recorded messages as appropriate. A third module presents the user with a highly user-friendly interface allowing various options such as assigning certain faults to a particular technician, getting more details about the particular error code reported or the customer site reporting the error, and allowing for the clearing of faults attended to by the technicians. Cleared Faults are transferred to a ‘Faults History’ database that can be used to generate various statistics and summary reports. Other modules take care of the creation and maintenance of various databases storing details of: Customers, Systems (PABXs installed, Number and Type of lines, etc.), Technicians, Errors and Error Codes as well as Faults History data.
In its second version, the Center system has been integrated with the ARCView2 desk-top GIS. Errors received at the Center are displayed in real time in GIS format on a map of the area, color coded by the severity of the error. ARCView2 allows users to focus at will on certain categories of errors, restrict attention to errors in a certain area or to selected important sites and provides an easy to use interface for the technicians to connect to any PABX to clear/rectify faults.
Besides allowing the user to turn on/off certain error categories at will, the GIS helps focus attention on possible external factors that may have caused the errors. (Multiple errors cropping up at the same time at sites geographically close to each other generally signal that an external cause must be suspected.)
Several other capabilities of ARCView2 are utilized to advantage.
The graphical editing tools are employed to create and store the physical layouts of cards in each of the PABX cabinets for each site. This relieves the technicians from having to physically open several cabinets to search for an empty slot (e.g. to place an expansion card).
The Hot-linking feature of ARCView2 is used to implement an intuitive interface allowing the technicians to perform various tasks (such as dialing-up and connecting to a remote PABX) by simply clicking on the site symbol.
Another feature that proved extremely handy is the charting capabilities of ARCView2. These capabilities are exploited to create visual representations and statistical summaries from the error history data that are accumulated on the various sites.
Finally, should the need arise for a technician to be physically present at certain sites to clear/rectify faults, the GIS allows the manager of the PABX maintenance section to assign maintenance crews to sites in an efficient manner that minimizes the travel time.
The complete system is highly modular and flexible allowing each user to load only the modules of interest to him at will. This section will briefly describe the function of each of these modules.
The client module is further decomposed into five sub-modules as follows:-
SerialCapture Module: Continuously monitors and captures the messages sent from the PABX through the RS232 port and places them in a queue, to be processed by other modules.
ClassifyError Module: Entries from the message queue are taken one by one and the error code is filtered out. The resident error database is then searched to find the severity of the error. If the error was found serious it will be queued for transmission to the Center. Otherwise, it is removed from the queue and discarded.
TransmitError Module: Continuously monitors the ‘Serious Errors Queue’ and if there are any entries, it dials up the Center to establish a connection. Once a successful connection is established, it communicates all the errors queued for transmission to the Center then hangs up. Transmitted errors are removed from the ‘Serious Errors Queue’, and kept elsewhere for a certain predetermined period before purging.
Host-Mode Module: This module is responsible for answering any incoming calls (usually from the Center) for polling (to make sure that the client-PC is OK), or, for remotely interrogating the PABX. In the latter case, all commands sent by the Center are passed to the PABX and all the PABX responses are relayed back to the Center.
The Backup Module: This module automatically activates itself at predetermined intervals to create an upto date backup of important information and settings of the PABX. The information to be backed up as well as the interval between backups are user defined in a text file and hence readily customisable.
Center Modules and their functions can be described as follows:
The DBMS modules allow standard database access, navigation and manipulations of various databases needed for the system. Facilities provided include searching for a specified record or navigating through the database-- based on any of the available keys as well as the normal Add, Edit, Delete, Recall for manipulating individual records in the above databases. Password management and security control modules are also provided. Various reporting formats of the available information are also made available.
The database available include the following:
Cutomer/System Details: Stores the remote site data such as customer name, address, telephone number, name and telephone numbers of the main account representative and the sponsor. Also, it includes the system specifications of the PABX installed at the customer site. The system category, number of lines, stations and the number of circuits in each category of lines as well as other details.
Technicians: Stores technician information including code, name, address, telephone numbers (office, home, pager, mobile), whether the technician is currently active or not (e.g. on leave) and remarks if any.
Faults and Faults History: The structure definition of both these databases are the same except that the Faults file contains only the uncleared faults. Information stored include error code and parameters, dates and times: reported, assigned to a technician and cleared as well as site and technician identification codes.
Error Codes: All the error codes that can be generated by the PABX will be stored in this database. Each error is identified by an error code, description, parameters and category (severity). This file is PABX system dependent and when another make of PABX is to be used, this table must be populated by the Error codes for that particular PABX make.
System Commands: This is a database of all the commands that can be issued to the particular PABX make including various Load commands, descriptions, responses and other information. It is used to assist the operator in selecting which commands are appropriate to be be issued based on the PABX responses.
It should be noted that several of the functions available in this module are also available through the GIS interface discussed earlier.
The overall reliability of the system is increased by allowing the clients to dial not just one Center number but a list of Center telephone numbers in a pre-determined sequence until a successful session is established. If the attempt to report the error(s) to the Center was unsuccessful for a pre- determined period, the client module dials the pager of the technician responsible for the site--and continues to try once again to establish a connection with the Center. During these dialing attempts (and throughout) the client modules continue to monitor all messages reported by the PABX, and if additional serious errors are received they are queued for transmission to the Center as soon as a connection is established.
Another feature of the client module is its ability to block errors that have just been reported to the Center within a pre-determined period from being reported again and again. This prevents jamming of the Center telephone lines from client modules connected to PABXs that periodically transmit an error situation over and over (e.g. every so many seconds or minutes).
Other features include the remote setup of any (or all) of the clients by the Center. This allows passwords or Center telephone number changes for example to be conveniently relayed to the clients without having to visit each client site.
Several enhancements to the GIS module of the system are planned for the next version. One such enhancement involves further customisation of the interface. Currently a query like: "Show me an empty slot to place an extra card" will have to be made through the ARCView2 query builder. Avenue scripts will be used in future versions to make these and other options readily available through customised menus/control bars.
Another option that will be added is an enable/disable button for each of the cards in the expansion slots layout diagram of the PABX. These buttons will be "Hot Linked" to scripts that will establish communication with the PABX and issue the appropriate commnads to enable or disable the card in question. This feature will allow faults to be cleared quickly from the Center by disabling a faulty card, until a more thorough and appropriate remedy can be excercised.