Introduction To Dataworks Enterprise Cache Object Model

Many of the properties of a field are derived from rules that are configurable on a field by field basis. For example, the rule that allows a field to determine its direction of update. Fields also provide a standard display method that allows the field to paint itself. For instance, a field can be painted by simply asking it to display itself in a given space. Standard colouring conventions can be applied.

The Cache also supports page data. Each field has an attribute that indicates whether the field supports atomic or partial update semantics. Most standard fields are atomic in that a new value will completely replace the existing value. Page-based data requires Cache support for partial update semantics in which new updates may partially update existing content. Updates to partial update fields are specified using a specialised ANSI derivative. The ANSI encoder/decoder �understands� standard ANSI including a number of vendor extensions to support internationalisation and other features. The interpreter also supports a proprietary TOSC extension to allow applications to define a variety of page layout forms including:

The Cache provides automation objects to create and interpret page data. Page fields can be painted using the standard field painting routines. This simplifies the creation of display applications, especially page displays.

Typically, a source application, be it a distribution system plug-in, a feed or some provider of data, normalises all data into standard Cache representations. This often involves encoding ancillary information, such as the meta-data that accompanies data, into standard fields. This strategy ensures that no incoming information is lost between the source application and the application-level functionality. Equally, client applications that intend to pass the information to other systems will decode the meta-data and standard Cache representation to the native system they must support. Note: there is no need for applications to provide separate caching of data since this function is performed for them.

Implementation Technology

The Real-Time Cache consists of a single COM DLL located in the Windows System directory. The DLL is an in-process server containing two �controls�, a �connectable� Cache manager object and a number of other COM objects. In addition to maintaining the Cache, the Real-Time Cache component manages the IPC mechanism used to transfer data between processes; the maintenance of configuration information; application error logging and a number of other related tasks.

Dataworks Enterprise programmable objects are all implemented using a combination of standard C++, SCL and ATL. Each programmable object in the system is a dual interface COM object. Where appropriate, some objects support connection points (events) and some are fully implemented ActiveX controls. ATL is employed to minimise the size of the overall system.

The Cache configuration is kept in one or more standard configuration files. The Cache configuration manager allows configurations to be applied on a site, group or user basis. See guide to configuration for more details on the mechanisms used to configure TOSC components.

Cache Object Model

The Real-Time Cache has an extensive object model. The object model supports Automation and can be directly used from VB, VBA and VBScript-based hosts. As part of the final distribution, a set of C++ wrappers is also provided to allow the Cache to be used from C++ application code. In addition, third-party tools and interfaces can be used to access Cache functionality from other languages. Appendix A provides more information on the use of the Cache in different language environments.

The Cache Object Model as viewed in the Visual Basic Object Browser

The DLL contains two full controls, RTServer and RTClient. The RTClient control allows users to subscribe to real-time data. Conversely, the RTServer control allows users to publish data either locally to the Cache or to the distribution system or to a publishing server. These controls can be directly inserted into VB forms or, in C++, can be instantiated directly using the CRTClient and CRTServer classes.

Client Control

The client control allows users to subscribe to information from the Cache. Information in the Cache is presented in the form of records and fields. Pages are simply records with a field that contains ANSI encoded data.

The Client Object Model

The client control has a collection containing the current sources in the system. This is used mainly to provide user feedback on the types of data currently available. Each source object contains information about the name of the source, its type and so on. Note that this object is identical to that used on the server, except that the client cannot change the source object properties.

Applications use the client to construct (or factory) record objects. This is the first stage in making a request for a record. The records are initially not bound to any real element in the Cache. The user sets the source, instrument and, in some cases, the options properties of the record and then binds the record to the Cache. The act of binding the record will cause the request to take place. In Visual Basic, this appears as:

Dim RecordList As New RTRecords

Private Sub RequestRecord()
   Dim record As RTRecord
   Set record = RTClient1.CreateRecord
   Record.Source = �Worldscope�
   Record.Instrument = �ICI�
   Record.Bind
   RecordList.Add record � to avoid the record being deleted
End Sub

The record will remain in existence until it is no longer referenced by the application. Note that, in the above example, the record object is placed in the RecordList collection to stop it being deleted when it goes out of scope of the RequestRecord subroutine.

Prior to binding a record, the client application should set its Source and Instrument properties and, optionally, its Options property. The Options property is a string that is in most cases empty. Where used, options are conventionally the Text property of a RTDataRef (see below). The Cache system uses the options as part of the namespace of the source. In other words, two records with the same Source and Instrument properties will be treated as distinct if their options fields are different. Note also that the server can set both the Instrument and Options properties on an Image and thus the value of these properties can change when an image is received.

At a later time, the record will be populated with data from the Cache and a client event generated providing both the record and the data for the record. NOTE: the system is completely asynchronous, the data will not be available until the image (or status) is received. The record can be unbound and bound to a new source and instrument name.

Records have an image in the Cache that can be accessed via the Fields collection described in detail below. When events are received, the part of the image to which the event applies is supplied in the event parameters. For instance:

Public Sub RTClient1_ModifyFields(ByVal Record As RTRecord, ByVal Fields As RTFields)

This is the event for updates arriving for a record. The Fields argument contains the fields that have changed. (Note that in VB the type statements may be prefixed by the library name, as in �ByVal Record as RTCtl.IRTRecord�. This does not, in anyway alter the functionality of the events). The complete set of fields for a record is contained in the Fields property of the record itself.

The RTFields collection of a record is identical on the server and client sides and is described below.

The client control also provides a synchronous mode of operation through a combination of the Blocking property and a Wait() method. The Wait() method allows the application code to wait for an internal data event or a timeout to occur (whichever comes first). This allows applications to request data and programmatically wait for a result. The Blocking property automatically enforces a wait whenever any data is access from a record in the cache. See the section below �Asynchronous vs. Synchronous Operation�.

Server Control

The server control allows users to publish information to the Cache. Information is published using Sources. A source is deemed to control a namespace from which instruments (or items) can be requested. In other words, items from a source are uniquely identified by an item name. Sources are created via the RTServer control. Each source has a name, a data type (e.g. Pages, Records, etc) and a source type (e.g. whether the source is to be exposed to the distribution system).

Note: sources are created using the CreateSource method of the server object, which returns the source. Like records, the source object must be stored by the application. For example:

Dim MySource As RTSource �Global to keep it in existence
 
Private Sub MakeSource()
    Set MySource = RTServer1.CreateSource(�Name�, �Record�, RTStandardSource)
End Sub 

Note that the source object is made global to ensure its lifetime will be for the lifetime of the form. Alternatively, it may be placed in a standard Collection object.

The Server Object Model

Each source has a list of items. An item is a server-side representation of a record. Items are associated with a source and have a name, which is unique for the source. On creation they have an empty list of fields. This field list is known as the current item state. Servers change the content of an item by:

The server performs each operation by applying a Fields collection (of Field objects) to the item. This is the same Fields collection class as used by the client. See below for more details.

The Real-Time Cache supports both client pull and server push mechanisms in order to populate the Cache. This only affects the mechanism used to create the initial item. In all cases, the server places content into an item using the same set of methods described above. Servers may elect to use both techniques.

In the client pull model, the server application creates the source and awaits requests from the client. At the point the Request event is delivered, the item is passed to the server application for receipt of data (it is already in the internal Items collection). This item will exist until the client application cancels the request. Upon cancellation, the server is notified through a Cancel event (to allow it to free up any external resources). The item is then removed from the internal Items collection, and deleted.

In the server push model, the server simply announces the existence of an Item by asking the Items collection to add a new item with a given name. This new Item is returned and the server can supply the item specific content. In this case the Item stays in existence until the server removes the Item from the Items collection. Subsequent client requests are satisfied directly from the item content retained within the Cache.

Like the client control, the server control also provides a Wait() method with the same functionality and the same constraints. See the section below �Asynchronous vs. Synchronous Operation�.

Hot Standby Sources

Typically, a Source of a given name is unique within the cache. All requests from clients for information are routed to the source. If more than one source of a given name are registered with the cache, the cache performs hot standby of those sources.

In hot standby, the first source is registered as the recipient of requests. Subsequent sources are placed on standby. If the server deregisters the original source, the cache searches for another registered source with the same name and registers it with the cache. Clients are automatically reconnected to the new source and operation continues. This process is known as a hot standby switch.

Hot standby switches occur either if the source is deregistered from the cache or marked stale by the server. There may be any number of hot standby sources in the system. Note, if all sources of a given name are marked stale (or disconnected), the client are notified that the source has become stale.

Fields and Field Information

The structure of fields is shared between the both sides of the system. Typically, servers create fields associated with items and clients retrieve fields associated with records.

The Field Object Model

It is frequently the case that there are two RTFields collections in use at any time. For instance, the Fields property of a record is a collection that refers to the entire current state of that record (the image). Many of the client events and server methods also specify a Fields collection that just represents the fields that are being operated on. For instance, in the context of a client ModifyFields event, the fields in the argument are the fields being modified, whereas the Fields property of the record is the image after the modification has been performed.

Each field in the Fields collection contains a set of properties including the name(s) of the field and its current value. For each field there is a range of representations of the name and the value.

Field Names

Each field may have a number of names to be used in different circumstances. Each field has:

Field Values

Fields have a �binary� and a display value as separate quantities. The binary value (represented as a variant) contains the actual value of the field and is used in calculations, etc. The display value is the textual value to be displayed to a user and may contain expanded fractions, etc. The binary value is accessed through the Value() property. The text value is held in a raw encoded form as the RawDisplayValue property.

Typically, when displaying a value to the user, an application would use the Text property. This checks the display value. If there is a display value, it is decoded and returned. If there is no display value, the Text property coerces the binary variant to get a textual representation of the binary value. Alternatively, the field has a Paint method, which will apply standard rules to paint the field.

Field values can carry a variety of types of information as defined by the Automation VARIANT structure, including null values, errors, integers, dates, times, currency values, floats, booleans, arrays and objects. However, there are limitations on the carrying of objects through the Cache. Objects are serialised using their proxy reference. When recovering the object from the internal Cache stream on the client side, the object is re-instanced using the proxy reference. The cache attempts to create a proxy for the original object, which may be in the same physical host (using COM) or a remote object on another physical host using DCOM. In the event of a (DCOM) permissions error or an inability to communicate with the remote host, the attempt will fail.

In addition, fields can only be placed on a native distribution system in the formats supported by that system. Depending on the capabilities of the distribution system, Dataworks Enterprise plug-ins (i.e. adaptors that interface with these distribution systems) will coerce types to forms that can be represented on that medium. For instance, an object may be serialised on the distribution system as binary data, tagged in some way to allow the plug-in to reconstruct the original object or a reference to it.

Page Data

Pages are also kept in a single field as ANSI encoded data. Getting the raw value of a field will get the ANSI data from the field. The Text property will return a string representing the whole page. In addition, the field has a DisplayData object that is used to break down the page image into rows.

The DisplayData object has the width and height of the page and a collection of rows, one for each row of the page. The display data also contains the default ANSIFormat object that encapsulates the formatting information for a cell of a page. The ANSIFormat object in the DisplayData represents the default for the page.

Each row object contains the text of the row and two collections of RowChunks, one for the current image of the row and the other for the current updates for the row. A RowChunk is a parcel of text containing common attributes such as colours, intensity, encoding, etc. Each RowChunk object contains the text of the chunk and the ANSIFormat, which applies to that text.

ANSIEncoder is a server-side object that facilitates the building of ANSI pages. Typically, an application creates the encoder, adds text and vertical bars with accompanying format objects and then requests the encoded buffer of ANSI data that is applied to the field using the Buffer property.

Like non-page data, the Paint method can be used to paint the page into a window for convenience.

Field Static Data

The field also supplies properties retrieved from its configurations. This includes whether the field is a Symbol field (displays up/down arrows), whether it is a well-known field such as BID, ASK, LAST, etc., and if it is a link, the link number.

Permissions System

In accordance with other systems of this type, the cache is able to propagate permissions provided either by a feed, a host application or by a plug-in. As far as clients are concerned, the operation of the permissions system is, for the most part, automatic.

In most distribution system architectures, each item has permissions information associated with it at the point of entry to the distribution system. This is usually performed by a feed handler on the basis of information either acquired from the feed or through separate static files. The feed handler places the permissions information onto the distribution system prior to passing data associated with the item. The permissions are applied by client software as the data is taken from the distribution system. In most cases, this is performed automatically by the distribution system infrastructure. Most distribution systems also have some interfaces that allow the customer or the vendor to interact with the system to set user permissions to data. Sometimes permissions need to be combined as in the case of a composite page server. Usually, the page server uses standard interfaces provided by the distribution system vendor to combine this information.

Dataworks Enterprise APIs allow a feed handler to create permissions data through the standard RTPermissions and RTPermission objects. Each item has a RTPermissions object that, by default, is empty, i.e. no restriction on permissions. Items can have zero or more individual RTPermission objects added to them, each permission object carrying a set of source specific product numbers to which the data belongs. A feed handler typically adds a single RTPermission object to the items RTPermissions collection and adds individual product numbers to that single RTPermissions object. The model here is that each item is considered to exist in one or more products as specified by the feed vendor. Access to the item is governed by whether the user has access to a product to which the item belongs. When adding a RTPermission object, the feed handler assigns the permission object an �authority�, which is simply the name of the original source and item being added to the system. This allows the system to:

Decode the permission product numbers

Combine multiple permissions from different authorities

Update permissions data from any single authority.

There can only be one RTPermission object for each authority for each item, but many authorities can apply permissions to the same item. In this way the system can support composite records sourced from many items and still maintain the underlying permissions data.

After editing the RTPermissions object, specifically on the next item update or image, the cache converts the item permission objects into a single field called RTPERMISSIONS, which is propagated, through the cache using the standard field mechanisms.

Feed handlers can update the permissions in the normal way by retrieving the permission object (with the appropriate authority name) and applying new permissions information to that object. In this case, a new RTPERMISSIONS field will be created and passed to clients. Feed handlers can also delete an RTPermission object by removing all product numbers from the record.

As with permissions objects, multiple RTPERMISSIONS fields can be applied to a single item. In this event, the cache will decode each incoming RTPERMISSIONS field, aggregate the new permissions with its internal representation and regenerate the RTPERMISSIONS field from the collected data.

Clients can query permissions information by retrieving the RTPermissions object associated with a record. Updates to permissions are indicated by receipt of an update to the RTPERMISSIONS field. Applying an RTPERMISSIONS field from one item to another causes the information to be unpacked and applied to the new item. Thus, if a client is a simple mirror, permissions are propagated automatically using the RTPERMISSIONS field.

Plug-ins are responsible for converting between standard cache permissions records and the representation used in the native distribution system. When a plug-in retrieves an item from the distribution system, it takes any permissions data associated with that item and, on its own authority, creates a permissions record for the item. When a plug-in is required to place an item onto the distribution system, it takes the permissions associated with the item and combines them in a single lock in native distribution system format. In most cases the lock will be a single permissions entry supplied by the original provider of the information. However, in some cases this lock may consist of all the individual locks for data that provided data to the instrument. The Publishing Server automatically performs the same function of combining locks using the RTPERMISSIONS field.

Cache and Trace Management

The DLL also supports internal Cache management objects plus objects designed to support logging and the management of those logs.

The Cache and Trace Management Object Model

Trace System

The Cache trace system controls the logging functionality for both internal Cache modules and for external applications. The RTLog object is used by applications to perform logging functions. RTTraceItem and RTTraceMgmt are used to control the amount of logging output by the trace system and to control where those logs are written.

The application wishing to log activities creates an RTLog object and sets the name property to identify its output logs. If two applications use the same name, they will share logging levels. Each log message is accompanied with a level number between -1 and 9. Logs of level -1 indicate a fatal fault has occurred and is never omitted from the file. Equally, logs of level 0 are also never omitted from the log file and are used to indicate warning conditions. Logs of level 1 to 9 are general informative messages with increasing detail and decreasing importance. These can be selectively added and removed from the trace file.

The TraceMgmt object controls the level of logging for each module and the name of the file to which the logs are sent. If there is no file specified, the logs are sent to the debug terminal or dropped if there is no terminal running. It has a collection of TraceItem objects, one for each of the registered modules in the system.

Cache Management

The Cache management system provides statistics on the internal operation of the Cache. The manager object provides two events:

An update event that allows applications which are logging statistics to write a new log on a regular basis. This event is generated every three seconds.

A source down event that is generated every time a source is disconnected from the Cache. This can be used to generate operator alerts.

Each Cache manager has an associated collection of RTSourceStats, one for each source present. This provides general current and historical information regarding the state of the source, such as the number of failures, numbers of items watched, number of snapshots, etc. This object also contains a list of RTReqStats objects, each element describing a Cache entry.

In addition, the Cache manager also supplies a statistics structure containing details of global Cache state, such as amounts of shared memory used, lengths of queues, numbers of clients, servers, etc. The example application Stats.exe illustrates the use of these statistics.

Users should also note that each Source in the system supports a special record called �WATCHLIST�. This contains a list of the items currently in the Cache. This record is automatically generated by the Cache, and updates in real-time. Low-level programs simply wishing to monitor the content of the Cache for a given source should use the WATCHLIST record rather than traversing requests using the Cache manager.

Miscellaneous Elements

The DLL contains a number of miscellaneous elements. These include RTMarker, RTSurface and RTFunctions.

Surfaces and Markers

Surfaces and Markers are used to paint data into screen real estate. A surface represents a Windows device such as a screen window onto which painting occurs. A marker represents the drawing attributes, such as colour and font, to be used to perform the painting. Both objects are used as arguments to the RTField object Paint method to allow the programmer to automatically paint the field (or Page).

RTFunctions

The RTFunctions module exposes a series of global utility functions. In Visual Basic, these appear as built-in functions. In C++, they appear as methods of the RTModule object. These functions include mapping of functions relating to getting and putting configuration information into the central configuration database, conversion JIS/CNS to ShiftJIS/Big5 and a function to map ANSI colours to standard Windows forms.

Additional Considerations

Data References

Many objects in the Cache have a property called Reference or References. This is used to get a description of the objects content for storage purposes. The Reference property returns a single RTDataRef object. The References property returns a RTDataRefs collection. Which of the above-mentioned properties is available depends on whether the object in question represents one element, e.g. an individual field, record or source, or a collection of elements, e.g. a fields collection.

Each RTDataRef object is an associative array that can map strings to strings. Each element of the associative array is called a property. Elements are placed in the array using the Property() property, i.e. ref.Property(�Rating�) = �A+�. The element can be retrieved using the same property, i.e. rating = ref.Property(�Rating�). Many commonly used reference elements are given their own names (which always have a single character element name), e.g. ref.Source = �ExchFeed�, which is a synonym for ref.Property(�P�) = �ExchFeed�. The RTDataRefs object is a standard automation collection of RTDataRef objects, i.e. an array of associative arrays.

Data references have the ability to serialise their content to and from strings using the Text() property. When reading the text property, the entire content of the associative array(s) is encoded into the string returned. When that string is applied to a reference, the entire associative array is reconstructed from the string. This makes them very easy to use in a variety of circumstances such as file storage and data transfer operations. In addition, the Cache frequently uses references where arbitrary collections of strings need to be manipulated, such as encoding optional request data and system command texts and when enumerating configuration information.

Tags

Most objects in the Cache have a Tag property. This should not be confused with the VB Tag string property created for all controls by the Visual Basic object extender, although its purpose is similar. Like VB, tags are used to store user data associated with an object. Unlike VB, the Cache Tag properties are defined to be variants allowing them to store strings, numbers and even references to other objects, making them significantly more flexible in use. Also they are not only defined in controls, but appear in a range of objects in the system.

RTField Caching

The Cache maintains an internal image of field data at all times allowing the application to retrieve the current state of the field. When a client accesses this internal representation, the Cache returns a COM-based RTField object which wraps this internal representation.

Clients and Sources provide a property (CacheRTFields) that allows the application to specify the lifetime of these RTField objects. This property is typically set once at startup for each client and server. By default, RTField objects are constructed on the fly as a result of a need to create them. This significantly reduces the memory overhead of the Cache and may, as a result in the reduction in size, improve performance by avoiding the need for additional paging. This is particularly effective where the application is dealing with large quantities of data and does not reference its own Cache very often.

A side effect of generating RTField objects on the fly (the default) is that it is possible for there to be more than one RTField objects relating to a given cached field. For instance, suppose that the �BID� field of a record has been modified. The Cache generates an RTField object to pass to the user in an event. If in the context of the event, the user requests the �BID� field from the currently cached image(e.g the records Fields collection), the Cache will generate a new RTField object to wrap the result of that request. The two field objects are different, but they refer to the same content.

However, in certain exceptional circumstances, it may be more efficient to allow the RTField objects to have the same lifetime as the internal field representations. This may be true where the application uses very little content and constantly refers to its own Cache images, such as a client display application that implements its own painting routines using the Cache image. In this case, for any given internal field, their is one and only one RTField object, irrespective of context.

Asynchronous vs. Synchronous Operation

In most circumstances, the Cache should be operated using asynchronous operations. When a record is requested from the Cache, the request method returns immediately. When the data arrives, the application is notified of the completion of the request. As updates arrive, the application is further notified of the changes. When the application is no longer interested it simply deletes the record. This mode of operation combines the best Cache performance with the ability for applications to service other functions (such as user input, a serial feed or some other resource) without additional threads of execution.

However, in some cases, a synchronous approach may be an absolute necessity. Dataworks Enterprise COM API supports both synchronous and asynchronous retrieval of information. As stated above, asynchronous mode is the most efficient and the default mode of operation. However, some environments, such as Web Servers, require a synchronous mode of operation. The problem with synchronous mode is that making each request to the back-end system synchronously has a massive impact on the overall performance of the application. Consequently, in Dataworks Enterprise, synchronous mode is actually only partially synchronous in that requests are made asynchronously and results are collected in the cache for use by the application also asynchronously. Only access to the local cache elements is made synchronously.

For instance, suppose that the application needs to request 50 individual records in order to populate a web page. With Dataworks Enterprise all fifty requests are issued asynchronously. The application then attempts to start outputting the page. When it attempts to access the first record, Dataworks Enterprise blocks until that record is available. However, in the meantime the other 49 records are also arriving in the cache, so that when it attempts to access the second record, it is probably already there (or in transit).

This has the feature that it is both fast and scaleable whilst still supporting the synchronous execution paradigm.

The client and the server objects also expose Wait() methods. The Wait() method allows an application to idle until either a specific timeout occurs or some Cache action is performed (such as receiving data) whichever is the sooner. Note: using Wait() can make the application become recursive in unexpected ways and care should be exercised when using a combination of asynchronous and synchronous operation, e.g. using a combination of Wait() and event handlers.

Statistics Collection

The cache collects a range of statistics during its normal operation. These statistics relate to the internal operation of the cache as well as statistics associated with sources mounted on the cache. These statistics are available through the standard API and can be viewed using a statistics tool supplied with the system. In the future, these statistics may be integrated with standard diagnosis tools such as PerfMon or SNMP agents.