Programming Dataworks Enterprise

 

 

 

Contents

 

Programming Dataworks Enterprise. 1

Contents. 1

Introduction. 1

Some preliminaries?... 2

Obtaining data using a Dataworks Enterprise Client3

Building our own client4

Using RTClient Events. 6

Timing and Blocking. 8

Record lifetime and automatic request cancellation. 9

Status and Fault handling. 9

Other RTClient Features. 11

A last look at our program?... 12

Providing data using a Dataworks Enterprise Server13

Building our own server14

Imaging an Item.. 14

Handling bad requests. 15

Updating an Item?s data. 16

Item lifetime. 17

Pushed Items. 17

Terminating Items. 18

A last look at our program?... 18

 

Introduction

?Dataworks Enterprise? refers to all the modules & tools that collectively make up the entire Dataworks Enterprise Distribution as supplied by Thomson Financial.  Also included in a standard distribution is various documentation, configuration information, and also Dataworks Enterprise software development kit. 

This document is intended to supplement Dataworks Enterprise SDK and assist developers in getting started in using Dataworks Enterprise to develop their own client and server programs.  Later in this document discussion moves toward more advanced topics regarding Dataworks Enterprise development and recommendations for your Dataworks Enterprise programs.

It is assumed that the reader is familiar with the basic principles of COM programming.  Further, since, for all its shortcomings, Visual Basic is the most widely used system for COM development, and is relatively easily understood, discussion and samples in this document will use Visual Basic except where specifically referring to other development environments.  Where the developer prefers to develop using other languages such as C++ the translation from VB should be readily apparent. 

Further, it is assumed that the reader has Dataworks Enterprise installed on their workstation.[1]   It is recommended that at least the ?Documentation? and ?Examples? be selected when installing Dataworks Enterprise for development purposes.

This document does not discuss the general Dataworks Enterprise architecture from an operational perspective.  If you want a discussion of Dataworks Enterprise? data distribution, resiliency, fail-over and load balancing features, or details regarding the use and configuration of the various Dataworks Enterprise modules please refer to the ?Dataworks Enterprise Operations Guide?. 

Some preliminaries?

Dataworks Enterprise has been designed from the outset to be a high performance system.  As such it has been built with careful optimisation and tuning to run natively on the Windows NT platform, however, a portable Java version, referred to as ?the JPress?, is soon to be released which will allow Dataworks Enterprise functionality to be available on non-NT platforms such as Unix systems, and most other platforms that have a standard Java implementation.

However, the native NT implementation remains the preferred way of running Dataworks Enterprise. The Java implementation is primarily intended as a facility that allows the functionality of the native NT implementation to be accessible on other platforms.  Thus, it is expected that most development against Dataworks Enterprise will be done using NT.  Where development is done using the Java interfaces the principles remains the same with a few caveats that will be noted in the appendixes of this document.

This document primarily discusses developing against the native NT implementation. 

At the heart of Dataworks Enterprise is the cache.  The cache provides the core messaging framework for communication between Dataworks Enterprise components.  An extremely minimalist installation of Dataworks Enterprise need only consist of one file, RTCache.dll.  This DLL exposes a collection of COM objects that implement the cache?s messaging functionality and provide the external interfaces to it. 

Any program that connects to the cache will need to use these COM objects; there is no ?lower-level? way to access the cache functionality. 

The COM interfaces exposed by all of the core Dataworks Enterprise components are ?automation compatible?.  Thus it is possible to build fully functional systems that use Dataworks Enterprise using just about any development system or language that can use COM, including scripting clients such ASP web pages, C++, Visual Basic, etc. 

Obtaining data using a Dataworks Enterprise Client

In this section we will demonstrate how to obtain data from Dataworks Enterprise by building a simple program that will connect to a Dataworks Enterprise source and collect a few items of data from it. 

First, we?ll need a Dataworks Enterprise source.  Although there are many server modules that are distributed with Dataworks Enterprise, we?ll start by using a sample server called ?TestSvr.exe? that will give us some sample data to work with. 

TestSvr can be found in the ?Examples\VB\TestSvr? subdirectory of Dataworks Enterprise installation directory. 

E.g.: "C:\Program Files\Primark\Platform\Examples\VB\TestSvr\TestSvr.exe"

When you run TestSvr you will find a few dialog options that allow you to adjust the type of Source to be created.  To get TestSvr to create a normal source specify the name of the source you wish to create (e.g. ?TEST?) and ensure that the options ?Standard Source? and ?Record Server? are selected.  Then press the Start button. 

TestSvr will now create a simple Dataworks Enterprise source of the name specified and should look like this: 

We can verify that this source has been created successfully using the ?Client.exe? program.  This is installed in Dataworks Enterprise Installation directory under the Bin directory. 
E.g.: "C:\Program Files\Primark\Platform\Bin\Client.exe"

When this starts it displays a list of available sources and a test box into which you can type the symbol of the instrument you wish to request of the selected source. 

In the case of TestSvr, all instrument names are valid, but lets take the instrument ?ABC? for example.  Select the ?TEST? source (or whatever name you chose for your source in TestSvr) and then type ?ABC? for the Instrument then hit return.

The Client program should now look similar to this:

What this shows is the dynamically updating list of fields available for the ABC instrument of the TEST source. 

If you see this, all is well.  Otherwise check that TestSvr is running as described above.

Building our own client

Lets start by running up Visual Basic and creating a Standard Exe project.

The first thing we need to do is reference the RTCache type library. 

Follow the ?Project-References? menu, select the ?TOSC Real-Time Cache?, and click OK.  (When coding we can now reference all the types declared in this Type Library using the prefix ?RT. ?). 

TIP:   New versions of Dataworks Enterprise sometimes include extensions to the original interfaces.  If you write code that relies on these new interfaces your program may not run successfully where an older version of Dataworks Enterprise is installed.  If you want to ensure that your program does not rely on newer features, you can instead choose to reference one of the older versions of Dataworks Enterprise? type library that are distributed as part of Dataworks Enterprise SDK.

Since our program is to be a client of Dataworks Enterprise, we need to create an instance of RTClient object to connect to Dataworks Enterprise cache. 

There are two mains ways to use the RTClient: the synchronous method and the event-based asynchronous method.  The synchronous method is slightly simpler to code however it is usually reserved for scenarios where we don?t can?t handle events, such as in web server scripts.  The event-based asynchronous method is the default, and for most purposes the preferred method. 

Initially though, we will use the synchronous method.  The RTClient has a Boolean property called Blocking that when set to true will cause synchronous semantics to be applied to the retrieval of data.

Now we can double-click our project?s form to view the code for our form including the Form_Load event handler, and add code to create a Blocking RTClient. 

 

TIP:   Dataworks Enterprise installs a help file describing all the cache objects.  To get help any cache object or method just select the object in VB and press F1.  Alternatively, if you?re not running VB then you can simply open the help file by typing its name - ?RTCache.hlp? - in Explorer?s ?Start-Run? box.

To make a request of this client we need to get it to create a record that will represent the request for an individual instrument, then set the appropriate attributes of that record, and ?bind? it to the cache. 

 

The call to the Bind method is where the important stuff happens.  Having ?bound? the record, the cache requests the specified source for the specified instrument.  In this case, the cache knows that the ?TEST? source is being provided by the TestSvr program, and sends it a request for ?ABC?.  The resulting data will subsequently be available in the Fields collection of our record.

The term ?Bind? is used since a record does not actually make a request until it is bound to the messaging system in the cache by calling this method.  It has been observed that either the terms ?Request? or ?Subscribe? might have been better names for this method.  However, changing it now would be much more trouble than its worth, so we must live with it!

Now that we have requested the record, we can access the fields that have been returned for this record.  Accessing an individual field can be done by name using the record?s Field property.  Alternatively, all the available fields can be accessed from the record?s Fields collection.  The Fields collection can be indexed by numerical index or by field name, or it can be iterated through using a standard COM enumerator.

In the following example, we enumerate the fields and for each field we trace out each fields Name and Value.

The Visual Basic Immediate window should show the output similar to the following (the values themselves may differ):

SYMBOL : ABC
BID : 37178.1152662037
LAST : 37178.2151967593
VOLUME : 2
EXCHTIM : 14/10/2001 02:45:59
NET_CHG : 0.3
PRCTCK : U

 

Rather than iterating through the fields collection, if you?re interested in a specific field it can be slightly more efficient to use the ?Field? property.  The alternative of using the Fields collection means that the cache must construct the collection object itself.  So for example, we could get the value of the BID field like so:

    somevariable = rec.Field("BID").Value

Using RTClient Events

We now have a simple working Dataworks Enterprise client.  However, Dataworks Enterprise is a dynamic cache that monitors the data for changes.  (As such, we often refer to ?subscribing? to an instrument rather than simply ?requesting? it.)  To be informed of changes to an instruments data we need to respond to events from the RTClient. 

The RTClient can fire several different events but we will start with the ImageFields and ModifyFields events.

Consider the following reworked code:

Some important changes have been made here.  Most notably, we are now accessing the fields only when an event occurs that indicates that new field values are available.  (We?ll look at what the ImageFields and ModifyFields actually mean in a moment.) 

Since we?re now using the event based asynchronous semantics of the RTClient we set Blocking to False and tell VB that we?re interested in RTClient events by specifying ?WithEvents? when declaring our rtc variable.  

Finally, it is important that we keep a reference to our record around somewhere (we?ll look at why later), so we create a records collection and store the record in that. 

TIP:   When using VB?s debugger to step though a program or wait at breakpoints then VB handles events in ways that may be less than ideal under some circumstances.  The designers of the VB environment wanted the program to continue to appear functional at all times, so if you use the debugger to single-step through the code and an event such as ImageFields occurs, VB just throws the event away.  This is useful if the event is something relatively unimportant like a button click, but our case failing to receive an Image event can be most confusing.  Therefore, I recommend that you don?t single step through this program beyond the call to Bind, because you will probably miss the ImageFields event.  Rather, if you want to examine what is going on inside the ImageFields event handler function, place a breakpoint inside the ImageFields function and let the program run without single stepping beyond the call to Bind. 

In summary, this is what actually occurs with this version of our program:

1)     As the program starts the form loads and the Form_Load function is called.

2)     We create our client, create our record and make the request to the specified source using Bind .

3)     The function exits, the form is displayed, and the program begins processing windows events. 

4)     The Cache receives data for the request and causes an ImageFields event to be fired.  Our ImageFields event handler loops through the available fields printing each one out.  The program then resumes processing windows events. 

5)     The Cache receives further updates for our instrument and for each causes the ModifyFields event to be fired.  Our ModifyFields event handler loops through the fields that have changed printing each one out.  The program then resumes processing windows events, and so on. 

Note the parameters to the ImageFields and ModifyFields functions.  Each has two parameters: a pointer to the record that is being updated and the list of fields that are being updated. 

ImageFields is used when the source is supplying the complete set of fields that make up the record.  Providing there is not a problem with the request then ImageFields is usually the first event that is fired after a record is requested.  ImageFields can be fired subsequently for a record but the client program should consider the list of fields supplied at that point to be the complete and correct list, ditching any previous fields that it had for that record.   As such the list of fields supplied as the event function parameter will exactly match the collection of fields obtained by examining the records Fields property. 

ModifyFields on the other hand is used when the source is supplying changes to the previously supplied fields for a record.  It typically will be just a subset of the list supplied for ImageFields.  By contrast, the records Fields property provides a collection that reflects the latest value of all available fields.  Because of this when our program receives a ModifyFields events it iterates through the fields collection supplied as the event function parameter rather then the records Fields property, since we don?t want to print out fields that haven?t changed in this case. 

When you run this version you should see something like the following in VB?s Immediate output window. 

Although the values used here are rather meaningless from the perspective of financial market data they do help to illustrate what is going on.  (In this case I let the program run for about six seconds.  Note that ModifyFields was called twice.  This is because TestSvr has a timer which it uses to update four of its fields every three seconds.)

Note that the fields LAST and PRCTCK are not being updated by the source so they aren?t being supplied to the ModifyFields.  If we did need to obtain the current value for the LAST field we could obtain it at any time from the records Fields collection (or using the record?s Field property).

Also, note that the source is choosing to send out the SYMBOL field with each update even though its value remains the same.  Just because the value is the same as it was doesn?t prevent a source telling you that the client should consider it to have a new value!  This is useful in various contexts, in particular when delivering financial market data ? a trade may occur at the same price as the previous trade but clients still need to know that a trade has occurred.

Timing and Blocking

In our previous example, consider what fields are available immediately after we call Bind, but before our Form_Load function exits.  We have Blocking turned off so we?re using events to be notified of data availability, but at that moment no events have yet been fired.  Therefore, our record simply won?t have any fields at that point.   That?s why in this example we have moved our code that uses the record?s Fields property into the ImageFields event handler.

How then does it work in our earlier example when Blocking was tuned on?  In that case, when Bind is called the cache immediately sends off a request to the source that then gets processed on another thread.  If you then access the Fields collection but no data has yet been received in response to the request, the cache blocks the thread until such data is available.  Note that the thread only blocks when necessary, that is, as you attempt to access properties of the record that would return data that isn?t yet available.  If you don?t access the Fields collection (or any other property that returns the contents or state of the record) the thread won?t block.  If on the other hand, if you only access the record?s data after the data has been internally received by the cache, then the data can be returned immediately, without needing to block the thread at all. 

 

TIP:   When you are writing code that must be synchronous yet efficient, such as building web pages using ASP script, it is worth keeping in mind these blocking semantics.  (In such circumstances, setting Blocking to be true is essential since the script-processing engine simply does not supply events at all.)  In this case, try to make your requests for all your records as early on as possible in the script, then access the record?s data as late as possible.  In the intervening time the cache may be able to obtain all the required data for all the records, thus avoiding thread blocking and minimising context switches. 

          On the subject of ASP development, do not be tempted to store references to the RTClient, RTRecord or any other Dataworks Enterprise objects at the ASP Application or Session scope.  Since all Dataworks Enterprise objects use the COM Apartment Threading model, you?ll cripple your web server performance if you do this.  (See the Advanced Topics section below and Microsoft Knowledge-Base article Q243543 for more information.)

Note that when you set Blocking to be true, the thread blocking actually only occurs in the outer layer of the cache and only on the clients calling thread.  Internally the cache continues to run asynchronously, efficiently servicing all other data handling required.

Record lifetime and automatic request cancellation

In our last example, we created an RTRecords collection in which we stored a reference to the record we created.   This is very important because it keeps the record in existence.  COM objects are reference counted ? this enables an object to delete itself when there are no more external references to it.  When this occurs for an RTRecord object it first un-binds itself from the cache causing it?s request to be cancelled. 

This behaviour is very convenient for most situations ? ?forget? about a record and it will quietly clean itself up and stop pestering you with events.  On the other hand, it is important to ?remember? a record while you?re still interested in its data. 

So in our example, if we had not stored a reference to the object in the records collection (which is declared at global scope) then when the Form_Load function exited then there would be no references left to the record we had just requested.  In other words the record would be bound and then immediately un-bound, and we would never get any data for it. 

(Note that although we used Dataworks Enterprise? RTRecords collection to store the record any method of holding a reference to the record would have been sufficient. ) 

If you want to explicitly unbind a record at anytime but keep the record in existence then you can simply set its Bound property to false.

Status and Fault handling

So far, we have successfully requested a record from a source.  What if something goes wrong?  For instance, what if the instrument name requested wasn?t available from that source?  Or, what if the source itself wasn?t available, or was subsequently shutdown, or had some other problem?

Situations such as these are handled by Dataworks Enterprise? status information which is maintains about each requested record.  This status information is always available from the record?s Status property, and clients are notified of any changes to a record?s status via another event named ChangeStatus. 

Here?s a trivial example of a ChangeStatus event handler for our program:

Here the supplied parameters specify the Record whose status is changing, the StatusType which specifies the new state of that record, a StatusCode which is really a just a code specifying the reason for the state change, and finally a human readable text string further explaining why the fault has occurred. 

ChangeStatus is only called when there?s a fault.  If a record gets some data in an image then it should be inferred that any previous faults on the record have been resolved (else we wouldn?t be getting any data anyway). 

Although there are 5 values in the RTStatusType enumeration the StatusType parameter in the ChangeStatus event can only ever be one of two values: either RTStale or RTFailure. 

?         If the Source issues an RTStale status, it means ?there is a problem with the data, I may be able to automatically recover the situation and provide correct data later on?. 

?         If the Source issues an RTFailure status, it means ?there was a problem with your request, I will not be able to supply any further data for this record?. 

RTFailure is a permanent fault, whereas RTStale is often referred to as a temporary fault (although this is sometimes a little optimistic since the fault can last indefinitely!).

A couple of example situations causing each kind of fault:

An RTStale might be returned if you request something from a Source that does not exist, since someone might start the source later in which case the request would then be applied against the source and data hopefully returned. 

An RTFailure might be returned if you request something from a Source that has a fixed list of possible instruments but you have requested an instrument that is not part of that list.  It?s never going to be able to supply the data so it puts the record into a permanent RTFailure state.

Of the other possible status types besides RTStale and RTFailure, the RTStatusType additionally has the following values (although these never occur in the ChangeStatus Event):

?         RTPending is the initial state a record has before it receives any data or other status notification.

?         RTConnected is the state a record has when it has received data but no subsequent fault.  It is the state a record will have following an ImageFields event, and can be interpreted as the ?normal? state of a record. 

?         RTReRequest is never seen by a client of the cache, it is only used by the cache servers.  (These are discussed later.) 

When using Blocking semantics, before examining the records field collection we should really check that the record has a state of RTConnected.  If it is not, we should respond to the fault rather than try accessing the fields, which will be invalid anyway.

Other RTClient Features

We have considered ImageFields, ModifyFields, and ChangeStatus.  What of the other events that the RTClient can fire? 

(Keep in mind that ImageFields provides the initial comprehensive list of fields, and ModifyFields provides updates to the values of fields in that list.)

You may have noticed the following other events are declared:

AddFields:             This supplies new fields that did not appear in the field list supplied with the Image.

DeleteFields:         This supplies a list of fields that may previously have been supplied via ImageFields or AddFields, but that no longer exist for this record and should no longer be considered to be part of the records fields collection.

MaintainFields:       This is a rarely used event whereby a Source can notify a client that the values of previously issued fields need to be corrected with new values.  It is commonly treated exactly like a ModifyFields event by client programs.  (The distinction between MaintainFields and ModifyFields is only really important when dealing with financial data that represents traded data.  In this case a Modify represents a change to previous data caused by a trade, whereas a MaintainFields is issued to correct previous data but the update should not be taken to represent a trade.)

SourceChange:      an RTClient object maintains a list of available sources.  This event notifies the client of changes to this list. 

Command:             sources can support ?Commands? that are sent by a client.  This events supplies the response that a Source can return to the client.  (Commands are discussed later.)

BeginTransaction    and?

EndTransaction:     Dataworks Enterprise does not currently use these two events.  At some future time these may be supported, but for now they can be ignored. 

As mentioned above, the RTClient object maintains a list of sources available on the local machine.  The current list of sources can be obtained from the RTClient object via its Sources property.  However, sources can be dynamically mounted or dismounted.  The SourceChange event will tell you if a particular source subsequently goes up or down. 

Try adding the following event handler to our client program, then watch its output while you press the Stop/Start button on the TestSvr.

 

TIP:   Be careful not to rely on the SourceChange event until you have accessed the RTClient?s Sources property.  The SourceChange events are only fired after the Sources list has been established, which only happens after you access the Sources collection.  Therefore, if you never access the Sources collection, you will never get a SourceChange event.  This is a side effect of the permissioning system in Dataworks Enterprise, which will be discussed later. 

A last look at our program? 

We now have a simple program that uses an RTClient object to connect to the cache.  It doesn?t do anything interesting with the data it receives, it just dumps it to an output window; manipulating the data or displaying it in more imaginative ways is left to your own imagination? 

Bear in mind that we have only requested one record from the RTClient for the sake of space and simplicity but we could easily request other records. 

In our Form_Load function we have a few lines of code that create and store a request for ?ABC?.  We could duplicate these to request a second record, lets say ?XYZ?, like so. 

Our records collection would now contain two RTRecord objects representing ABC and XYZ respectively.  When events such as ImageFields are received, we know which record is being updated because a reference to the appropriate record is passed as an event parameter.

Tags

Another useful feature of almost all the commonly used Dataworks Enterprise objects is that they store a ?Tag?.  This user-defined value is associated and maintained for each object into which you can place anything that will fit into a Variant.  What you use this for is up to you but it is often useful to associate an index of other object with, for example, each record object for use later in your program. 

 

 

Samples

You can find a slightly more complete example of a client program written in Visual Basic in the Samples\VB\Client directory of Dataworks Enterprise SDK.  This example has similar functionality to the program we have just built except that it also displays the record?s status and data on the form in a more user-friendly manner. 

Providing data using a Dataworks Enterprise Server

Thus far, we have shown how to retrieve data from a Dataworks Enterprise source using its RTClient interface.  For simplicity, we have only looked at one source mounted by one Dataworks Enterprise server, the TestSvr.  However, Dataworks Enterprise is supplier with a wide variety of data feed handlers what expose their data as Dataworks Enterprise sources. 

Although Dataworks Enterprise excels as a data retrieval mechanism both in terms of ease-of-use and performance, this in itself is not enough to distinguish it, as there are other tools in the market place that allow you retrieve data from various systems. 

Where Dataworks Enterprise really show its true colours is the ease with which new data sources can be created and mounted onto Dataworks Enterprise system.  This section will demonstrate the creation of a new data source. 

Dataworks Enterprise ?Servers? are programs that use the RTServer object to create ?Sources?, where each source is a logical collection of data.   Each server can mount multiple sources.  Dataworks Enterprise servers are commonly referred to as ?Handlers? since they usually ?handle? an external data feed, serving up the data to Dataworks Enterprise system.

Creating a Dataworks Enterprise server source is about as easy as creating a Dataworks Enterprise client.  Experience shows that in practice the task of writing a real-world Dataworks Enterprise server to interface to a typical external data feeds almost entirely consists of the effort of handling the details of that feed.  The amount of code to necessary to write to the RTServer interface is usually minimal by comparison. 

The process of writing a Dataworks Enterprise server is effectively the inverse of that involved in writing a Dataworks Enterprise client. 

In brief, these are the steps required to mount a source: 

1)     Create an RTServer object. 

2)     Ask it to create a Source of a given name and type. 

3)     Wait for Dataworks Enterprise to send an event describing a request made of that source for a given item.  (A server-side item corresponds to a client-side record.)

4)     Obtain the appropriate data by some means and supply it to Dataworks Enterprise by calling methods on the Item. 

5)     Continue to update Dataworks Enterprise with any changes to the Items data until Dataworks Enterprise sends your server an event notifying it that no more clients are interested in that item?s data. 

Like the RTClient, the RTServer has two basic modes of operation.  You can wait for RTServer events as outlined above or you can pre-emptively ?push? data into the cache regardless of whether it is currently required by any clients. 

Most servers will supply data in response to events so we shall this technique first, and leave pushing items into the cache until later. 

Building our own server

Since in this example we want to focus on Dataworks Enterprise interfaces, and not become bogged down in the details of obtaining data from some external repository, we?ll just conjure up some data in our server program. 

In this example, we?ll create a source called ?SCOOBY? which will supply data for one or two named items (or records from the RTClient?s perspective) each corresponding to a hypothetical individual.  The data we supply for each individual will consist that persons age, sex etc. 

 

As with Dataworks Enterprise client, lets start by running up Visual Basic and creating a new Standard Exe project.  Once again, we will want to reference the RTCache type library. 

Now we can add some code to our new form?s Form_Load event.  We shall first create ourselves an instance of an RTServer object.  Then we will ask it to create a source called ?SCOOBY? for us:

We have stored a reference to these objects at global scope because as with RTRecord objects created by the RTClient, RTSource objects are functional only while they exist and we are responsible for keeping them in existence by holding a reference to them. 

Imaging an Item

Now we can add a ?Request? event handler that will be called when a client requests an item from us. 

The parameters for the Request event specify the Source, the Item that represents the client?s request, and a RequestType, which we shall discuss later.

We only receive events for Sources that have been mounted by our server so in this case the Source parameter will be that same as the srcScooby object that we created earlier.

Next, this code examines the Item?s Instrument property to discover exactly what the client has asked for.  If they are asking for something that we recognise then we create an RTFields collection object, and populate it with some data using the Add method. 

Then we need to supply our fields collection to the cache, telling it to update the corresponding Item.  Since this is the first time we are supplying data for this item we use the ImageFields method of the Item.

Note that an RTItem object has methods that correspond directly to the events that an RTClient object fires as discussed in the previous section. 

In this example, we have supplied the data to satisfy a request in the Request event function itself, but it doesn?t have to be done in this way.  Depending on the implementation of a server the data may not be immediately available and it may not be desirable to wait until it is available before returning from the Request function.  If our server must perform a substantial amount of work to be able to produce the data required for the Image, then it is perfectly acceptable for our server to return from the Request function and then supply the data for the Image some time later.  This can significantly improve the responsiveness and efficiency of various types of server.

Bear in mind that it is important for the server to return a response to client clients request in a reasonable time frame (even if that response is a failure notification using ChangeStatus as described below).  This is because some clients may be blocking waiting for a response for their request.  If the server fails to provide a response in a timely manner, or even at all, then the client may be blocked for a long time, which is rarely desirable. 

Handling bad requests

What should we do if the client requests something we don?t recognise or for which we cannot supply data?  Then we should set the status of the item to indicate this fact. 

This code now also supports a second person, but if the client is requesting neither ?SHAGGY? nor ?DAPHNE? then we call ChangeStatus to alert the client to the problem by calling ChangeStatus. 

When we call ChangeStatus we specify RTFailure to indicate that this is a permanent failure from we cannot not recover.  We also give some indication as to what the problem is by specifying a status code of RTStatusNoSuchItem along with a short textual description.  (See the RTCache help file for other possible values for the RTStatusCode parameter and recommended usage.)

At this stage, we have a minimalist but usable program that we can run and test. 

Try running this server program, and then run the standard Dataworks Enterprise Client.exe program as described earlier in this document. 

You should see that the Sources list now contains our ?SCOOBY? source.  Next to this, you can type the name of one of our people, try ?SHAGGY?, and see the data returned. 

TIP:   Remember that Dataworks Enterprise is case-sensitive with respect to Source names, Record/Item names, and Field names.

If, instead of ?SHAGGY, you request something like ?PETERPAN? the client program will display our status message indicating that this is an unknown item in the window?s title bar. 

Updating an Item?s data

Next, we?ll update the Item?s data using the ModifyFields method.

In this example, we?ll add a timer to our form and use it to update an item every few seconds.  We?ll do this for just one of our items, ?SHAGGY?.  To be able to update the item we need to be able to reference this specific item later on, so we will store a reference to it.  In this instance we will do it the trivial way ? by storing the reference in a global variable. 

To make sure we don?t hold onto this item longer than is appropriate we will also handle another RTServer event ? Cancel. 

Here?s the full code for our new server:

Dim WithEvents rts As RTServer
Dim srcScooby As RTSource
Dim itmShaggy As RTItem

Private Sub Form_Load()
    Set rts = New RTServer
    Set srcScooby = rts.CreateSource("SCOOBY", "Record", RTStandardSource)
    Timer1.Interval = 1000    
End Sub

Private Sub rts_Cancel(ByVal Source As RT.IRTSource, _
                       ByVal Item As RT.IRTItem)
    If Item Is itmShaggy Then
        Set itmShaggy = Nothing
    End If
End Sub


Private Sub rts_Request(ByVal Source As RT.IRTSource, _
                        ByVal Item As RT.IRTItem, _
                        ByVal ReqType As RT.RTRequestType)
    Dim fields As New RTFields

    If Item.Instrument = "SHAGGY" Then

        Set itmShaggy = Item ' remember this item
        fields.Add "AGE", 20, "Twenty"
        fields.Add "SEX", "M"
        Item.ImageFields fields
    ElseIf Item.Instrument = "DAPHNE" Then
        fields.Add "AGE", 21
        fields.Add "SEX", "F"
        Item.ImageFields fields
    Else
        Item.ChangeStatus RTFailure, RTStatusNoSuchItem, "Person unknown"
    End If
End Sub

Private Sub Timer1_Timer()
    If Not itmShaggy Is Nothing Then
        Dim currentAge As Integer
        currentAge = itmShaggy.Field("AGE").Value
        
        Dim updateFields As New RTFields
        updateFields.Add "AGE", currentAge + 1

        itmShaggy.ModifyFields updateFields
    End If
End Sub

In our new program, if ?SHAGGY? has been requested, and not subsequently cancelled, then the variable ?itmShaggy? will reference the item representing this request.  In the timer event function, which fires every second, we create a new field collection and add a single field it which contains the ?AGE? of shaggy which increments each time the function executes.  We call ModifyFields with this field collection which sends the update to the client. 

In our program we have stored a reference to the item we want to update by storing an explicit reference to it at global scope, but there are a numerous alternative techniques.  We could have simply looked up the Item by name from the source?s ?Items? collections for instance. 

Item lifetime

As has been previously noted the heart of Dataworks Enterprise is it?s cache.  The cache is responsible for maintaining a database of the items on watch and passing the appropriate messages between the clients and servers of this data. 

If a client requests a given item while another client already has the same item on watch Dataworks Enterprise delivers up the current data to the second client from its cache, without making any requests to the server.  The Request for an item occurs when the first client subscribes to it and the ?Cancel? message for the item is only sent once all clients have dropped their subscriptions to it.  In other words there is a many-to-one relationship between records and items.  The cache ensures that all the interested clients get any updates for it. 

It can be seen from this that the lifetime of a request is really driven by the clients.  The clients keep the Record around for as long as they want and the cache keeps the Item that corresponds to these records around for as long as is required by the clients.  Thus the lifetime of a client-side RTRecord object is the responsibility of the client program; whereas the lifetime of a server-side RTItem is the responsibility of the Cache, rather than the server program. 

In our server program, we do not need to hold a reference to the Item to keep it alive; the cache will do that for us anyway.  The only reason we hold a reference to it is because it makes it easier for us to update the fields of the item.

Does this mean that the server has no control over the lifetime of an item?
No.  It can both create an item and deliver its data to the cache independent of any client request ? this is called ?Pushing an Item?.  Also, it can ?Terminate? an item which forces its early cancellation, even if clients still have it on watch. 

Pushed Items

Most servers will supply data in response to events.  This ensures that the server only goes to the effort of generating data is a client is interested in it.  Further there are many scenarios where the server has no reasonable mechanism for producing the data until a client has specified what it requires.  However there are scenarios where these issues do not apply.  In this case a server may choose to pre-emptively push one or items into the cache regardless of whether a client has requested the item. 

Pushing data into the cache is usually done where the number of items involved is small since we don?t want to unnecessarily overburden the cache. 

To push an item into the cache we simply force the creation of a new item for a given Source by adding the item by name into the source?s ?Items? collection. 

For example to create an item called ?FRED? to our source and then provide some data for it we can simply do the following:

Terminating Items

Items are normally cancelled once all clients have dropped their subscriptions to the Instrument, or in the case of pushed items, by removing the items from the Source?s Items collection.  However, it is often useful to inform clients that an item that they have on watch will no longer receive any updates.  For failure situations this can be accomplished by calling ChangeStatus with the RTFailure flag, but what about situations where a failure has not occurred but we still want inform clients that an item will simply no longer update? 

In this case we can Terminate the item by setting its ?Indicator? to be ?RTTerminated?.  This will cause the cache to cancel all client subscriptions to the item.  E.g.

When an item is terminated the cache cancels all client requests for the item, as if the interested clients had unbound their records themselves.  However the cache knows that knows that the client does in fact still hold onto the record, and will re-request the record on the clients behalf if the Source of that item goes down and another comes back up again with the same name.  Thus, it can be said that a terminated item will receive no more updates ?for the lifetime of its source?. 

A last look at our program? 

We now have a functional Dataworks Enterprise server that provides updating data to its clients. 

Although this document does not cover the operational aspects of Dataworks Enterprise, we now have a server that with only a little configuration, but without any further development, can expose its data while leveraging the various distribution and resiliency features of Dataworks Enterprise Data Platform.