Network Configuration, Load Balancing and Resiliency

Dataworks Enterprise supports connection between hosts on a network through the use of the Remote Client/Server modules, which are described in detail above. Using facilities available with the launcher, cache and Remote Client/Server modules, it is possible to build systems that provide load balancing, resiliency, hot standby and fail over.

The role of the launcher is to provide automatic restart of a failed system process, the first line of defence against system failure. The launcher is described in detail above.

The cache has a role in providing hot standby sources, which are automatically switched on feed/server failure.

Remote Server makes its sources available to hosts running Remote Client using a single TCP/IP virtual circuit. Applications running in the client host can access sources from its local cache. Requests are propagated to the server host where the server cache satisfies them. The operation is invisible to the client application.

Dataworks Enterprise Finder Protocol

All Dataworks Enterprise server modules (Remote/Publishing Servers) operate a Finder protocol. When a server is attached to the network, it regularly broadcasts a message, containing:

  • the name of the service it is providing (Remote or Publishing);
  • the protocol version it supports;
  • the IP address of the host;
  • the cumulative load of the host.

    Client software receives and stores this information. When a client connects to a server, it elects to connect to the host with the lowest load. In effect, configuration of servers and clients is automatic.

    If a client is disconnected from a server, as would be the case if the server were disconnected from the network for any reason, the client automatically, reconnects to the lowest load server available in the network. This mechanism provides load balancing of client connections.

    The Finder protocol can be selectively disabled and static routes employed where there are issues associated with broadcast traffic. This is achieved through configuration of both the Remote Client and Server modules.

    The Finder protocol utilises UDP using the same socket port as the port used by the actual client-server connection. For instance, the Remote Server default port for connection is 6510.

    Multiple Remote Connections

    The combination of the Finder protocol and the Remote connection can be duplicated across multiple ports in a running system to provide additional resiliency. In this environment, two Remote Clients are run (on different ports) on each client host. Each server runs a single Remote Server configured on the different ports.

    For instance, in the diagram below, Remote Server A exposes a finder and connection on port 6511 and Remote Server B exposes on port 6512. In each client hosts, there are two Remote Clients in operation, one on 6511 and the other on 6512 and, thus, each client host is connected to both Remote Servers.

    In this configuration, the cache hot-standby facility comes into operation. Suppose, there is a feed called “Equities” with a handler running on each Remote Server and mounting a source called “Equities”. As a result, both Remote clients will attempt to mount the “Equities” source on the client host. The first client to mount will be used to satisfy requests, the second being placed into standby by the client cache. If the “Equities” source on Remote Server is disconnected (either as a result of a source going down or a server disconnection), the client will automatically failover to the standby source from the second server.

    Note that server A and server B may be replicated within the network for additional load balancing and resiliency.

    It is also possible to run a Remote client and Server on a single host connected to separate ports. This forms the basis of one of the more “standard” network configurations:

    A Standard Network Configuration

    In this example, feed servers mount sources directly from feeds and make them available to platform servers through Remote Server. Each feed server is on a separate port as with Remote Servers A and B in the previous example. Each platform server has a pair of Remote Clients connected to a feed server making the content of both feed servers available to the platform servers in a hot standby configuration. In addition, each feed server runs a single Remote Server on a third port. Clients connect to one on these Remote Servers. If a platform server fails, the client connects to the other platform server (fail over). If a feed or feed server fails, the platform server will automatically fail over to the hot standby server.

    Typical Network Configurations

    The typical configuration will depend on the requirements of the customer and whether the system is being deployed for test and evaluation purposes or as part of a full deployment.

    In a typical evaluation system (and some fully deployed systems) data is ingested into the platform at the Thomson Financial central site and distributed via the Thomson Financial WAN to the customer site. Here it is ingested into Platform Servers and made available to client of the customer network. In addition, local Feed Servers at the client site take local (non-Thomson Financial) content and also make it available via the platform.

    A Typical Evaluation Configuration

    When the evaluation system is to be upgraded to a full deployment, real-time content on the platform servers is propagated to the customer site via satellite and ingested into the platform locally (providing better price-performance). NOTE: some content is still deployed using Dataworks Enterprise over the Thomson Financial WAN.

    A Typical Fully Deployed Network Configuration