Petroware Logo

Fluxens

Petroware delivers server technology to the Fluxens live streaming project, the next generation real-time services for the energy business and the successor of technologies like WITSML and PRODML.

Petroware is responsible for implementation and operation of the Fluxens scalable real-time server running on the Amazon AWS cloud.

Fluxens in action

The video below shows live streaming data from the Fluxens server and how the information can be applied in a domain specific application.

Fluxens Architecture

Fluxens is based on a centralized server running on the Amazon AWS. It is using bidirectional web socket technology to acheive maximum throughput and minimum latency. All communication is done over standard Internet protocols using established encryption standards.

Data is delivered through the open JSON Well Log Format in both the producer and the consumer end and are communicated over the JSON-RPC protocol.

Fluxens architechture

Note in particluar: Fluxens is not a framework. It is not open source (as there is no source), there are no code libraries and no downloads.

Fluxens is a technology built on open concepts, readily available for anyone who wants to connect, see below.

Data model

Fluxens can serve any time series data that fits the JSON Well Log Format model.

The JSON Well Log Format organize data in curves and rows in a spreadsheet like structure. Curve values can be of different type, including floating point, integral, logical, date/time and strings. No-values are supported according to JSON null.

The format superseeds legacy data formats used for well log data like LAS, DLIS and WITSML, but it is much more versatile and can easily be applied to other domains and industries.

Business model

Stream owners are customers of the Fluxens service. Becoming a Fluxens customer is free of charge, contact info@fluxens.com to register. By self service through the Fluxens web page customers defines streams by name/id. and clients (producers or consumers) of streams. Customers and clients are registered with username/password and standard contact information.

When this is in place producers can upload and consumers download stream data using the simple mechanisms described below. All bytes uploaded and downloaded are counted and is the basis for the customer payable amount. Clients are entities of their owners and are not payable.

FXP - The Fluxens transfer protocol

All communication with the Fluxens server is done by sending and receiving text messages in the very simple JSON-RPC (Remote Procedure Call) version 2.0 format.

Fluxens messages

To execute a method on the server a client sends a request message to the server, each of which are followed by one or more response messages from the server back to the client.

JSON-RPC request messages have the following generic structure:

{
  "jsonrpc": "2.0",
  "method": <method>,
  "params": [<parameter1>, <parameter2>, ...],
  "id": <id>
}

This is simply a remote way of making the following server call:

Object result = method(parameter1, parameter2, ...);

Response messages are defined as follows:

{
  "jsonrpc": "2.0",
  "result": <result object>,
  "error": {
    "code": <error code>,
    "message": <error message>,
    "data": <additional error information>
  },
  "id": <id>
}

In any particular response only one of result or error is present.

Web socket communication is asynchronuous and the id properties are used to associate responses with their requests; The ID of a response will match the ID of the corresponding request.

Producer methods

Below is a detailed description of the methods that can be called by a stream producer.

writeData

Method writeData
Description Writing stream data to the specified stream in the form of JSON Well Log Format.
Parameters username (String)
password (String)
streamId (String)
streamData (JSON Well Log Format)
Result-

The streamId is provided by the stream owner.

Example:

{
  "jsonrpc": "2.0",
  "method": "writeData",
  "params": ["nov_user", "da!a1$3", "000157", <data>],
  "id": 42
}

where the data parameter can be any valid JSON Well Log Format object, for instance:

[
  {
    "header": {
      "name": "EcoScope Data",
      "well": "35/12-6S"
    },
    "curves": [
      {"name": "MD"},
      {"name": "BS"},
      {"name": "GR"}
    ],
    "data": [
      [2907.79, 8.5, 29.955],
      [2907.80, 8.5,   null]
    ]
  }
]

The associated response message will either be an empty result on success or an error in case anything went wrong.

The first data that is written to a stream defines the index curve name and its value type.

For subsequent write operations these restrictions applies:

  • The index curve name must match the existing
  • The value type of a curve must match if the curve exists already
  • The dimension of a curve must match if the curve exists already
Note in particular the following non-restrictions:
  • Header data can be changed/added at any time
  • If header data is unchanged it can be left out
  • Curve metadata (description, unit, quantity etc.) can be changed at any time
  • If curve metadata is unchanged it can be left out
  • New curves can be introduced at any time
  • Not all curves needs to be present in each write operation
  • Curves can be arbitrarily ordered
  • Any number of rows can be written in each write operation
  • Rows may be written out of index order
  • Curve values can be replaced by resending an existing index

Stream data will be organized in the server so that consumers receive historic data properly sorted and aligned. When consumers are in real-time mode they will however get exctly what is written and must be prepared to organize this on the client side if required.

resetStream

Method resetStream
Description Reset the specified stream by removing all its data from the server.
Parameters username (String)
password (String)
streamId (String)
Result-

Example:

{
  "jsonrpc": "2.0",
  "method": "resetStream",
  "params": ["nov_user", "da!a1$3", "000157"],
  "id": 79
}

The associated response message from the server will either be an empty result on success or an error in case anything went wrong.

Consumer methods

Below is a detailed description of the methods that can be called by a stream consumer.

getStreams

Method getStreams
Description Get all streams available for the specified consumer.
Parameters username (String)
password (String)
Result streams (JSON Array)

Example:

{
  "jsonrpc": "2.0",
  "method": "getStreams",
  "params": ["akerbp_user", "liverp00|"],
  "id": 143
}

If all is fine, this will cause a response like:

{
  "jsonrpc": "2.0",
  "result": <data>,
  "id": 143
}

where the data entry is an array of stream id/name pairs like:

[
  {"id": "000679", "name": "EcoScope time"},
  {"id": "000721", "name": "EcoScope depth"},
  {"id": "001013", "name": "13/4-4 MD"},
  :
]

startStreaming

Method startStreaming
Description Start a stream operation on the specified stream.
Parameters username (String)
password (String)
streamId (String)
Result streamData (JSON Well Log Format)

Example:

{
  "jsonrpc": "2.0",
  "method": "startStreaming",
  "params": ["akerbp_user", "liverp00|", "001013"],
  "id": 190
}

The request will initially send all existing data for the stream. Data are chunked into manageble units so this might be divided into several responses. When synchronized, the consumer enters the real-time state and will get responses whenever data is written by a producer.

A typical response might be:

{
  "jsonrpc": "2.0",
  "result": <data>,
  "id": 190
}

where data is a valid JSON Well Log Format object, for instance:

[
  {
    "header": {
      "name": "EcoScope Data",
      "well": "35/12-6S"
    },
    "curves": [
      {"name": "MD"},
      {"name": "BS"},
      {"name": "GR"}
    ],
    "data": [
      [2907.79, 8.5, 29.955],
      [2907.80, 8.5,   null]
    ]
  }
]

stopStreaming

Method stopStreaming
Description Stop streaming on the specified stream.
Parameters username (String)
password (String)
streamId (String)
messageId (Integer)
Result -

The messageId is the corresponding ID from the associated startStreaming request made earlier.

Example:

{
  "jsonrpc": "2.0",
  "method": "stopStreaming",
  "params": ["akerbp_user", "liverp00|", "001013", 190],
  "id": 244
}

The request will stop all further streaming from the specified stream.

The request will cause a response message with an empty result on success or with an error in case anything went wrong.

Test data

Fluxens is pre-equipped with test data that makes it possible for software providers to test their systems in a real-time environment at any time.

Logging in as guest (username: "guest", password: "guestpw") will give access to drilling data from the disclosed Volve field (Equinor). The data is streaming at a random rate of a few samples per second. The demo set contains depth based and time based data, and single valued as well as multi-dimensional (image) curves.

Latency

A fundamental question when it comes to real-time services is what latency there is in the system at any point in time.

With WITSML/SOAP this has been difficult to measure due to its poll-based nature, but for time logs it is possible to get a rough estimate by comparing the time of the measurements with the time they arrives at the client. Typical latency in such systems are 8-12s. Seconds that is.

For WITSML/ETP this is expected to be lower, but as live ETP streams are far between we have yet to measure this accurately and there is anyway still a considerable time lost in XML/ETP processing.

Fluxens changes all this. By using technology from the online gaming industry the latency is reduced by several orders of magnitude. But more importantly: The latency is made an inherit property of the system.

The illustration below shows an overview of a typical setup. The total latency in the system consists of processing time in the producer, transfer time between producer and server, processing time in the server and finally transferring time from the server to the client.

Fluxens latency

With Fluxens, the time entries in the figure are all included in the delivery so that the latency can be tabulated or otherwise visualized in the client during the process:

Fluxens latency

The actual latency in a given case depends on the geographical location of producer, server and client and also somewhat on data frequency and volume. Typical values are 20-200ms. Milliseconds that is!

More information

Get more information on Fluxens from fluxens.com.