HP Feeds

Git Repository



The "hpfeeds" project implements a lightweight authenticated publish/subscribe protocol for exchanging live datafeeds. We tried to design a simple wire-format so that everyone is able to subscribe to the feeds with his favorite language in almost no time.

Different feeds are separated by channels and support arbitrary binary payloads. This means that the channel users have to decide about the structure of data. This could for example be done by choosing a serialization format.

Access to channels is given to so-called Authkeys which essentially are pairs of an identifier and a secret. The secret is sent to the server by hashing it together with a per-connection nonce. This way no eavesdroppers can obtain valid credentials. Optionally the protocol can be run on top of SSL/TLS, of course.

To support multiple data sources and sinks per user we manage the Authkeys in this webinterface after a quick login with a user account. User accounts are only needed for the webinterface - to use the data feed channels, only Authkeys are necessary. Different Authkeys can be granted distinct access rights for channels.

Wire Protocol:

Each message carries a message header. The message types can make use of "parameters" that are being sent as (length,data) pairs.

struct MsgHeader {
    uint32_t messageLength; // total message size, including this
    uint8_t opCode;        // request type - see table below

For example the publish message would consist of message header, length(client_id), client id, length(channelname), channelname, payload. The payload, can be arbitrary binary data.
On the wire this would look like:

length | opcode | next | identifier | next | channelname | payload
    85        3   9      [email protected]    9      mwcapture     137941a3d8589f6728924c08561070bceb5d72b8,

We got some Wireshark screenshots which show the structure of messages on the wire as well.

Message types:
  • error (0): errormessage
  • info (1): server name, nonce
  • auth (2): client id, sha1(nonce+authkey)
  • publish (3): client id, channelname, payload
  • subscribe (4): client id, channelname

For further details and definition of each message type, consider the page about the example CLI which describes how to speak the wire protocol.

  1. Server sends a nonce per connection
  2. Client sends id and sha1(nonce+authkey) hash
  3. Server looks up authkey by id and checks hash
  4. Server looks up subscribe/publish ACL for this client

State Diagrams:

A description of the different states a hpfeeds session can be in is available here.

Example session

For demonstration purposes, some example messages in plain text without length fields:

Sensor subscribe to a channel:

S: info,@hp1,1622303995
C: auth,[email protected],73a2d688e3cf1e43da604568163a752617ede2cb
C: subscribe,[email protected],mwrequest

Sensor announcing an event to a channel:

S: info,@hp1,1622303995
C: auth,[email protected],73a2d688e3cf1e43da604568163a752617ede2cb
C: publish,[email protected],mwcapture,137941a3d8589f6728924c08561070bceb5d72b8,

Sensor authentication failed:

S: info,@hp1,1622303995
C: auth,[email protected],b0c4b900960a4004f282302904491221bf4fffb9
S: error,authfail

Client subscribing to a channel with events announced by the server:

S: info,@hp1,2248827466
C: auth,[email protected],add48c9123fff468c87f251599400d9ea1d2da68
C: subscribe,[email protected],mwrequest
S: publish,[email protected],mwrequest,137941a3d8589f6728924c08561070bceb5d72b8


  • mwcapture channel: sensors announcing events (sensorid, sample hash, sample url)
    {"url": "", "daddr": "", "saddr": "", "dport": "445", "sport": "3725", "sha512": "a77687f49dff9baf425048e97931de39fc799efdad759d514bcf359e29106cbd6a6bf5aac5d9130d03f9d5dba352110f623c166c31f6ef7a21557acf6a4aa52b", "md5": "78c9042bbcefd65beaa0d40386da9f89"}
  • mwbinary-sensorunique channel: sensors pushing samples that are locally unknown to them (sensorid, sample)
    -> payload is just the binary itself
  • dcerpcrequests channel:
    {"uuid": "4b324fc8-1670-01d3-1278-5a47bf6ee188", "daddr": "", "opnum": 31, "saddr": "", "dport": "445", "sport": "3606"}
  • shellcodeprofiles channel:
    {"profile": [{u'return': u'0x7df20000', u'args': [u'urlmon'], u'call': u'LoadLibraryA'}, {u'return': u'0', u'args': ['', u'', u'x.', u'0', u'0'], u'call': u'URLDownloadToFile'}, {u'return': u'0x00000000', u'args': [u'x.'], u'call': u'LoadLibraryA'}, {u'return': u'0', u'args': [u'0'], u'call': u'ExitThread'}]}