User Tools

Site Tools


unifieduploader:start

Unified Uploader and Data Interchange Format

Goal

Create a unified uploader format to be used by any market data driven site. This data format can also be used for syndication amongst sites.

Discussions are ongoing on the EVE-O Forums: https://forums.eveonline.com/default.aspx?g=posts&find=unread&t=18466

Implementations

See the Unified Upload Format Implementations for sites and clients which support this format.

HTTP API Entry Points

Uploads

[SITEROOT]/api/upload

Syndication

[SITEROOT]/api/syndicate

Allowed HTTP Methods

Multiple methods MAY be supported by the specific sites in question. For details on what site supports what formats, please see the implementations page.

Use of Compression

Data can presented using Content-Encoding: gzip or Content-Encoding: deflate for endpoints which support uploads of this encoding.

Methods

Based upon discussions, here are the suggested methods and encodings, in order of preference:

POST (entity)

An HTTP POST request with the entity (the content) as the JSON message.

POST (Form encoding)

The body should contain the JSON message as the value with “data” as the key. I.E.

data={“resultType”:“orders”,“version”:“0.1”,“uploadKeys”:[{“name”:“emk”,“key”:“abc”},{“name”:“ec”,“key”:“def”}], …}

Make sure the data values are correctly urlencoded.

Message Formats

Examples

The following examples are shown in an expanded form to make them easier to read the actual data would be a single line and not have all the extra whitespace.

Orders

Market orders, example:

{
  "resultType" : "orders",
  "version" : "0.1",
  "uploadKeys" : [
    { "name" : "emk", "key" : "abc" },
    { "name" : "ec" , "key" : "def" }
  ],
  "generator" : { "name" : "Yapeal", "version" : "11.335.1737" },
  "currentTime" : "2011-10-22T15:46:00+00:00",
  "columns" : ["price","volRemaining","range","orderID","volEntered","minVolume","bid","issueDate","duration","stationID","solarSystemID"],
  "rowsets" : [
    {
      "generatedAt" : "2011-10-22T15:43:00+00:00",
      "regionID" : 10000065,
      "typeID" : 11134,
      "rows" : [
        [8999,1,32767,2363806077,1,1,false,"2011-12-03T08:10:59+00:00",90,60008692,30005038],
        [11499.99,10,32767,2363915657,10,1,false,"2011-12-03T10:53:26+00:00",90,60006970,null],
        [11500,48,32767,2363413004,50,1,false,"2011-12-02T22:44:01+00:00",90,60006967,30005039]
      ]
    },
    {
      "generatedAt" : "2011-10-22T15:42:00+00:00",
      "regionID" : null,
      "typeID" : 11135,
      "rows" : [
        [8999,1,32767,2363806077,1,1,false,"2011-12-03T08:10:59+00:00",90,60008692,30005038],
        [11499.99,10,32767,2363915657,10,1,false,"2011-12-03T10:53:26+00:00",90,60006970,null],
        [11500,48,32767,2363413004,50,1,false,"2011-12-02T22:44:01+00:00",90,60006967,30005039]
      ]
    },
    {
      "generatedAt" : "2011-10-22T15:43:00+00:00",
      "regionID" : 10000065,
      "typeID" : 11136,
      "rows" : []
    }
  ]
}

As shown in the purposed version multiple data groups would be allowed for different typeIDs. It would also be possible to have the same typeIDs but different regionIDs or both combined.

History

History, example:

{
  "resultType" : "history",
  "version" : "0.1",
  "uploadKeys" : [
    { "name" : "emk", "key" : "abc" },
    { "name" : "ec" , "key" : "def" }
  ],
  "generator" : { "name" : "Yapeal", "version" : "11.335.1737" },
  "currentTime" : "2011-10-22T15:46:00+00:00",
  "columns" : ["date","orders","quantity","low","high","average"],
  "rowsets" : [
    {
      "generatedAt" : "2011-10-22T15:42:00+00:00",
      "regionID" : 10000065,
      "typeID" : 11134,
      "rows" : [
        ["2011-12-03T00:00:00+00:00",40,40,1999,499999.99,35223.50],
        ["2011-12-02T00:00:00+00:00",83,252,9999,11550,11550]
      ]
    }
  ]
}

Field Explanations

Field JSON Type Description
resultType string The type of upload. Valid values are “orders” and “history”
version string A Major.Minor number with optional addition “alpha”, “beta”, etc signifying the data format version. Once format is fixed this will start with “1.0” and only change if format is updated
uploadKeys array An array of objects containing per endpoint upload keys
generator object An object with a “name” and “version” pairs to identify the JSON message generator
currentTime string The current data/time in ISO 8601 format (example 2011-10-22T15:46:00+00:00) when the JSON message was created. It may also be changed by any forwarders as well when the message is re-sent
columns array An array of column names for the data rows. See rows format descriptions below for expected column names
rowsets array An array of data objects as specified in Rowsets

UploadKeys

Name JSON Type Description
name string The name (initials) of the endpoint the “key” is for
key string The “key” used by endpoint to ID the 'source' the data is from. This can be a simple ID number, or some kind of message digest. See the Key Types section for more in depth information

Generator

Name JSON Type Description
name string The name of the generator (Application, Library, etc) used to generate (create) the JSON message
version string An undefined version string used in identify the version of the JSON message generator

Rowsets

Name JSON Type Description
generatedAt string The data/time in ISO 8601 format (example 2011-10-22T15:46:00+00:00) when the rowset was created. This allows tracking the 'freshness' of the contained data which is important since data from multiple sources can be generated and received at different times
regionID null OR number Id of the region. Can be null if client lacks data
typeID number The typeID of the upload, for history and orders
rows array Contains the actual data rows. See Order Rows and History Rows descriptions below
Order Rows
Name JSON Type Description
price number The price of the order
volRemaining number Remaining volume in order
range number The range the order extends
orderID number The ID of the order
volEntered number The starting volume of the order
minVolume number The minimum volume that can be ordered
bid true OR false Whether the order is a bid (possible: true, false)
issueDate string The date/time the order was issued in ISO 8601 format (example 2011-12-03T10:53:26+00:00)
duration number The amount of days the order will be active
stationID number The station where the order was placed
solarSystemID null OR number The solar system where the order was placed. Data can be null if client lacks information for result rows

The order of the 'columns' values determines the order that the 'rows' values should be in. The above order is preferred but should not be assumed by any software decoding the rows instead they must assure they map them in the order given by 'columns'.

History Rows
Name JSON Type Description
date string The date/time the history was issued in ISO 8601 format (example 2011-12-03T00:00:00+00:00)
orders number The number of orders
low number The lowest price of the orders
high number The highest price of the orders
average number The average price of the orders
quantity number The quantity of items in the orders

The order of the 'columns' values determines the order that the 'rows' values should be in. The above order is preferred but should not be assumed by any software decoding the rows instead they must assure they map them in the order given by 'columns'.

Comparisons

API MarketOrder API columns and Eve cache export columns for comparison:

orderID,charID,stationID,volEntered,volRemaining,minVolume,orderState,typeID,range,accountKey,duration,escrow,price,bid,issued price,volRemaining,typeID,range,orderID,volEntered,minVolume,bid,issueDate,duration,stationID,regionID,solarSystemID,jumps

Key Types

How each endpoint (application, web site , whatever) uses its “key” is up to them. Any of the other parts of the JSON message maybe used in making the “key” with the restrictions that “uploadKeys” and “currentTime” can not be included. Forwarders should not modify anything outside of “uploadKeys” and “currentTime” in the message. If they do they should remove any array elements for endpoints in “uploadKeys” where they don't understand how the “key” is formed to prevent a possibly bad message from being past to that endpoint which can cause unnecessary extra work for them. All endpoints should publish information about any parts of the message that are used in creating their “key” so anyone developing a forwarder can know what things to watch out for. It is recommended they also document the formula or process used to form their “key” so it can be implemented by others. In the case of endpoints that use any kind of two part public-private encryption digest (think PGP) on the JSON message they should make sure the public part is easily accessible.

Data/Times

So to clarify The ISO 8601 standard and how it should be used in this format. As in all the examples above it needs to be in full data and time including time zone offset format. The zone zone information is important as date/times without it are assumed to be in local time which will cause problems since often the data will be sent to sites in other time zones. The standard allows for the time offset to be done as just 'z' if using UTC but it is recommend everyone use the +00:00 form instead as it would make manual decoding of the offset easier. It is recommended that everyone writing a decode take both cases into consideration as using 'z' instead will not be considered invalid just not recommended per the format.

unifieduploader/start.txt · Last modified: 2012/05/25 17:27 by jimi