Allow both the EVE-Cental and EVE-Metrics market uploaders to publish to one location via an agreed upon data format.
The protocol used internally will be AMQP.
The protocol used by upload clients to communicate with the unified upload receiver will be HTTP (specifically POST).
data - String, CSV data with headers
sort - String, order, history or other sort of upload
target - String, Optional, if specified denotes a target AMQP exchange
identity - String, in the format “em:HASH;ec:HASH;” - either may be omitted eg “em:;ec:HASH;”. Hash is a unique ID for a given user
client_identity - String, as above, to identify a given upload client. May be a simple name or number as opposed to a hash, or a hash (em uses developer keys)
client_version - String, version number/name for a given upload client
type - Optional (Required for order/history uploads), the typeID of the uploaded data
region - Optional (Required for order/history uploads), the regionID of the uploaded data
generated_at - Time the data being uploaded was generated
hash - SHA1 of the string formed by “data+sort+target+identity+client_identity+client_version+type+region+generated_at” for error checking, lowercase hex-encoded.
200 - All is well, upload forwarded
403 - Upload client or client version not accepted
400 - SHA1 hash inconsistency
418 - Any other error (Really can't think of any, hence the code)
The response will be an error code of one of the above. Any response will be plaintext with no formatting and should be displayed to the user in any log in the upload client so that custom error messages may be implemented server-side. If the response body is empty then the above messages may be used as defaults.
A HTTP server running on zofu's box will take the HTTP requests, perform the consistency checks and filter out any client versions known to have bugs or be uploading bogus data, and push the received data onto a given AMQP exchange- either order_uploads, history_uploads or another exchange based on the target value (this will be restricted to a list of exchanges). The arbitrary exchange option allows for non-market data to be uploaded using the same mechanism (loot logs, fittings, etc), allowing the unified upload architecture to work with pretty much anything.
The upload receiver will obscure the em/ec identity hashes using a secret salt. This is needed to give public feed consumers the ability to match uploads to a user (whose identity remains unknown), while protecting em/ec from upload spoofs. These resalted messages will be pushed onto the public exchanges, with the unsalted messages being confined to private exchanges for EM/EC.
The messages pushed onto the exchange will be YAML formatted and include all the HTTP POST parameters that are received.
Logging will be performed to the extent of one line per request noting source IP, the identity/client_identity/client_version parameters and the response, with logs rotated weekly. The bridge will have facilities for blocking specific IP addresses, identities and client versions.
do the error checking hash over all other field values, ordered by key. this is actualy less work than doing it for just a subset, and more flexible.
specify what encoding is used for the SHA1 parts. common choices would be hex or base32. specify whether those are upper or lower case.
unless there is some way for the upload receiver to verify the identity strings being valid, the “one must be filled” is somewhat pointless.
EM generates short random string → USERKEY
EM generates hash(USERKEY+EMSALT1) → USERHASH
USER puts USERKEY+USERHASH into uploader
HTTP receiver knows EMHASH1, can check if USERKEY+USERHASH is a valid pair
HTTP receiver does hash(USERKEY+USERHASH+EMSALT2) → FEEDHASH
The 'one must be filled' part was assuming that EC required authentication on uploaded data. As this is actually not the case it can be discarded. Merged the rest. —
James Harrison 2009/12/22 17:08
