edcd / eddn Goto Github PK

It is proposed that the FSSDiscoveryScan event be allowed in the Journal schema.

Please discuss any objections, caveats or possible problems with this.

The tables for 'Software' are sorted on 'Total hits' descending. When making a new release of a software it's more interesting to look at the 'Today' and 'Yesterday' hits numbers. Obviously this can be done by eye, but it would be more convenient if a user could click on the requisite column heading to have the sorting switch to it (and toggle ascending/descending if already selected).

Obviously then either the Highchart pie chart would need some labelling or even better, switch to the selected column's data.

The same could be applied to the Schemas data table.

Recent changes have removed the older schemas from the master branch. They're also not hosted on http://schemas.elite-markets.net/eddn. This makes finding an authoritative copy of the older schemas much harder than it should be for when tools transmit data using the older schemas. It would be preferable if the older schemas remained available. I understand the desire to push producers to new schemas, but consumers still have to deal with older data. At present, I have to look back through the commit history and hope I find the schemas as were published. Assistance appreciated.

In several schemas, including for example Commodities, MarketId is specified to be "number" instead of "integer". This means that a producer which strictly conforms to schema may send market ID as "1234567.0".

Some consumers (EDDB) consider ".0" suffix incorrect. I suggest to change the marketId field type to be integer, to avoid confusion

It would be great if each message schema had its own topic.

A simple map of schema ref to topic:

• http://schemas.elite-markets.net/eddn/commodity/[Any_Version] -> commodity
• http://schemas.elite-markets.net/eddn/shipyard/[Any_Version] -> shipyard
• http://schemas.elite-markets.net/eddn/outfitting/[Any_Version] -> outfitting

The HTTP API would not have to change. I believe all current subscribers are subscribing to ALL, meaning adding topics would not affect them.

I'm writing a service for Coriolis and I have to discard commodity messages at the moment. This is far from a huge issue, but it does use a decent amount of bandwidth since these messages are large and frequent.

Thoughts? 🎱

At present, the commodity schema allows unknown elements.

As EDDN matures, this should be made stricter, so that only defined elements are permitted.

Is there a way to subscribe to only one of the schemas (e. g. Commodity v3) as a consumer? I've googled around but it seems there's no way of getting a list of topics from the EDDN endpoint as a subscriber, nor did I find any information on how a consumer could set topic filters on EDDN, if there are topics at all to subcribe to.

Thank you :)

With the introduction of Fleet Carriers, it is entirely possible for a station to have a commodities market, but nothing actually available to buy or sell. It’s important to be able to send this information to EDDN, since otherwise a station’s market information can get “stuck” reporting stale data. For example, imagine the following scenario:

1. A Fleet Carrier is selling 1 ton of Meta-Alloys (and nothing else).
2. A ship docks at the Fleet Carrier and sends its market information to EDDN. Naturally, EDDN reports that the Fleet Carrier’s market is selling Meta-Alloys at a particular price.
3. Someone purchases the single ton of Meta-Alloys. Now, rather than displaying as selling Meta-Alloys with a supply of 0, the Fleet Carrier information actually claims it’s buying and selling nothing at all!
4. Because it’s impossible to report this “commodityless commodities market” information to EDDN, the market information is forever “stuck” claiming the station is selling Meta-Alloys, even though none are left in stock.

I opened a downstream issue for EDMC’s reporting of this situation at EDCD/EDMarketConnector#721, but the fix is ultimately blocked on a change to EDDN itself to support empty lists of commodities in the API.

Planetary Vehicle Hangar comes in ratings G and H.
outfitting/1 schema doesn't allow these.
I get:
FAIL: [<ValidationError: "{'category': 'internal', 'rating': 'H', 'name': 'Planetary Vehicle Hangar', 'class': '2'} is not valid under any of the given schemas">]

Hi,

since a few days the states are not updated anymore:

http://eddn.ed-td.space/#software
http://eddn.ed-td.space/#uploaders
http://eddn.ed-td.space/#schemas

The HTTP method OPTIONS is used by many browsers in order to perform a 'pre flight' check on CORS setup.

Currently the bottle-powered Relay.py code doesn't support this:

Bottle v0.12.15 server starting up (using GeventServer(keyfile='/home/eddn/etc/privkey.pem', certfile='/home/eddn/etc/fullchain.pem'))...
Listening on http://0.0.0.0:9090/
Hit Ctrl-C to quit.

<SSLSocket fileno=24 sock=192.168.1.7:9090 peer=A.B.C.D:53070>: Invalid HTTP method: 'OPTIONS\n'
A.B.C.D - - [2021-05-17 09:06:03] "OPTIONS" 400 - 2.790872

The Monitor also utilises bottle for some direct API access (i.e. it doesn't go through a reverse proxy), so that will need addressing as well. I don't think the Gateway has anysuch.

The lack of OPTIONS support causes, e.g. Firefox 88, to not load the stats for some time. I'm actually not sure why it does eventually work.

With the way the schema validation, performed on the Gateway, is specified it might start getting convoluted to disallow some fields only for some events and not others.

One proposal is to just split the Journal schema out into a schema per event that is allowed.

We don't allow that many events over EDDN so it's not like this will explode out into 10s more schemas, let alone 100s.

It is proposed that the FSSAllBodiesFound event be allowed in the Journal schema.

Please discuss any objections, caveats or possible problems with this.

Every so often the gateway stops responding to HTTP requests. The logs show:

Traceback (most recent call last):
File "/opt/eddn/env-2.7/lib/python2.7/site-packages/gevent-1.0.2-py2.7-linux-x86_64.egg/gevent/baseserver.py", line 140, in _do_read
File "/opt/eddn/env-2.7/lib/python2.7/site-packages/gevent-1.0.2-py2.7-linux-x86_64.egg/gevent/server.py", line 93, in do_read
File "/opt/eddn/env-2.7/lib/python2.7/site-packages/gevent-1.0.2-py2.7-linux-x86_64.egg/gevent/socket.py", line 309, in accept
client_socket, address = sock.accept()
error: [Errno 24] Too many open files
WSGIServer at 0x7f07cd469190 fileno=12 address=0.0.0.0:8080 failed with error

It looks like the Bottle/gevent web server isn't closing connections, as there are hundreds left open at the moment.

This is the cause of the last several EDDN outages.

I was changing the EDAPI plugin of TD (Trade Dangerous) to use the FdevIDs mapping for the names.

I tried to include the "id" field in the commodities and outfitting message but it returned an validation FAIL.

The "id" fields are part of the current schemas (commodity-v2.0.json and outfitting-v1.0.json).

I added a comment at issue #26 but got no reaction so far.

Hi there.

at the moment I try to send with the v2 schema (RegulatedNoise) but I've got some trouble
and I don't know, whats wrong. Maybe someone see the reason for the error...

I get follwing answer:
"FAIL: [<ValidationError: "{'name': 'Biowaste', 'supply': 23731, 'supplyLevel': 'Med', 'buyPrice': 17, 'demand': 0, 'sellPrice': 13, 'demandLevel': u''} is not valid under any of the given schemas">]"

this is the send string (I changed "http" to "h__p" to avoid viewing problems):

{"header":{"softwareVersion":"1.84_0.26","gatewayTimestamp":"2015-06-20T16:28:17+02:00","softwareName":"RegulatedNoise__DJ","uploaderID":"Duke Jones -[Bla]-"},"$schemaRef":"h__p://schemas.elite-markets.net/eddn/commodity/2/test","message":{"commodities":[{"name":"Biowaste","buyPrice":17,"supplyLevel":"Med","supply":23731,"demand":0,"sellPrice":13,"demandLevel":""},{"name":"Container Mit Sap 8-Kern","buyPrice":0,"supplyLevel":"","supply":0,"demand":104480,"sellPrice":60252,"demandLevel":"High"},{"name":"Hydrogen Fuel","buyPrice":106,"supplyLevel":"Med","supply":407000,"demand":0,"sellPrice":101,"demandLevel":""},{"name":"Domestic Appliances","buyPrice":0,"supplyLevel":"","supply":0,"demand":8437,"sellPrice":530,"demandLevel":"Low"},{"name":"Clothing","buyPrice":0,"supplyLevel":"","supply":0,"demand":75284,"sellPrice":348,"demandLevel":"Med"},{"name":"Tauri Chimes","buyPrice":924,"supplyLevel":"High","supply":11,"demand":0,"sellPrice":924,"demandLevel":""},{"name":"Consumer Technology","buyPrice":0,"supplyLeve
l":"","supply":0,"demand":15777,"sellPrice":7442,"demandLevel":"Med"},{"name":"Beer","buyPrice":0,"supplyLevel":"","supply":0,"demand":171223,"sellPrice":283,"demandLevel":"Med"},{"name":"Liquor","buyPrice":0,"supplyLevel":"","supply":0,"demand":17813,"sellPrice":825,"demandLevel":"Med"},{"name":"Tobacco","buyPrice":0,"supplyLevel":"","supply":0,"demand":11548,"sellPrice":5006,"demandLevel":"Med"},{"name":"Wine","buyPrice":0,"supplyLevel":"","supply":0,"demand":151201,"sellPrice":376,"demandLevel":"Med"},{"name":"Power Generators","buyPrice":0,"supplyLevel":"","supply":0,"demand":1427,"sellPrice":530,"demandLevel":"Low"}],"timestamp":"2015-06-20T16:28:17+02:00","systemName":"39 Tauri","stationName":"Porta"}}

Hi. I'm looking to add support for Market events in EliteLogAgent and has hit a data sourcing issue. There is no data about items prohibited on a station in log anywhere - however, I can still provide the market commodities and pricing. Is there a way to send a message omitting the 'prohibited' value?

The NavRoute.json file, written when a journal NavRoute event occurs, contains:

1. System Name
2. System co-ordinates
3. SystemAddress (id64)
4. Main star's class

for each step in the plotted route. It would be simple to take the route array out of the file and insert it into the journal event before sending that over EDDN. Then any listeners could choose to utilise it to potentially gain co-ordinates and systemaddress for extra systems.

As the provided data in the route array is what is currently required as the bare minimum in a journal schema message this shouldn't actually require any changes to the schema other than adding the event type to the allowed list. This issue mostly for reference and any further discussion. It would require a small update to the documentation to explicitly state that the route array needs adding from the file.

I'd like to propose an addition to the schema to carry station outfitting data. The proposed change to the schema is below - basically it just adds an optional "modules" property which is an array of integer IDs.

The semantics of the "modules" array is that it represents the availability of outfitting at the station at the time of reporting. Each element in the array is an integer ID which represents a module available in the station's outfitting screen. An empty "modules" array indicates that the station does not provide outfitting. (Publishers that do not provide outfitting info should therefore omit the "modules" property rather than supply an empty array).

Here is a not-quite exhaustive list of mappings from IDs to modules. I'd like to also propose that the EDDN project own and maintain this list.

Some Q&A:

Q: Why use integer IDs rather than strings (like we do for the commodity names)?
A: Using IDs does add an extra level of complexity/indirection, but offers some advantages over strings:

• Less ambiguity - no chance of variant spellings etc.
• Easier for non-English speaking subscribers to translate.
• More robust in the face of updates to E:D - subscribers can process IDs and store them in their databases even if they don't (yet) know how to display them.
• More compact - about a third of the size of the corresponding strings.

Q: For consistency, why not convert the commodities array to also use integer IDs instead of string names?
A: Converting the commodities array to use IDs would be a barrier to adoption. The proposed schema change is narrowly scoped in order to be trivial for subscribers to adopt - subscribers can just handle the proposed schema identically to the v2 schema by ignoring the new "modules" property. They can then choose to exploit the outfitting data encapsulated in the "modules" property later, or not at all.

Q: What about prices?
A: Unlike commodities, module prices don't typically vary from station to station. They may vary according to other factors including the player's allegiance, ranking, discounts etc but these are necessarily player-specific.

Q: What if the station provides outfitting, but doesn't have a commodities market?
A: The publisher should supply an empty "commodities" array.

Q: What about shipyard info (c.f. issue #16) ?
A: A new "ships" property could work in the same way and have similar semantics to the proposed "modules" property. Here is a not-quite exhaustive list of mappings from IDs to ships.

The proposed addition to the schema:

--- schema2.xml 2015-06-02 02:43:35.000000000 +0100
+++ schema3.xml 2015-06-12 00:20:06.000000000 +0100
@@ -1,6 +1,6 @@
 {
   "$schema": "http://json-schema.org/draft-04/schema#",
-  "id": "http://schemas.elite-markets.net/eddn/commodity/2#",
+  "id": "http://schemas.elite-markets.net/eddn/commodity/3#",
   "type": "object",
   "additionalProperties": false,
   "properties": {
@@ -96,6 +96,15 @@
                       }
                       ]   
           }
+        },
+        "modules": {
+          "type": "array",
+          "additionalProperties": false,
+          "uniqueItems": true,
+          "items": {
+            "type": "integer",
+            "additionalProperties": false
+          }
         }
       },
       "required": [

Once other code cleanup is finished (i.e. ensuring all services log proper IP addresses, not 127.0.0.1 due to reverse proxying) we should investigate enabling IPv6 connectivity for both uploads to the Gateway and listening on the Relay.

The current host has IPv6 addresses and this would introduce an extra wrinkle to changing servers in the future, but that's a problem for just that, the future.

The uploaderId is currently beeing scrambled every 12 hours and there is no option to avoid that.

The uploaderId should be scrambed and regenerated by default, but an additional attribute/option in the uploaded document should allow to disable this feature for those who want their Plaintext uploaderId visible to the public.

• The monitor files aren't installed by setup.py
• The schema files aren't installed by setup.py
• requirements.txt contents work, but we might be able to increase the version on some things whilst still on Python 2.7.
• Add example Apache ProxyPass setup if someone isn't using nginx.
• The /stats/ URL used for the web page is served from Relay.py, when I would have expected Monitor.py to take care of this. This was a misunderstanding of the architecture.
• Monitor schema stats assumed there WILL be 'yesterday' stats returned. Not the case when testing with a fresh database.

NEVER MIND. I must have made a mistake. This is fixed and I made my first successful EDDN update. :)

Is this the right place to discuss issues that I'm having with a publisher I'm writing? I already posted about this at the Frontier forums. I am DRY411S at those forums.

Perhaps that wasn't the right place?

https://forums.frontier.co.uk/showthread.php?t=57986&page=54&p=3433064#post3433064

I've been playing with the sample code for Python v34 and it does very little given that the vast majority of updates are from EDMC and that is not authorised in the example code.. Would you welcome a push that supports the Commodity v3 schema and EDMC?

Whilst https://github.com/EDCD/EDDN/wiki has mention of 'horizons' it doesn't actually document what the flag is, and when it should be set (in EDMC code it's if we see certain station/port types, or known horizon SKUs, or ...).

The upcoming 'odyssey' flag is different as it is to be sent based purely on the LoadGame event having an Odyssey flag and as per its value. False if running under Horizons and the key isn't even in the event.

Reminder for new schema to discuss.

I think this line:
https://github.com/EDSM-NET/EDDN/blob/52a7061d8ff29c779b8d2313f2156462b3bc899d/src/eddn/Relay.py#L65
causes troubles with non ascii commander names.
This should fix it:

return hashlib.sha1("{}-{}".format(self.uploader_nonce, uploader.encode('utf8'))).hexdigest()

I'm a bit of a git noob here, so please bear with me. Was attempting to run the client.py file, I install pyzmq, and made the change to the print statement to accommodate python 3 syntax.

Problem comes with:

import zmq.green as zmq
...
ImportError: No module named 'gevent'

After spending a bunch of time trying to install gevent on a python 3 system, I eventually gave up, was wondering if gevent was needed for the example code.

It is proposed that the NavRoute event be allowed in the Journal schema. Specifically this is for picking up the data about all systems mentioned in the NavRoute.json file.

Please discuss any objections, caveats or possible problems with this.

The Monitor (for the stats) and Relay (to save outbound bandwidth) detect if a message is a duplicate based on recent messages.

If the Gateway also did this detection then it could return a HTTP status code and message to tell the uploading software that the user likely has another software also sending to EDDN (it might also be a rather quick replay). Then the software could choose to inform the user so that they can change settings to ensure they're only sending each message to EDDN once.

The changes to Gateway.py should be fairly simple, but we need:

• To agree on the HTTP status code, and any in-body message.
• Have the major softwares set up to properly handle this, so as to not spam the user with an error per EDDN message.

EDO Update 5 is adding an array of available pads/sizes to both Docked and DockingRequested.

We might feel we get enough data about available pads at EDO settlements (specifically, but obviously it'll work for anywhere you could possibly dock) from just the enhanced Docked events, but perhaps we want to gather it from DockingRequested as well for those cases where the player was in a too-large ship.

I don't think there's any utility to adding DockingDenied because:

1. If it's due to occupied pads, a legality reason, or ship too large, we'd have picked it up from DockingRequested immediately prior.
2. If the settlement literally has no pads then you won't even be able to request docking, the option won't be on External/Contacts/.

It is proposed that the ApproachSettlement event be allowed in the Journal schema.

Please discuss any objections, caveats or possible problems with this.

For example:

  {
    "name": "Imperial Slaves",
    "buyPrice": 0,
    "supplyLevel": null,
    "demandLevel": "Low",
    "supply": 0,
    "demand": 105,
    "sellPrice": 15859
  },

New shipyard schema isn't visible here: http://schemas.elite-markets.net/eddn/shipyard/1

Something's happened to the StatsCollector, and it no longer does what it should. The reported values for 1min are always zero.

Steps to reproduce:

• Run a gateway
• Submit a message
• Visit /stats/ and see that tallies have been added for inbound (and either outbound or invalid)
• Wait a minute
• Hit refresh

Expected outcome: a string such as

{"inbound": {"1min": 1, "60min": 1, "5min": 1}, "uptime": 62, "version": "0.5", "invalid": {"1min": 1, "60min": 1, "5min": 1}}

Actual outcome:

{"inbound": {"1min": 0, "60min": 1, "5min": 1}, "uptime": 62, "version": "0.5", "invalid": {"1min": 0, "60min": 1, "5min": 1}}

Because of the heavy value people have placed on the stats pages, I'm blocking the 0.5 release on this issue.

This is a regression since v0.4.

There are some events that are meant to have StarPos added, for example. This is currently not documented on the Wiki page.

Some one docked but we don't know at which station :(

Fundamentally you could include almost all the information from the Docked event from the Journal
which does include the StationName and MarketID,DistanceFromStar etc.

Along with whole bunch of other "useful" information.

Would it be possible to add events to the Journal schema (or create a dedicated one?) to handle market trade events (MarketBuy & MarketSell)?

More than commodities price, that would give an idea of volume exchanges and allow for some nice consequences:

• Predicting market prices variations on next update
• Best trades discovery, universe-wise (leaderboard?)
• Track Community Goal participation & stats estimations (since missing the individual sign-up status): near-realtime participants, overall progress, tiers' progress, tier's threshold, etc.
• ... any other market stats based on individual transactions collection you might imagine! :)

there are nothing that distinguish between special and normal modules.... like the shield generators and the prismatic shield generators

Considering 2.4 update to FSDJump Event on population are you including that on the Journal Schema ?

https://forums.frontier.co.uk/showthread.php/372993-2-4-Open-Beta-Patch-Notes

Why just market? i need update a planet discovery, how to use its? what app?

sry english

It is proposed that the following Fleet Carrier related events be allowed in the Journal schema:

• CarrierDecommission
• CarrierCancelDecommission
• CarrierJumpRequest
• CarrierJumpCancelled
• CarrierDockingPermission
• CarrierNameChanged

Please discuss any objections, caveats or possible problems with this.

See this issue: EDCD/EDMarketConnector#113

Particularly: EDCD/EDMarketConnector#113 (comment)

hi,

I was the last developer of "RegulatedNoise DJ" and now
it's successor "ED-IBE" is available.

https://github.com/Duke-Jones/ED-IBE

I'm not sure if it's necessary but for safety
I would inform you about this tool, because it can send and recieve
EDDN messages.

It can have two different sender-ids
(depends on the used interface to interpret the trustworthiness of the data)

ED-IBE (API)
ED-IBE (OCR)

thank you,
Duke Jones

btw: "RegulatedNoise DJ-version" supports v2 messages for send and for recieving also v1 messages.
Maybe you can correct the information in your README.md

Hi,

I am having a bit of difficulty connecting to the data network with a node JS application. I was hoping you could point me in the right direction. I've based it mostly off of your py script, but converted it to node which zmq does have a library for. My basic connection code is as follows:

var zmq = require('zmq'),
subscriber = zmq.socket('sub'),
zlib = require('zlib'),
host = 'tcp://eddn-relay.elite-markets.net:9500';

subscriber.subscribe('');
subscriber.connect(host);

subscriber.on('message', function(reply){
console.dir('Received message: ', reply.toString());
/* I understand I'm not unzipping with zlib here, but the message listener never fires */
});

process.on('SIGINT', function(){
subscriber.close();
});

I get no errors, but also no output so I am rather confused.
Any help would be much appreciated. Thanks!