Observations of EDDN Journal Schema usage by tools

[vanderaj]

interfaces.zip
During a recent EliteBGS update to strongly type Journal data from EDDN to better allow data validation of incoming data, as well as incorporate local and improve tick detection, it was necessary to look at the state of data accepted by EDDN to understand what data we might see.

Methodology. I investigated ross.eddb.io looking for how different tools uploaded various types of journal schema messages. The systems used are: Deciat (populated, active, pending, and recovering states, many factions), HIP 36601 (anarchy with no factions but heaps of carriers), Maia (retreat pending, many active states), and HR 1172 (anarchy, 1 megaship, zero factions or pop). If replicating this research, please make sure you use systems with many different states as possible, but within a single tick. Note: I limited my research to tools that had results within the last tick in interesting systems, so the lesser-used tools have no representation even in busy systems like Deciat or an unpopulated system like HIP 36601.

As a result, I have the following recommendations for EDDN and tool data providers:

journal/1schema improvements:

Simplify the message.faction to have a single class Factions?: FactionElement[]

Some tools provide a Faction[] array, which is a subset of FactionElement[]

EDDI Journal FSD Jump subschema, populated system

export interface Faction {
  activeStates?: ActiveState[]
  allegiance?: string
  factionState?: string
  government?: string
  happiness?: string
  influence?: number
  name?: string
}

EDDI - Journal Location subschema, populated system

export interface FactionElement {
  allegiance?: string
  factionState?: string
  government?: string
  happiness?: string
  influence?: number
  name?: string
  recoveringStates?: IngState[]
  activeStates?: ActiveState[]
  pendingStates?: IngState[]
}

This would simplify faction storage for EDDN and consumers, and improve faction data validation and processing by tools like EliteBGS, EDSM, or Inara. Where game journal data doesn't exist, by making the data optional within a common FactionElement will help simplify things.

Simply the faction state used by Station and System Factions

Most tools: Docked, FSD Jump, Location

stationFaction?: StationFaction
systemFaction?: SystemFaction

export interface SystemFaction {
  factionState?: string
  name?: string
}

export interface StationFaction {
  name?: string
}

Some tools, like GameGlass and EliteLogAgent

stationFaction?: StationFactionClass
systemFaction?: StationFactionClass

export interface StationFactionClass {
  factionState?: string
  name?: string
}

NB: Note that StationFactionClass is different to most tools' definition of it, but it's the same as the other tools' SystemFaction definition.

Recommendation: Make a single FactionClass element that has the following definition:

export interface FactionClass {
  name: string
  factionState?: string
}

Please with tool authors to submit a common definition for system and station faction updates, regardless of location, FSD Jump, populated or not, or docked.

Missing fields by certain tools

EliteBGS requires faction updates, and we care about little else, other tools and sites require a lot more detail than EliteBGS. Many tool data providers do not seem to supply data that might be possible to supply from the journal. I would encourage all tools to review the included .zip file to see if they could add additional missing fields in their updates to EDDN.

Noting that the game's journal may be missing data depending on gameplay or indeed client version (Odyssey versus Horizons), I have tried to account for this by using a single tick with different tools in a busy system used by many CMDRs (in this instance Deciat) using a wide variety of tools. Unpopulated or faction-less systems shouldn't have such variation, because gameplay journal entries should be very similar regardless of tick. I have tried to use the most recent journal entries to account for differences in EDH / EDO clients over time reporting variations as bugs are fixed or game loops change. Additionally, I only took into account one record from each tool, which could be a mistake. A better study might sample all records and then work out a maximal set for each tool in a single tick. Where a tool does not appear doesn't mean it doesn't have missing fields, it simply means there was insufficient data during the tick.

Some of the differences are explained by the presence of a populated system or not, or if the jump occurs from deep space or near a station (so gameplay may have affected these results!). This is perfectly acceptable, as tools can't supply data that is not in the game's journal.

Recommendation: Toolmakers should review their code compared to entries from EDDN from many, many players, their own tools' records and other tools' records to see if they are providing all the relevant fields.

Docked

• EDDI - body, bodyType
• ED Discovery - body, bodyType, multicrew, landing pads, taxi
• ED Lite - body, bodyType, multicrew, stationAllegiance
• EDMC - horizons, stationAllegience
• EliteLogAgent - body, bodyType, horizons, landingpads, multicrew, odyssey, stationAllegience, taxi
• G19s - body, bodyType, horizons, landingpads, multicrew, odyssey, taxi

FSD Jump - unpopulated

• EDSM Console - docked, horizons, marketID, multicrew, odyssey, stationEconomies, stationFaction, stationGovernment, stationName, stationServices, stationType, taxi
• ED Discovery - stationEconomies, stationFaction, stationGovernment, stationName, stationServices, stationType, taxi
• EDMC - horizons, multicrew, taxi
• GameGlass - docked, horizons, marketID, odyssey, stationEconomies, stationEconomies, stationFaction, stationGovernment, stationName, stationServices, stationType

FSD Jump - populated

• ED Discovery - multicrew, taxi
• EDMC - horizons
• EliteLogAgent - horizons, multicrew, odyssey, taxi
• GameGlass - horizons, multicrew, odyssey, taxi

Location - unpopulated

• EDDI - distFromStarLS, multicrew, taxi
• EDMC - horizons
• ED Discovery - distFromStarLS, taxi
• EliteLogAgent - distFromStarLS, multicrew, odyssey, taxi
• G19s - odyssey
• Gameglass - distFromStarLS, multicrew, odyssey, taxi

Location - populated

• EDDI - distFromStarLS, marketID, stationAllegiance, stationEconomies, stationEconomy, stationFaction, stationGovernment, stationName, stationServices, stationType
• ED Discovery - distFromStarLS, powerPlayState, Powers
• EDMC - distFromStarLS, horizons, stationAllegiance
• EliteLogAgent - horizons, odyssey, stationAllegiance, powerPlayState, Powers
• G19s - horizons, odyssey, multicrew, taxi, powerPlayState, Powers
• GameGlass - horizons, odyssey, stationAllegiance, powerPlayState, Powers

These differences might be explainable by where Cmdrs landed or popped up. More research is required, but by honing in on missing fields, it might be a simple fix to see if the tool actually uploads this data at all, or it was just missing from the data I inspected.

[vanderaj]

During discussions, there's a good argument for eliminating taxi, multicrew, and realistically Odyssey vs Horizons is of limited value too. EliteBGS cares primarily about the missing system and faction information, and would appreciate any improvements to the quality of data supplied, such as marketID, stationFaction, etc.

[klightspeed]

Multicrew and Taxi attributes are only present in the journal when you are in physical multicrew or in a taxi . These should probably be removed, as they indicate what mode of transport the player took.

horizons and odyssey are added to events by some tools (copied from the LoadGame event) to indicate which game client generated the data.

PowerPlayState and Powers are only present when a player is pledged to a power and the system is controlled or exploited by a power.

[Tkael]

Docked - body, bodyType. These are not an omission relative to the schema. Rather, EDMC is sending extra data not currently called for by the schema. https://github.com/EDCD/EDDN/blob/master/schemas/journal-v1.0.json

Location (populated or otherwise) - Many of the noted fields are not always written to the player journal, depending on the specific location of the commander. Station data for example is only written when docked. DistFromStarLS may not be written if "close to the main star" per the journal manual. This may be less a problem with consistency between clients and more a problem with inconsistent source data.

[DarkWanderer]

@klightspeed is correct about powers and state flags

stationAllegiance is sometimes missing from source events in the journal itself - it's a "known" if rare bug

[robbyxp1]

My observations are that re-engineering any of the current schema to an updated schema is high cost. There are multiple listeners out there.

I think you need to accept that the responsibility is on the listeners to be robust. Accept that clients will be in a variety of states (there is no way to mandate updates either by the programmers or the users).

[robbyxp1]

Odyssey/Horizons must stay, we need to know what client is sending data for tools like EDDB.