Specify Murmurations-compatible schema for SEA data

[ColmMassey]

SEA share their current schema/field definitions, with the intention of creating a Murmurations-compatible schema for SEA data

https://github.com/MurmurationsNetwork/MurmurationsProtocol/wiki/SEA-Murmurations-interoperability

[ColmMassey]

What do we think needs doing here?

[wu-lee]

See also notes here: https://nextcloud.solidarityeconomy.coop/index.php/s/miCmWMEpPCZmm8n

Ok, just thinking out loud here for the moment. There are two directions of exchange:

1. SEA LOD -> Murmurations
2. Murmurations -> SEA LOD (and so to a map)

The first case probably needs (pending a closer look at their docs):

• Publishing files consumable by Murmurations along with our static LOD data (JSON-LD?)
• Registering one or more of those with the Murmurations aggregator (if that's the right term) to be consumed
• Notifying the Murmurations aggregator of changes?

The second case probably needs:

• Murmurations adding a web API call on their aggregator to query the list of some category of entities in a form which can be converted into LOD by our open-data processing service (the sausage machine).
• Ideally that would publish an ETAG header or something equivalent indicating when the content has changed
• Creating a data set within open-data targeting that API as a source

I think we can probably go ahead with implementing the first, and ask for help with the second when we show the Murmurations people.

[wu-lee]

A bit more research makes me suspect that we'd need to publish a schema somewhere using this API: https://github.com/MurmurationsNetwork/MurmurationsProtocol/blob/master/library_api-v1.md

And an index of entities employing that schema, using this API: https://github.com/MurmurationsNetwork/MurmurationsProtocol/blob/master/index_api-v1.md

That index would refer to the actual data stored on our servers.

[ColmMassey]

Added a note about what Adam is doing for Global Ecovillages Network at end of this page.

https://nextcloud.solidarityeconomy.coop/index.php/s/miCmWMEpPCZmm8n

[ColmMassey]

Shall we attempt to do 1 with a small graph of data where we are not worried about licensing issues? How about the Oxford data?

[ColmMassey]

Can I suggest you write the tickets to do 1 for the Oxford data and add them to

https://github.com/orgs/SolidarityEconomyAssociation/projects/14

Then invite Adam to take a look at them?

[wu-lee]

I might also draft a schema definition for our data in their terms. This will overlap with our schema review.

[wu-lee]

I've invented a schema and generated some profile data for all the initiatives in the Solidarity Oxford project.

Pull request here (which includes the schema): MurmurationsNetwork/MurmurationsLibrary#34

Need to wait until that is accepted before I can submit out profiles to Murmurations' index server.

Our profiles are hosted here in files with .murm.json suffixes: https://dev.lod.coop/sea-lod/oxford/

Currently these are being rebuilt whenever Oxford gets rebuilt, as Oxford is able to use the experimental branch of the se-open-data repo which has the profile generation code.

An example of an individual profile: https://dev.lod.coop/sea-lod/oxford/65.murm.json

Any comments? It's deliberately fairly simple for now, as I don't want to invent any new field definitions just yet, only prove that we can connect. There's a name, description, IRI, a location ("Oxford, GB"), a geolocation, and some English-language tags taken from the "Organisational Structure" field. New fields of our own might include ones for ESSGLOBAL terms, or perhaps something more generic.

I've posted a fairly long list of questions on their Flarum forum, now waiting for my pull-request approval before submitting the profiles to see if they appear on the Murmurations' aggregator's map.

https://murmurations.flarum.cloud/

[wu-lee]

Ok, so I've managed to get my schema merged into the MurmurationsLibrary, and have started submitting profiles which are being generated already for the Oxford dataset (as mentioned above). I have adjusted the profiles slightly to fit the latest schema, and also linked to the murmurations_map-v1 schema, which requires an urls field. This replicates the website field, but I'm doing this in the spirit of experimentation (plus it avoids me making another PR to remove the website field in favour of the urls field, which is less specific, semantically).

This is working, in that many (if not all) of the initiatives are appearing on the Murmurations map aggregator here:

https://aggregator.murmurations.network/ (search for 'solidarity economy' to narrow the forest of pins down, for example)

The main problems are that the index server limits the number of API calls, making registering even 73 profiles rather slow (takes 6 minutes at 5 seconds each), so a larger dataset would be even more cumbersome to index. Not to mention that the sausage machine currently does one dataset at a time, which means it happens serially.

The limiting seems to kick in a bit unpredictably too, so sometimes not all profiles get registered.

I'm discussing these points on the murmurations forum and in GitHub tickets, and hopefully the problems will be resolved (I think we need a bulk submission API call), but in the mean time I'm holding off merging the code so that the sausage machine registers every time it runs.