Update Co-ops UK Map data with latest interim drop

[ColmDC]

The data in https://dev.data.solidarityeconomy.coop/coops-uk/ is from the last public open data drop by Co-ops UK. They have postponed doing public drops for a while, as they decided to review their whole approach to it. However, they have provided us with an iterim drop which can be found here.

Update our published data to use this new iteration.

[ColmDC]

Set date in about box to reflect date of drop i.e 2023-05-15.

Also add this logo https://upload.wikimedia.org/wikipedia/en/thumb/2/21/Co-operatives_UK_logo-2015.png/225px-Co-operatives_UK_logo-2015.png

[wu-lee]

The headings in the CSV has changed.

Attempting to match them up I can get all of the important ones bar one: "SIC Sector", which has no equivalent in the new data, either looking at the name or looking at the content. The csvstat command shows the number of unique values and the most common ones - there's no match I can see. SIC Text looked like the closest match based on the name, but it has 349 unique values, whereas SIC Section only has 21. SIC_Level_1 only has 2. So both are nowhere close.

But I think I can infer the SIC Section from the old data, by mapping SIC Codes to SIC Section. Will see if this works shortly.

Matches below.

new field id	new schema	old field id	old schema	mapping to new field id
1	Co-ops UK Identifier	1	CUK Organisation ID	1
2	Organisation ID	2	Registered Number	10
3	Organisation Name	3	Registrar	26
4	Type of Organisation	4	Registered Name	3
5	Type of Organisation Subcategory	5	Trading Name	3
6	SIC Code	6	Legal Form	16
7	SIC Text	7	Registered Street	11
8	Registered Status	8	Registered City	12
9	SIC_Level_1	9	Registered State/Province	13
10	Registered Number	10	Registered Postcode	14
11	Registered Street	11	UK Nation	22
12	Registered City	12	SIC Code	6
13	Registered State/Province	13	SIC section	7?
14	Registered Postcode	14	SIC code - level 2
15	Registered Country	15	SIC code - level 2 description
16	Legal Form	16	SIC code - level 3
17	Legal Form Subcategory 1	17	SIC code - level 3 description
18	Legal Form Subcategory 2	18	SIC code - level 4
19	Incorporation Date	19	SIC code - level 4 description
20	Dissolved Date	20	SIC code - level 5
21	Co-ops UK Industry Sector	21	SIC code - level 5 description
22	UK Nation	22	Sector - Simplified, High Level	21
23	Ownership Classifications	23	Ownership Classification	23
24	Trading Name	24	Registered Status	8
25	Website	25	Incorporation Date	19
26	Registrar	26	Dissolved Date	20
27	Retail Society	27	Website	25
28	Registered Admin County Code	28	Registered Admin County Code	28
29	Registered Admin County Name	29	Registered Admin County Name	29
30	Registered Admin District Code	30	Registered Admin District Code	30
31	Registered Admin District Name	31	Registered Admin District Name	31
32	Registered Admin Ward Code	32	Registered Admin Ward Code	32
33	Registered Admin Ward Name	33	Registered Admin Ward Name	33
34	Registered Constituency Code	34	Registered Constituency Code	34
35	Registered Constituency Name	35	Registered Constituency Name	35
36	Registered LSOA Name	36	Registered LSOA Name	36
37	Registered MSOA Name	37	Registered MSOA Name	37
38	Registered Parish Code	38	Registered Parish Code	38
39	Registered Parish Name	39	Registered Parish Name	39
40	Registered Region

[wu-lee]

I've had to add some SIC code mappings. Most of these fall in blocks of mappings to the same SIC section label, which make it reasonably straightforward to guess. The remaining few I've mapped to null, which is what some codes are mapped to in the previous dataset. Otherwise, this conversion seems to work.

(NI postcodes notwithstanding)

Need to see how much it's changed on the map next.

[wu-lee]

Ok, changes but they don't look insane. Hard to tell if they're completely correct, but here are screenshots:

Last dataset (Q2 2022):

This dataset (Q2 2023)

[wu-lee]

Have deployed a the data but kept a local copy of the old standard.csv.

Have updated the date on the the "about" panel of the co-ops UK map to Q2-2023.

Note to self - ideally we wouldn't need to do that, it should be a property of the data shown in the map, so if the data changes we don't need to choose between out of data information on the map, and a rebuild and redeploy of exactly the same code but with correct information. Need to create an issue for this.

[ColmDC]

Small changes from the previous version are not a problem as it was not a version publically promoted. I'll do some manual testing.

[ColmDC]

Note to self - ideally we wouldn't need to do that, it should be a property of the data shown in the map, so if the data changes we don't need to choose between out of data information on the map, and a rebuild and redeploy of exactly the same code but with correct information. Need to create an issue for this.

They are going to change how they approach publishing this data after the summer, and we should have an opportunity to influence those changes.

[wu-lee]

Ok. Note, this will be a problem anywhere we want to say "this data is version X" in the "about" dialog, because the about text is versioned with the map site and not the data (or mykomap).

Issue here: DigitalCommons/mykomap#209

[wu-lee]

Note, the map has also been bumped to mykomap 3.0.6

[wu-lee]

Note also, #17 coming up next- this will change the locations too, quite definitely.

Do you want to check the map before of after that? @ColmDC

[ColmDC]

Create a ticket to come back at some point and compare the location differences.

[ColmDC]

It seems unlikely that this is caused by new data, but the search panel doesn't open in https://dev.coopsuk.solidarityeconomy.coop/

[ColmDC]

[ColmDC]

Hard to QA this any further with this bug.

[ColmDC]

Created own issue for this bug #21