Mutual Aid Spreadsheet no longer being uploaded to map

[ColmMassey]

It looks like the last few entries to the spreadsheet have not been added to the data and this the map. I spotted and fixed one error in the data where a website url was appended to the project title, but I don't think that was it.

[ColmMassey]

Also, what are the rules that need to be followed when adding facebook page links. Might that be the problem?

[wu-lee]

Running the conversion locally, seems to work fine, although I see what you mean about the last few fields.

But I think what's stopping them being uploaded is that they haven't had a true value added to the approved column...

[wu-lee]

Checking this earlier:

• Opening Mutual-Aid-Mini.ods in the Mutual-Aid shared folder in Nextcloud, using the Only Office web UI
• I added an 'x' in the description of the last item, named 'Unity Center Glasgow'
• I ran the sausage machine's download locally in my check out of open-data
• The file is downloaded, with a new timestamp, but the change I made does not appear in the .ODS file downloaded (and so is not in the exported CSV)
• The file on Nextcloud's file listing does not have an updated timestamp (it says it was last modified 3 months ago)

The downloaded file also does not have approved values for many of the entries, so I suspect it might be an old copy. Maybe there is something wrong with the interaction between Nextcloud and Only Office? The changes in the latter persist, but don't seem to show in the former.

Downloading the spreadsheet file directly from Nextcloud and opening it in Libre Office on my desktop gets the unedited version.

The URL used by the sausage machine is: https://nextcloud.solidarityeconomy.coop/index.php/s/YDctt5nbP4e7Rx7/download

The URL I see in the footer of the browser window when hovering over the filename in the Nextcloud directory view is: https://nextcloud.solidarityeconomy.coop/index.php/s/fkgt6EbEFKiDe28/download?path=/MutualAid&openfile=61064

[wu-lee]

Our problem sounds a bit like this one:

https://help.nextcloud.com/t/onlyoffice-documents-not-saved-in-nextcloud/76192/20

Or these:

ONLYOFFICE/onlyoffice-nextcloud#369
nextcloud/documentserver_community#100

Caching again. I think it needs a look by the Nextcloud hosting service. There are suggestions it can be worked around, but seems ultimately to be a problem with Nextcloud and/or the OnlyOffice config.

[ColmMassey]

I am at my wits end with our Nextcloud hosting for other reasons. I really don't want to invest any more time into support for Nextcloud spreadsheets. What is the easiest of our other supported options for you to transistion to?

[wu-lee]

I would say, either LimeSurvey, Google Sheets, or possibly Kobo Toolbox (with a bit less certainty).

[wu-lee]

(They all have their niggles)

[ColmMassey]

Does the document work when you download it manually?

[ColmMassey]

I am thinking of just dropping back to manual support for the moment.

[wu-lee]

Does the document work when you download it manually?

Yes and no.

• No, if you download it from NextCloud, you get the 3 month old version. That's either by the web UI or using that shortcut URL the sausage manchine uses
• But yes, it does seem possible to download it directly from Only Office's UI - this gets the one OO has been editing

The danger seems to be that the most up-to-date copy in OO may be volatile - just a cached copy. So any changes there may not be retained if the cache gets flushed, perhaps by a server restart.

Even if you don't expect or care for this to be sorted, I think for the sake of other users of the Nextcloud service you use, you should report it, so they know they've got a potentialy dangerous data-losing hole in their document management. I expect they'll be keen to sort it!

I'll feed the downloaded version into the SM locally and deploy it manually. To stop the automated one from clobbering that with the old one at any point. I should probably take mutual aid off the list of datasets to build.

[wu-lee]

BTW, I heard from Oli SB that OpenCoop are launching a NextCloud service. Assuming you've not written NC off entirely.

https://openweb.systems/

[ColmMassey]

https://openweb.systems/

Is it up and running already. If so, I guess we could give is a go. Set a fresh account up for DC. Transfer some SEA stuff across and see if it is any more reliable/usable that GreenNet's service.

[ColmMassey]

For now, can update the MA map manually now. I'll get Jo and Channy to ping you after they add new content.

[wu-lee]

An update:

• I've disabled automatic data generation for mutual-aid in the sausage machine (AKA open-data)
• I've used the exported OnlyOffice doc to rebuild the mutual aid data (dev and prod)
• I've updated the version of sea-map used and redeployed the mutual-aid maps sites (dev and prod)

In order to do that there were there were various problems to solve.

• the latest sea-map didn't work with mutual aid because:
  • it had hard-wired dependencies on the activities and organisational structure vocabs which mutual-aid data doesn't have - so I had to remove these dependencies
  • in particular, the default popup code also had some of these, so I had to strip it down to something which didn't and also made fairly minimal assumptions about the fields present: name, description and location of course, but it looks for address fields and tries to combine them sensibly, plus email, facebook and twitter links
  • sea-map expects twitter links to have had the domain removed in the data, so I fixed the mutual-aid conversion to do this
• the new dialogueSize config setting for the ICA map has no default, so I added one (using the ICA defaults)
• this exposed a few bugs in sea-map's config schema code, as well as that the documentation generation was broken, so I fixed and regenerated that
• the mutual-aid site hadn't been deployed on prod-0 before (it was still covid-mutual-aid) so I had to add the necessary production configuration to do that
• to deploy this I made a new minor version release of sea-map (v1.2.1)
• in order to check that I wouldn't be breaking other maps by altering the default popup etc. I added a column to shipshape showing the version of sea-map in use

A couple of things I've not fixed so far:

• I noticed that there is no redirection for lod.coop/covid-mutual-aid currently (just dev.lod.coop/covid-mutual-aid), which requires some thought to fix as prod-0 is on an older branch of it
• I also noticed that the "about" page is now generated programmatically inside sea-map using internationalised labels, so that the about.html is redundant, but also, this code hardwires a link to the ICA source data, so something needs to be done about that (currently this info isn't easily accessible in the browser-side)
• the mutual aid map has nothing in its directory tab

[ColmMassey]

* it had hard-wired dependencies on the activities and organisational structure vocabs which mutual-aid data doesn't have - so I had to remove these dependencies

Is there any reason why we should have any bare the bare minimum of fields as hard requirements?

[ColmMassey]

* the mutual aid map has nothing in its directory tab

They weren't interested in supporting a Directory. Can this option be hidden in the config? Was it added when this work was done?

[ColmMassey]

• I noticed that there is no redirection for lod.coop/covid-mutual-aid currently (just dev.lod.coop/covid-mutual-aid), which requires some thought to fix as prod-0 is on an older branch of it

• I also noticed that the "about" page is now generated programmatically inside sea-map using internationalised labels, so that the about.html is redundant, but also, this code hardwires a link to the ICA source data, so something needs to be done about that (currently this info isn't easily accessible in the browser-side)

Can you create new tickers for these issues?

[wu-lee]

Is there any reason why we should have any bare the bare minimum of fields as hard requirements?

Well, the bare minimum would be a name, but just a name would make for a very empty looking pop-up. I was concerned that if I ripped everything out then I'd then have to go and create custom pop-ups for nearly every new map site. My compromise was to check for certain fields which seemed to be provided commonly in the standard.csv schema. They aren't hard requirements, but get shown if present. The vocab stuff I kept out because that adds a lot of extra machinery and require configuration.

I suppose alternatively we could default to a tiny pop-up or no pop-up?

[wu-lee]

Can you create new tickers for these issues?

Done. see mentions above

[wu-lee]

They weren't interested in supporting a Directory. Can this option be hidden in the config?

It's using the latest latest sea-map, so yes, that's supported. Well, it is now I just fixed a bug. Redeloyed a new version of sea-map v1.1.2 and dev and prod versions of mutual-aid-project.

[wu-lee]

Redeployed the map with the latest data added by Jo.

Slight niggle with the downloaded spreadsheet not working as downloaded from OnlyOffice - it needs to be opened in LibreOffice and saved again, otherwise the 'approved' column's TRUE values aren't converted, for some reason.

A heads-up: the version retained and editable by OnlyOffice is not being saved in NextCloud, which may mean it will disappear if the OnlyOffice cache is cleared, perhaps by a server restart. I have a copy of the latest version right now, but i still think that the NextCloud service provider needs to be notified they have a serious problem with their OnlyOffice integration which could lose data.

One other point: I can't add the project description provided by Jo until I unhardwire the "about" tab contents in a way which preserves the language internationalisation. See #125

[ColmMassey]

The issue has been addressed, in that there is a workable system, but the Nextcloud bugs identified are still there. Can you submit a bug report to Nextcloud and close then close the issue here?

[wu-lee]

Update: Green Net replied:

Thank you for reporting the problem. Another NextCloud user of ours had reported the problem, but had not supplied enough detail for us to reproduce the problem.

We spent a few hours this evening investigating the problem, it seems it was an issue with the OnlyOffice file format conversion process that is used to edit and save non-DOCX (or XLSX) files, like ODT or ODS files. It was quite a strange one, but we've managed to fix the problem. We carefully preserved the OnlyOffice file cache and so the latest version of those files is still stored there. So, for files where this problem occurred, you will now just need to open the file one more time in OnlyOffice and make any change there. The file will then be correctly saved within NextCloud.

Let us know if you have any questions or there is anything else we can do.

This seems to be working, so I've restored the mutual-aid dataset to the list processed by the sausage factory on dev-1. I also needed to amend the converter to accomodate OnlyOffice, which encodes a "TRUE" the "approved" columns as a "1", unlike LibreOffice, which uses "true". However, changes the data are now appearing on the map again.