Feature branch for sdg-data

[Norric1Admin]

I’m trying to test some maps on a feature branch. Could you help me to set up a feature branch for the sdg-data repo please? We currently have one for sdg-indicators. This will allow us to test maps, multilingual features, etc.

[Norric1Admin]

Response from Doug

We can discuss this on the call. But to make a new branch, from the main GitHub page for sdg-indcators click on the branch button:

Then add a new branch name. Make sure it says that it will branch from develop.

Make any changes you want in this branch. Travis will build it and if it’s successful it will deploy to the test server. It can sometimes take a while, I need to play with file compression to speed up deployment.

By the way, my email system hates the IP addresses so use http://sdg.mango-solutions.com/

Oh by the way, it “slugifys” the branch names. So you might find they get renamed a bit. Looks like the underscores are being changed to hyphens. Not strictly necessary but the idea with slugifying is to make a definitely safe name.

[dougmet]

The comment above misunderstood that you want to test data deployments. This has now been done with ONSdigital/sdg-data#582

Care is needed. You're pointing a test branch here with a test branch at sdg-data. If you don't remove that test data link it could end up on the staging and eventually live.

[Norric1Admin]

Thanks for doing this so quickly. Just to confirm - as long as the feature branch (in this case 'MAPS') is not merged into Staging accidentally, there is no risk of the changes appearing on the live site? In which case, deleting the test branch should prevent this. Let me know if I have misunderstood.

[Norric1Admin]

Doug is on the case - #2923

[Norric1Admin]

@dougmet I've run into an error - the graphs are not displaying because (although the .json files are accessible from data link), the access permissions are not allowing the deployment to read these files.

(index):1 Failed to load http://sdg.mango-solutions.com/data/maps/comb/4-2-1.json: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://***#!!☠💀*' is therefore not allowed access.

I've searched, but can't find this 'index' file to add the permissions to the first line. If this is what you recommend we do, I think adding the following to line 1 of the index file could work:

Access-Control-Allow-Origin: http://sdg.mango-solutions.com
OR...
Access-Control-Allow-Origin: *

I wonder if this is a file that is created during deployment, in which case the above line would have to be added to the deploy.sh somewhere probably. I'm guessing though. What do you recommend?

[dougmet]

I have this odd feeling we've fixed something like this before. It's to do with calling external APIs. Could be done programmatically from the databaseurl field. Wonder why this has never been an issue before?

[Norric1Admin]

Perhaps because this feature build relies on calling .json scripts stored in sdg-data, but the build itself is located under sdg-indicators. I wonder if this is similar to the Prose.io issue, where http links are blocked? Relevant issue: axios/axios#853

[dougmet]

it might be something I need to configure in apache. I'll investigate.

[dougmet]

which branch isn't working? It's looking OK for me at the moment

[Norric1Admin]

I thought it was the test_data_deploy branch, but actually all of them fail to show the graph. It seems to be reacting in the same way as if JavaScript were disabled, as I do see data tables with accurate data (except for the indicators where a map tab is present).

E.g. 4.2.1 gives the 'Access Control...' error, whereas 3.2.1 shows a headline data table.

[Norric1Admin]

Options for an additional map on the NRP...

onsdigital.github.io/sdg-indicators/2-3-1/ (possible but Units are a problem)
onsdigital.github.io/sdg-indicators/3-a-1/ (best match overall) !
onsdigital.github.io/sdg-indicators/4-2-1/ (England-only, but a good backup)
onsdigital.github.io/sdg-indicators/9-1-1/ (possible but silly data)

[dougmet]

Can you tell me if you're still getting the Access Control error? I made a change on the server.

[Norric1Admin]

Only for a certain indicators (4.2.1, there may be others). The majority of them load without their charts and show the 'Active content removed' error (not in the console, but the page border).

In the meantime, me and @LucyGwilliamAdmin are going to attempt this on a virtual machine.

[Norric1Admin]

Progress! Lucy suggested using the Mango server URL, and this resolved the active content errors.

I made a data edit to 4.2.1 (East Midlands, 2017) and this is working as hoped. Maps are proving slightly more problematic.

[LucyGwilliamAdmin]

Ok, so my feeling is that the maps aren't the problem but rather metadata as a whole. I've just changed the 3.a.1 metadata so that data_show_map is true and also changed the graph title from "Percentage of population aged 18 and over who smoke cigarettes" to "Percentage of population aged 18 and over who smoke cigarettes TEST" and neither the map is showing or the new graph title.

Chris pointed the test-data-deploy branch at the MAPS branch of sdg-data and it is getting data from there fine, so not sure why it's not getting metadata in the same way.

Is there something in the code doug as I know for us to be able to see metadata changes on the site the sdg-indicator repo needs to be pushed but for data changes to be be seen on the site only the sdg-data repo needs to be pushed.

Any ideas??

[Norric1Admin]

I can confirm that this is the case - metadata is not updating.
Note that the test branch is now reading from 'featurebranch' on sdg-data.

[dougmet]

Hi @Norric1Admin @LucyGwilliamAdmin . Feel a bit of an idiot for this one. I had not updated the data url in enough places. In the config it needs to be update as follows:

remotedatabaseurl: "http://sdg.mango-solutions.com/data/featurebranch"
# permalink: /news/:year/:month/:day/:title.html

jekyll_get_data:
  - data: meta
    json: '"http://sdg.mango-solutions.com/data/featurebranch/meta/all.json'
    # cache: true
    # directory: 'remotedata'
  - data: headlines
    json: '"http://sdg.mango-solutions.com/data/featurebranch/headline/all.json'
    # cache: true
    # directory: 'remotedata'
  - data: schema
    json: '"http://sdg.mango-solutions.com/data/featurebranch/meta/schema.json'

[dougmet]

OK please try again. I've fixed the CORS (Access-Control-Allow-Origin) issue on the server and updated the config on the test-data-deploy branch. Apologies for the oversight above.

[LucyGwilliamAdmin]

Great, the metadata changes are now visible on the test-data-deploy branch

[dougmet]

That's great. So in summary I had to:

1. Specify data location in four places (not just remotedatabaseurl)
2. Set Access-Control-Allow-Origin: "*" on test server

Is it OK to close?

[Norric1Admin]

Great - thanks Doug, and no problem. This is working now.
How/where did you set CORS: '*' (for future reference)?

[dougmet]

It was in the apache setup. I did find this page useful because my first attempt failed: https://awesometoast.com/cors/ I made my changes in the /etc/apache2/sites-available/000-default.conf file. But this is specific to both Apache and Debian so isn't a fix for all situations.

[dougmet]

Oh, and enable mod_headers

[Norric1Admin]

Great - one more question. Does this need to be done for each new deployment branch, and if so is there a location that we can go to to do it (or is it only Mango that has access to these server files)?

[dougmet]

no it's a one time config of the server (which only I have access to) but it's all done and you won't need to touch it now.