Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updates to UN SDG scripts #871

Merged
merged 33 commits into from
Oct 4, 2023
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
6c92e19
add UNGeoRegions to SDG scripts and update submodule
n-h-diaz Jul 28, 2023
db1aa0d
fix test
n-h-diaz Jul 28, 2023
21e9aa2
update submodule
n-h-diaz Aug 15, 2023
e597022
Merge branch 'master' into tea
n-h-diaz Aug 15, 2023
9a34a81
ADD NEW SCRIPTS
n-h-diaz Sep 14, 2023
2ff9752
Merge branch 'master' of https://github.com/datacommonsorg/data into tea
n-h-diaz Sep 14, 2023
5c64b9f
Merge branch 'tea' of https://github.com/n-h-diaz/data into tea
n-h-diaz Sep 14, 2023
bbf3b8a
delete some old files
n-h-diaz Sep 14, 2023
c1fa05b
add some files to lfs
n-h-diaz Sep 14, 2023
ac32a40
some updates
n-h-diaz Sep 14, 2023
7e284cc
add footnotes to main script
n-h-diaz Sep 25, 2023
f238784
update geography
n-h-diaz Sep 27, 2023
c571f42
update process
n-h-diaz Sep 28, 2023
bba3686
update util
n-h-diaz Sep 28, 2023
cefb35a
readme
n-h-diaz Sep 28, 2023
9941313
Merge branch 'master' into tea
n-h-diaz Sep 28, 2023
ca3e953
tests
n-h-diaz Sep 29, 2023
11592ce
Merge branch 'tea' of https://github.com/n-h-diaz/data into tea
n-h-diaz Sep 29, 2023
d64807a
tests
n-h-diaz Sep 29, 2023
a69beeb
test test
n-h-diaz Sep 29, 2023
9718e21
test test
n-h-diaz Sep 29, 2023
d5dddf9
test test
n-h-diaz Sep 29, 2023
4481b2d
tests
n-h-diaz Sep 29, 2023
98c0cf1
more tests
n-h-diaz Sep 29, 2023
b7f6975
tests
n-h-diaz Sep 29, 2023
345fe99
even more tests
n-h-diaz Sep 29, 2023
bb7b0fe
tests
n-h-diaz Sep 29, 2023
06aa2fa
tests
n-h-diaz Sep 29, 2023
c8d1479
test
n-h-diaz Oct 2, 2023
1f84e6a
clean
n-h-diaz Oct 2, 2023
f74d8a0
update submodules
n-h-diaz Oct 2, 2023
485db38
fix bug
n-h-diaz Oct 2, 2023
93e00a8
comments
n-h-diaz Oct 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
[submodule "scripts/un/sdg/sdg-dataset"]
path = scripts/un/sdg/sdg-dataset
url = https://code.officialstatistics.org/undata2/data-commons/sdg-dataset.git
[submodule "scripts/un/sdg/sssom-mappings"]
path = scripts/un/sdg/sssom-mappings
url = https://code.officialstatistics.org/undata2/sssom-mappings.git
1 change: 1 addition & 0 deletions scripts/un/sdg/.gitattributes
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
csv/* filter=lfs diff=lfs merge=lfs -text
schema/* filter=lfs diff=lfs merge=lfs -text
dc_generated/* filter=lfs diff=lfs merge=lfs -text
geography/* filter=lfs diff=lfs merge=lfs -text
18 changes: 13 additions & 5 deletions scripts/un/sdg/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,19 @@
# UN Stats Sustainable Development Goals

This import includes country, city, and select region-level data from the [UN SDG Global Database](https://unstats.un.org/sdgs/dataportal). Data is read from the submodule `sdg-dataset` which is managed by UN Stats.
This import includes data from the [UN SDG Global Database](https://unstats.un.org/sdgs/dataportal). Data is read from the submodule `sdg-dataset` which is managed by UN Stats. Geography mappings are read from the submodule `sssom-mappings` which is also managed by UN Stats.


To generate city dcids:
To generate place mappings:
```
python3 cities.py <DATACOMMONS_API_KEY>
python3 geography.py
```
(Note: many of these cities will require manual curation, so this script likely should not be rerun.)
Produces:
* geography/ folder:
* un_places.mcf (place mcf)
* un_containment.mcf (place containment triples)
* place_mappings.csv (map of SDG code -> dcid)

Note that the `place_mappings.csv` is required before running the `process.py` script.

To process data and generate artifacts:
```
Expand All @@ -23,7 +29,9 @@ Produces:
* unit.mcf
* csv/ folder:
* [CODE].csv
(Note that the `schema/` folder is not included in the repository but can be regenerated by running the script.)
(Note that these folders are not included in the repository but can be regenerated by running the script.)

When refreshing the data, the `geography`, `schema`, and `csv` folders might all get updated and will need to be resubmitted to g3. The corresponding TMCF file is `sdg.tmcf`.

To run unit tests:
n-h-diaz marked this conversation as resolved.
Show resolved Hide resolved
```
Expand Down
Loading
Loading