Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DEV] unittests: compare map files with osmium in Windows, park static files, have static geofabrik .json #186

Merged
merged 4 commits into from
Feb 28, 2023

Conversation

treee111
Copy link
Owner

@treee111 treee111 commented Feb 27, 2023

This PR…

  • a long awaitet change parking static files
  • might be an enabler for using osmium-tool in Windows for file processing as well and maybe not needing osmium-tool to be installed via brew in macOS ?!

Considerations and implementations

compare map files with osmium in Windows

comparing on Windows was handled solely with filecmp.cmp:

# some file extensions can not be comapared using osmium
if given_file.split('.')[-1] in no_osmosis_file_extensions or \
platform.system() == "Windows":
self.assertTrue(filecmp.cmp(given_file, calculated_file,
shallow=False), f'not equal: {calculated_file}. Using filecmp.cmp.')

After implementing geofabrik-only processing, the boundaries of the bounding boxes are calculated instead of taken from the static tile .json files in the repo.
The calculation via num2deg in geofabrik.py is not rounded and thus this is the result:
item_1 = {'x': 138, 'y': 100, 'left': 14.0625, 'top': 36.59788913307021,
'right': 15.46875, 'bottom': 35.4606699514953, 'countries': ['italy', 'malta'],
'urls': ['https://download.geofabrik.de/europe/italy-latest.osm.pbf',
'https://download.geofabrik.de/europe/malta-latest.osm.pbf']}

using static .json files, this would be the values for top: 36.597889 and bottom 35.46067. Because of this difference, the split-malta.osm.pbf and split-malta-names.osm.pbf were not equal in Windows because of the comparison using filecmp.cmp. I'll keep it not rounded and compare via osmium as doing in macOS some time already

have static geofabrik .json

equal to the other prepared files before running test_generated_files.py, geofabrik.json is also handled now.

park static files

Unittests run with older map, land polygons and geofabrik files. After running test_generated_files.py, map files and land polygons need to be downloaded again if having max. days e.g. 100 or 24. With this PR, the actual productive files are parked, then the unittests prepared, run and afterwards the parked files are copied back into productive location ;-)

How to test

  1. run unittests. Create updated gdal-dev environment first.

Pull Request Checklist

the same way as on macOS, install osmium-tool via conda therefore
unittests errored because split-malta.osm.pbf and split-malta-names.osm.pbf were not equal (most likely because of having more than 6 decimals after num2deg in geofabrik.py)
@treee111 treee111 merged commit f400f89 into release-v4.0.0 Feb 28, 2023
@treee111 treee111 deleted the unittests-osmium-tool-win branch February 28, 2023 14:07
treee111 added a commit that referenced this pull request May 5, 2023
…c files, have static geofabrik .json (#186)

* compare unittest files with osmium-tool on Windows

the same way as on macOS, install osmium-tool via conda therefore
unittests errored because split-malta.osm.pbf and split-malta-names.osm.pbf were not equal (most likely because of having more than 6 decimals after num2deg in geofabrik.py)

* park productive files during unittest run and restore after

* handle geofabrik .json the same way as other static input files

* refactor and add excpeption handling
alfh pushed a commit to alfh/wahooMapsCreator that referenced this pull request Nov 4, 2023
…c files, have static geofabrik .json (treee111#186)

* compare unittest files with osmium-tool on Windows

the same way as on macOS, install osmium-tool via conda therefore
unittests errored because split-malta.osm.pbf and split-malta-names.osm.pbf were not equal (most likely because of having more than 6 decimals after num2deg in geofabrik.py)

* park productive files during unittest run and restore after

* handle geofabrik .json the same way as other static input files

* refactor and add excpeption handling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant