-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rewrite rapidml
from scratch
#158
Comments
So it's not really
so |
wow ;) |
I now know a lot more than when this package started, and can see the
rapidml
header could fairly easily be re-written from scratch as a custom OSMXML
parser that would do the storage during initial reading. Interestingly, and excitingly for @mdsumner'ssilicate
work, this direct store-on-read procedure is only really possible with and because ofsilicate
. The entire OSM structure is in essence fullysilicate
-compliant, and can be directly stored line-for-line as read.This should ultimately enable the entire package to be re-written to simply dump directly to
SC
format, and then usesilicate
to convert outputs to other formats. (Plus some additional fiddling to insert the "hidden" but necessary row names containing OSM IDs.) MIke, I've done some preliminary comparisons of direct SC-storage, and for the test data set (a chunk of about 1/3 of Melbourne streets), the current 15-16s reduces to about 0.4s. So we're looking at least a tenfold boost in speed, which is well worth pursuing.Related to general
osmdata_sc
issue #148.The text was updated successfully, but these errors were encountered: