-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Modernize, standardize, and otherwise improve parser code #215
base: master
Are you sure you want to change the base?
Commits on Oct 15, 2016
-
Updated download_fms_fixies, added new deps
- download_fms_fixies is now Py2/3 compatible and updated to 2016 python code rather than 2013; added argparse for specifying start and end dates on the command line; no more hand-rolled datetime stuff, and no more hard-coded holidays (thx to arrow and holidays) - added and pinned arrow and holidays deps to requirements file
Burton DeWilde committedOct 15, 2016 Configuration menu - View commit details
-
Copy full SHA for c278c35 - Browse repository at this point
Copy the full SHA c278c35View commit details
Commits on Jul 3, 2017
-
General cleanup and slight refactor of download fms fixie script
Burton DeWilde committedJul 3, 2017 Configuration menu - View commit details
-
Copy full SHA for c554ddb - Browse repository at this point
Copy the full SHA c554ddbView commit details -
Bugfix in fixie dir-switching; randomized time sleeps
Burton DeWilde committedJul 3, 2017 Configuration menu - View commit details
-
Copy full SHA for 34e8706 - Browse repository at this point
Copy the full SHA 34e8706View commit details -
Linted the FUCK out of the parser code
(hope to god i didn’t introduce any errors)
Burton DeWilde committedJul 3, 2017 Configuration menu - View commit details
-
Copy full SHA for 0be79ae - Browse repository at this point
Copy the full SHA 0be79aeView commit details -
Added logging, CLI to parse_fms_fixies.py
Also loaded resources properly, so file handles are closed automatically.
Burton DeWilde committedJul 3, 2017 Configuration menu - View commit details
-
Copy full SHA for 613d89e - Browse repository at this point
Copy the full SHA 613d89eView commit details
Commits on Jul 4, 2017
-
Added utils, constants modules for shared funcs, vals
- Expanded and standardized CLI args for download and parse fms fixies modules - CLI defaults now only download or parse fms fixies that haven’t already been downloaded or parsed - Made parser into python package, to enable relative imports within modules - Bumped arrow version requirement
Burton DeWilde committedJul 4, 2017 Configuration menu - View commit details
-
Copy full SHA for cdaa1a9 - Browse repository at this point
Copy the full SHA cdaa1a9View commit details
Commits on Jul 7, 2017
-
Added parse_all_fixies(), which includes result I/O
- continued putting the pieces together in download_and_parse_fms_fixies CLI - better logging and empty result handling in downloader and parser - tweaked CLI arg names for data dirs and shifted default end dates back one day - utils.get_daily_csvs_by_date() keys now arrow objects, just like utils.get_fixies_by_date()
Burton DeWilde committedJul 7, 2017 Configuration menu - View commit details
-
Copy full SHA for fd1b09f - Browse repository at this point
Copy the full SHA fd1b09fView commit details
Commits on Jul 8, 2017
-
Added module to aggregate parsed fixies into csvs and sql db
- added table keys and db table names to constants - default startdate now 8 days prior to today rather than earliest possible date; seems like default shouldn’t be “run over every single thing available” - restructured daily_csvs_by_date dict; now nested dict by table name rather than tuple of filenames - bumped pandas version requirement
Burton DeWilde committedJul 8, 2017 Configuration menu - View commit details
-
Copy full SHA for 6615c46 - Browse repository at this point
Copy the full SHA 6615c46View commit details -
Clean up unused imports and funcs
- add logger to utils module, to prevent bug - add csv text stuff to constants, temporarily - clean up the readme a bit
Burton DeWilde committedJul 8, 2017 Configuration menu - View commit details
-
Copy full SHA for c7ebd8c - Browse repository at this point
Copy the full SHA c7ebd8cView commit details -
Remove download_and_parse_fms_fixies script
- Improve logging messages / levels throughout parser code - I changed my mind: make default start date be the earliest available date, but in practice, defaults will only get the new stuff - Update run.sh and reset_data.sh scripts to use new parser CLI -
Burton DeWilde committedJul 8, 2017 Configuration menu - View commit details
-
Copy full SHA for 8ab1ba3 - Browse repository at this point
Copy the full SHA 8ab1ba3View commit details -
Burton DeWilde committed
Jul 8, 2017 Configuration menu - View commit details
-
Copy full SHA for cca51c7 - Browse repository at this point
Copy the full SHA cca51c7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0ad0e13 - Browse repository at this point
Copy the full SHA 0ad0e13View commit details
Commits on Sep 21, 2017
-
fixed 'Attempted relative import in non-package'
Cezary Podkul committedSep 21, 2017 Configuration menu - View commit details
-
Copy full SHA for b568730 - Browse repository at this point
Copy the full SHA b568730View commit details