Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modernize, standardize, and otherwise improve parser code #215

Open
wants to merge 13 commits into
base: master
Choose a base branch
from

Commits on Oct 15, 2016

  1. Updated download_fms_fixies, added new deps

    - download_fms_fixies is now Py2/3 compatible and updated to 2016
      python code rather than 2013; added argparse for specifying start
      and end dates on the command line; no more hand-rolled datetime
      stuff, and no more hard-coded holidays (thx to arrow and holidays)
    - added and pinned arrow and holidays deps to requirements file
    Burton DeWilde committed Oct 15, 2016
    Configuration menu
    Copy the full SHA
    c278c35 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2017

  1. Configuration menu
    Copy the full SHA
    c554ddb View commit details
    Browse the repository at this point in the history
  2. Bugfix in fixie dir-switching; randomized time sleeps

    Burton DeWilde committed Jul 3, 2017
    Configuration menu
    Copy the full SHA
    34e8706 View commit details
    Browse the repository at this point in the history
  3. Linted the FUCK out of the parser code

    (hope to god i didn’t introduce any errors)
    Burton DeWilde committed Jul 3, 2017
    Configuration menu
    Copy the full SHA
    0be79ae View commit details
    Browse the repository at this point in the history
  4. Added logging, CLI to parse_fms_fixies.py

    Also loaded resources properly, so file handles are closed
    automatically.
    Burton DeWilde committed Jul 3, 2017
    Configuration menu
    Copy the full SHA
    613d89e View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2017

  1. Added utils, constants modules for shared funcs, vals

    - Expanded and standardized CLI args for download and parse fms fixies
    modules
    - CLI defaults now only download or parse fms fixies that haven’t
    already been downloaded or parsed
    - Made parser into python package, to enable relative imports within
    modules
    - Bumped arrow version requirement
    Burton DeWilde committed Jul 4, 2017
    Configuration menu
    Copy the full SHA
    cdaa1a9 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2017

  1. Added parse_all_fixies(), which includes result I/O

    - continued putting the pieces together in
    download_and_parse_fms_fixies CLI
    - better logging and empty result handling in downloader and parser
    - tweaked CLI arg names for data dirs and shifted default end dates
    back one day
    - utils.get_daily_csvs_by_date() keys now arrow objects, just like
    utils.get_fixies_by_date()
    Burton DeWilde committed Jul 7, 2017
    Configuration menu
    Copy the full SHA
    fd1b09f View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2017

  1. Added module to aggregate parsed fixies into csvs and sql db

    - added table keys and db table names to constants
    - default startdate now 8 days prior to today rather than earliest
    possible date; seems like default shouldn’t be “run over every single
    thing available”
    - restructured daily_csvs_by_date dict; now nested dict by table name
    rather than tuple of filenames
    - bumped pandas version requirement
    Burton DeWilde committed Jul 8, 2017
    Configuration menu
    Copy the full SHA
    6615c46 View commit details
    Browse the repository at this point in the history
  2. Clean up unused imports and funcs

    - add logger to utils module, to prevent bug
    - add csv text stuff to constants, temporarily
    - clean up the readme a bit
    Burton DeWilde committed Jul 8, 2017
    Configuration menu
    Copy the full SHA
    c7ebd8c View commit details
    Browse the repository at this point in the history
  3. Remove download_and_parse_fms_fixies script

    - Improve logging messages / levels throughout parser code
    - I changed my mind: make default start date be the earliest available
    date, but in practice, defaults will only get the new stuff
    - Update run.sh and reset_data.sh scripts to use new parser CLI
    -
    Burton DeWilde committed Jul 8, 2017
    Configuration menu
    Copy the full SHA
    8ab1ba3 View commit details
    Browse the repository at this point in the history
  4. Oops (bugfixes)

    Burton DeWilde committed Jul 8, 2017
    Configuration menu
    Copy the full SHA
    cca51c7 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    0ad0e13 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2017

  1. fixed 'Attempted relative import in non-package'

    Cezary Podkul committed Sep 21, 2017
    Configuration menu
    Copy the full SHA
    b568730 View commit details
    Browse the repository at this point in the history