Create cron jobs docker-compose service

[cdrini]

Fixed in #4387
Fixed in internetarchive/olsystem#118
Subtask of: #4060

We have a number of cron jobs that are run on ol-home ; in order to run this in our docker-based ol-home0 node, we should place them in a service. Note these cron jobs live in olsystem.

• Transfer ol-home cron jobs to a new cron file.
• Ensure all ol-home cron jobs run on ol-home0
  • Need to get oldump cron job running in Docker #4622 /olsystem/bin/cron/oldump.sh ol-db1 --> Fix dump.py to read str, instead of bytes #4627 + internetarchive/olsystem#118
    • dumping the data table # for a long time... 102 minutes!
    • Fixed in this PR by correcting $SCRIPTS in bin/cron/oldump.sh
    • Got an error on a job on the IA side; it was trying to rsync from the docker container: rsync://e1d610933819/var_1/tmp/dumps/ol_cdump_2021-01-30
    • STATE: Running; waiting to see if it completes
  • $SCRIPTS/store_counts.py * 2 -- fixed in Use empty str instead of None in HTTP headers infogami#134
    • if _is_illegal_header_value(values[i]): TypeError: expected string or bytes-like object
    • Errored: Unable to access infobase?
  • $SCRIPTS/2011/08/expire_accounts.py -- fixed in Use empty str instead of None in HTTP headers infogami#134
    • if _is_illegal_header_value(values[i]): TypeError: expected string or bytes-like object
    • Errored: Unable to access infobase?
      * Needed to change infobase link from ol-home0 -> ol-home in openlibrary.yml
  • openlibrary.solr.process_stats * 2 (loan stats)
    • cat /var/log/openlibrary/loanstats.log
    • /run/secrets/ia_db_pw_file: not found should have been fixed by Edition RDF contains Work URI as rdf:resource, instead of (quoted) rdf:Literal. #115
    • From https://github.com/internetarchive/olsystem/blob/master/etc/openlibrary.yml#L31
    • process_stats.py erroring for some loans #4542 Python 3 decode error for 1 loan?
  • python $SCRIPTS/manage-imports.py
    • /run/secrets/ia_db_pw_file: not found should have been fixed by Edition RDF contains Work URI as rdf:resource, instead of (quoted) rdf:Literal. #115
    • No errors now.
• Ensure all ol-www1 cron jobs run in docker + py3
  • (Moved to Move ol-home / ol-www ipstats cron job + ol-home solr to ol-home0 #4775 and separated from this issue) sitemap rsync should run on ol-home0 Slack thread
    The sitemaps are generated on ol-home0 and copied over to ol-www1 where they are served. We can actually make the rsync command run on ol-home0 and target ol-www1 (and/or ol-www0). This is not blocked by ol-www0 migration. saddr $CLUSTER proto tcp dport rsync ACCEPT; in /etc/ferm/ferm.conf is vital.
  • cron-jobs service stuck in restart loop cron-jobs service stuck in restart loop #4707 fixed in Fix failing cron-jobs in Docker #4717
  • (Moved to Move ol-home / ol-www ipstats cron job + ol-home solr to ol-home0 #4775 and separated from this issue) MOVE ALL CRON JOBS TO OL-HOME0 ipstats.py. Unclear what ideal course of action is for these, but if we could get them running on ol-home0 with other cron jobs: that would be nice!

[cclauss]

Reopening until we can land internetarchive/olsystem#118

[mekarpeles]

@cclauss can we move

MOVE ALL CRON JOBS TO OL-HOME0 ipstats.py. Unclear what ideal course of action is for these, but if we could get them running on ol-home0 with other cron jobs: that would be nice!

to a separate issue (as we'll probably not attack it in the near-ish future, that way we can close this monolith and prioritize the new issue accordingly