Adding sink() to init_etl()

[pbchase]

We want to dump all of the R log to test files to ensure a complete run/crash log. We will do these things:

• Read an environment variable named SINK_DIR from env file that sets the root folder of the logged files.
• Add SINK_DIR in the example env file setting the value to ~/rcc_sink_log
• Add sink() to init_etl() so that existing calls to init_etl will get the new functionality without modification
• Test for the environment variable. If it is missing, use ~/rcc_sink_log for the sink logging hierarchy. This will place folders named 2023, 2024, etc. in ~/rcc_sink_log.
• Write files into a hierarchical structure with a year folder at the top level.
• Name the files YYYYMMDD-HHMMSS-script_name.log
• The default file path is $SINK_DIR/$YEAR/filename. An example of this with full defaults for the script foo_hotness run on 2023-03-14 would be ~/rcc_sink_log/2023/20230314-123456-foo_hotness.log.
• Revise the cron example to show the mounting of a volume to persist the log files

Note, the discussion about this feature has underscored the value of using docker-compose and yaml files to run these containers. That will be addressed in another issue.

[ljwoodley]

Would this be an option? This is an example with render_report.R run locally. Output attached.

docker run --rm --env-file .env \
  -v $(pwd):/home/rocker/redcapcustodian \
  redcapcustodian /bin/bash -c "Rscript report/render_report.R \
  report/sample_report.Rmd > /home/rocker/redcapcustodian/$(date +%Y%m%d-%H%M%S)-sample_report.log 2>&1"

20240311-192047-sample_report.log

[ChemiKyle]

Separate goal of switching out docker run ... crons to docker compose:

docker-compose.yaml

version: '3.8'

services:
  redcapcustodian:
    container_name: redcapcustodian
    env_file:
      - /path/to/default.env
    environment:
      - FOO=${FOO}
    image: redcapcustodian
    restart: "no"
    volumes:
      - /path/to/host/dir:/mnt/homedir

new cron format:

* * * * * root docker compose -f /path/to/redcapcustodian/docker-compose.yaml --env-file /path/to/override.env run --rm redcapcustodian Rscript report/sample_report.Rmd 

Note that override.env does not seem to actually end up replacing variables sourced from Sys.getenv if they aren't also specified in the "environment" section; however, using -e KEY=var in the docker compose string after run does work (--env-file cannot be placed after run). This is not the case for docker compose up.

[ljwoodley]

Is this proceeding with the docker compose solution?

[pbchase]

On 2024-04-30 @ljwoodley, @pbc, and @saipavan10-git discussed using https://airflow.apache.org/docs/apache-airflow/stable/index.html to manage our redcapcustodian tasks. @ljwoodley will do a proof of concept to run one workflow under airflow and produce a log file.