A GitHub action that leverages the Jinja2 and j2cli python libraries for templating files in your repository.
Use dynamic templating to keep your templates up to date with any external source like a scraped item from the web or a database entry etc.
- Features
- Quick Start
- Project Rationale
- Configuration
- Usage
- FAQ
- What's new in the next version
- License
- Meta
- Use various sources for variable inputs:
- Use scripts to scrape dynamic data from various sources
- Use data source files of various data types
- Use workflow 'env' variables to source variables
- Data source files can be:
- dotenv
- ini
- toml
- yaml
- json
- specify target files to protect from accidental templating
- Leverage the Jinja2 templating language
- Create your template file
config.conf.j2
USERNAME = {{ username }}
HOST = {{ host }}
PORT = {{ port }}
- Configure your workflow file
...
jobs:
template:
runs-on: ubuntu-latest
- name: Jinja templating with environment variables
uses: stephen-ra-king/[email protected]
with:
template: config.conf.j2
target: config.conf
env:
USERNAME: sking
HOST: 192.168.0.1
PORT: 5432
...
I have a project with a readme which references the current project count on PyPI. Obviously this count is very dynamic and changes hour by hour. Manually updating this reference is out of the question. I needed a templating solution utilizing the Jinja templating language with an automated solution periodically running. None of the GitHub action that I searched for provided for this scenario.
| Input | Description | Required | Default |
|---|---|---|---|
| data_source | Path, filename for a source file containing key, value pairs | False | "" |
| data_type | Data type of the data_source - env, toml, yaml, yml, ini, json | False | "" |
| dynamic_script | Path, filename of a script that retrieves dynamic data | False | "" |
| env | Key, value pairs in yaml format (key: value) | False | "" |
| protect | Turns protection from accidental overwrite for a target on | False | "" |
| requires | A list of python requirements for the dynamic_script (if any) | False | "" |
| template | Path , filename for the template file that will be rendered with data | True | "" |
| target | Target for the rendered template | True | "" |
| variables | Key, value pairs in .env file format (key=value) | False | "" |
| strict | Determines if action will fail on missing values | False | "" |
I wont be covering Jinja and will assume that you already some knowledge of the language. And yes it is a language which has conditionals and loop structures etc. Jinja is incredibly useful and used by many well known applications such as: Django, Flask, Ansible, NAPALM, Pelican etc. Jinja's combination of logic, control structures, and variable interpolation makes it highly useful for creating dynamic output based on various input data.
Some Jinja Resources:
- Jinja2 Documentation
- Real Python - "Jinja2: The Python Template Engine
- The Flask Mega-Tutorial Part II: Templates
Github workflow environment variable can be utilized in the usual way in the workflow file. Key, value pairs are specified in the in yaml format (key: value)
jobs:
template1:
runs-on: ubuntu-latest
steps:
- name: Jinja templating with environment variables
uses: stephen-ra-king/[email protected]
with:
template: templates/env_variables.txt.j2
target: targets/env_variables.txt
env:
SERVER_HOST: staging.example.com
TIMEOUT: 90
Key, value pairs can be specified in the dotenv style format (key=value)
jobs:
template2:
runs-on: ubuntu-latest
steps:
- name: Jinja templating using variables
uses: stephen-ra-king/[email protected]
with:
template: templates/variables.txt.j2
target: targets/variables.txt
variables: |
server_host=staging.example.com
timeout=45
The variables can be specified in a separate file that can be one of the following formats: dotenv, yaml, json, ini or toml.
create the data source file (e.g. an ini here)
ini_data_file
[APP]
ENVIRONMENT = development
DEBUG = True
[DATABASE]
USERNAME = sking
PASSWORD = ********
HOST = 127.0.0.1
PORT = 5432
DB = database
Then in the workflow file specify the data source file name and data type as follows:
jobs:
template3:
runs-on: ubuntu-latest
steps:
- name: Jinja templating with data file - ini
uses: stephen-ra-king/[email protected]
with:
template: templates/ini_template_file
target: targets/ini_target_file
data_source: templates/ini_data_file
data_type: ini
Create the python script necessary to extract the required data. (More on this later)
Then specify the script in the workflow file:
jobs:
template4:
runs-on: ubuntu-latest
steps:
- name: Jinja templating using dynamic script
uses: stephen-ra-king/[email protected]
with:
template: templates/counter.txt
target: targets/counter.txt
dynamic_script: scripts/templater.py
Values to use in the workflow file
| File Type | workflow value to use |
|---|---|
| Dotenv | env |
| Tom's Obvious, Minimal Language | toml |
| YAML Ain't Markup Language | yaml or yml |
| Initialization | ini |
| JavaScript Object Notation | json |
ENVIRONMENT=development
DEBUG=True
HOST=127.0.0.1
PORT=5432
[APP]
ENVIRONMENT = development
DEBUG = True
[DATABASE]
HOST = 127.0.0.1
PORT = 5432
---
APP:
ENVIRONMENT: development
DEBUG: true
DATABASE:
HOST: 127.0.0.1
PORT: 5432
{
"APP": {
"DEBUG": true,
"ENVIRONMENT": "development"
},
"DATABASE": {
"HOST": "127.0.0.1",
"PORT": 5432,
}
}
For obvious reasons I cannot write these scripts for you.
However it must follow a pattern and contain certain structures:
e.g. dynamic_script.py. You can use this as a template (pun intended)
#!/usr/bin/env python3
from pathlib import Path
env_file = "".join([Path(__file__).stem, ".env"])
def write_to_env_file(key, value):
entry = "".join([key, "=", value, "\n"])
with open(env_file, mode="a") as file:
file.write(entry)
def get_value1():
# Write the steps necessary to get "value1" here
write_to_env_file("KEY1", "value1")
def get_value2():
# Write the steps necessary to get "value2" here
write_to_env_file("KEY2", "value2")
def main():
get_value1()
get_value2()
if __name__ == "__main__":
SystemExit(main())This will produce the following 'env' file:
dynamic_script.env
KEY1=value1
KEY2=value2
Essentially you can get as many variables as you like, with whatever methods you like.
The bottom line is that it must create an 'env' file with the same name as the script (in the same location) except with an 'env' extension and thats it.
The ideal way of scheduling a script is with cron. This is very similar to the crontab scheduling system in Unix-like operating systems. see GitHub workflow schedule.
This can be setup using an entry similar to the following in your workflow file:
on:
schedule:
- cron: '30 5 * * 1,3'
You can test your schedules with various online resources:
If you are using 3rd party modules in your scripts then you can use the 'requires' input keyword.
It is advisable to pin your requirements as in the following example.
- name: Template readme file
uses: stephen-ra-king/[email protected]
if: always()
with:
template: templates/README.md.j2
target: README.md
protect: true
requires: |
beautifulsoup4==4.12.2
requests==2.31.0
dynamic_script: templater.py
Obviously with templating you are accepting the fact that the target will be overwritten each time the template is rendered. So if you make a change to a target file (commit and push), you will loose those changes the next time the template is rendered.
You can 'protect' a file from a single templating operation using the 'protect' input keyword.
...
jobs:
template:
runs-on: ubuntu-latest
- name: Jinja templating with environment variables
uses: stephen-ra-king/[email protected]
with:
template: config.conf.j2
target: config.conf
protect: true
env:
USERNAME: sking
HOST: 192.168.0.1
PORT: 5432
...
If the action determines that the target has been altered since the last templating the action will fail and you will get a message similar to the following in the action run log:
ValueError: Target file has been updated since last templating
This will give you a chance to revise your work workflow. The next time the action runs however the target will be overwritten. The design of this may change in future.
By default, when a variable is undefined in a Jinja2 template, the engine will treat it as an empty string ("") and continue rendering the template without raising any errors. This behavior can lead to potential bugs and make it harder to detect issues when working with templates.
The 'strict' mode helps improve template robustness and prevent silent errors caused by undefined variables. When StrictUndefined is enabled, Jinja2 raises an exception whenever an undefined variable is encountered during template rendering. This makes it easier to identify and handle missing or incorrect data in your templates.
...
jobs:
template:
runs-on: ubuntu-latest
- name: Jinja templating with environment variables
uses: stephen-ra-king/[email protected]
with:
template: config.conf.j2
target: config.conf
strict: true
env:
USERNAME: sking
HOST: 192.168.0.1
PORT: 5432
...
You can use multiple templating steps in one job. However, if one step fails the all the following steps will be skipped. You can avoid this ny using multiple jobs but this has an overhead.
Up until now I've only concentrated on the jinja-genie action. But to have a fully operation workflow you will need to utilize other actions as well:
For example you will need to 'checkout' your repository, fetch and merge remote repository changes and use git to add and commit changed files:
Here is what a fully functional 'jinja-genie.yaml' workflow file looks like:
name: Jinja-Genie templater
on:
push:
branches: ["main"]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout@v3
- name: Fetch and Merge Remote Changes
run: git pull origin main
- name: Jinja templating using dynamic script
uses: stephen-ra-king/[email protected]
if: always()
with:
template: templates/counter.txt
target: targets/counter.txt
dynamic_script: templater.py
- name: Jinja templating using variables
uses: stephen-ra-king/[email protected]
if: always()
with:
template: templates/variables.txt.j2
target: targets/variables.txt
protect: true
variables: |
server_host=staging.example.com
timeout=45
- name: Jinja templating with environment variables
uses: stephen-ra-king/[email protected]
if: always()
with:
template: templates/env_variables.txt.j2
target: targets/env_variables.txt
env:
SERVER_HOST: staging.example.com
TIMEOUT: 90
- name: Jinja templating with data file - ini
uses: stephen-ra-king/[email protected]
if: always()
with:
template: templates/ini_template_file
target: targets/ini_target_file
data_source: templates/ini_data_file
data_type: ini
- name: Commit changes
uses: EndBug/add-and-commit@v9
if: always()
with:
author_name: Jinja Genie
author_email: [email protected]
message: "Jinja2 template successfully applied"
add: .
Q. Can I use any other language apart from Python to get 'dynamic' variables?
A. No. This would be way to complicated for the docker container.
Q. Does the target file need to exist before template rendering.
A. No. The target will be created by the Jinja engine. The containing directory must exist though. Git generally ignores empty directories though. You can create empty directories and use a ',gitkeep' file inside these.
Q. Can I use multiple templates / targets for a single step.
A. No. You can use multiple variables per step with a single template / target but you can only use a single template / target pair. This may change in subsequent releases. I have already have several ideas on how to implement this.
- to be decided
Distributed under the MIT license.
Author: Stephen R A King ([email protected])