Python Boilerplate

python_boilerplate is a boilerplate project for Python. Based on template sourcery-ai/python-best-practices-cookiecutter.

Official Docker Image

Features

Here are the highlights of python_boilerplate:

1. Inherited from modern and the latest Python technologies:

   Python -

   Pipenv is to Python virtualenv management tool for the project.

2. Data validation using Python type hints with Pydantic.

3. Highly customizable data analysis with pandas, enhanced array operation with NumPy. Supports CSV, Excel, JSON, and so on.

4. Data persistence with peewee, SQLite3 as a local database.

5. Simple and flexible retry with Tenacity.

6. Environment variable and configuration with pyhocon. Read ${ENVIRONMENT_VARIABLE} when startup.

7. Sensible and human-friendly approach to creating, manipulating, formatting and converting dates, times, and timestamps with Arrow.

8. Generate fake data with Faker.

9. Customized function decorator @async_function to enable function to run asynchronously; @peewee_table class decorator to register ORM tables; @elapsed_time(level="INFO") to profile a function elapsed time.

10. Testing with pytest, integrating pytest-mock for mocking, pytest-cov for code coverage analysis and pyinstrument for Python stack profiler.

11. Formatting with black.

12. Import sorting with isort.

13. Static typing with mypy.

14. Linting with flake8.

15. Git hooks that run all the above with pre-commit.

16. Deployment ready with Docker.

17. Continuous Integration with GitHub Actions.

18. Loguru logging configuration. The log sample,

2022-09-17 14:13:52.385 | ⚠️ WARNING  | 6860 | MainThread      | python_boilerplate.repository.model.base_model.<module>:24 - SQLite database created. Path: [/Users/johnny/Projects/PyCharmProjects/python_boilerplate/data/python_boilerplate.db], <peewee.SqliteDatabase object at 0x1191e1390>
2022-09-17 14:13:52.386 | ℹ️ INFO     | 6860 | MainThread      | python_boilerplate.common.orm.peewee_table:16 - Registering peewee table: StartupLog
2022-09-17 14:13:52.387 | 🐞 DEBUG    | 6860 | MainThread      | peewee.execute_sql:3185 - ('CREATE TABLE IF NOT EXISTS "startup_log" ("id" INTEGER NOT NULL PRIMARY KEY, "current_user" VARCHAR(50) NOT NULL, "host" VARCHAR(50) NOT NULL, "command_line" TEXT NOT NULL, "current_working_directory" TEXT NOT NULL, "startup_time" DATETIME NOT NULL, "created_by" VARCHAR(50) NOT NULL, "created_time" DATETIME NOT NULL, "modified_by" VARCHAR(50) NOT NULL, "modified_time" DATETIME NOT NULL)', [])
2022-09-17 14:13:52.530 | ℹ️ INFO     | 6860 | MainThread      | python_boilerplate.<module>:53 - Started python_boilerplate in 0.117 seconds (117.26 ms)

Usage

1. Clone or download this project.

   $ git clone https://github.com/johnnymillergh/python_boilerplater.git

2. Build with the newest PyCharm.

3. Click the green triangle to Run.

Setup

1. Setup the development environment

   # Install pipx if pipenv and cookiecutter are not installed
   $ python3 -m pip install pipx
   $ python3 -m pipx ensurepath
   
   # Install pipenv using pipx
   $ pipx install pipenv

2. Install dependencies

   $ pipenv install --quiet --dev

3. Install mypy types

   $ pipenv run mypy --install-types

4. Setup pre-commit and pre-push hooks

   $ pipenv run pre-commit install -t pre-commit
   $ pipenv run pre-commit install -t pre-push

Useful Commands

Run Python Module

$ python3 -m python_boilerplate

Run Python Script

Append your project’s root directory to PYTHONPATH — In any environment you wish to run your Python application such as Docker, vagrant or your virtual environment i.e. in bin/activate, run the below command:

How to Fix ModuleNotFoundError and ImportError

For macOS or Linux,

# Ensure `pwd` is the root directory of the project
$ PYTHONPATH=`pwd` pipenv run python3 python_boilerplate/demo/pandas_usage.py
$ PYTHONPATH=`pwd` pipenv run python3 python_boilerplate/demo/multithread_and_thread_pool_usage.py

# Run the main module
$ PYTHONPATH=`pwd` pipenv run python3 python_boilerplate/__main__.py

# Run a pytest script
$ pytest --log-cli-level=DEBUG --capture=no tests/common/test_debounce_throttle.py

# Run a pytest script with `-k` EXPRESSION
$ pytest --log-cli-level=DEBUG --capture=no tests/common/test_debounce_throttle.py -k 'test_debounce'

# For more details of pytest command
$ pipenv run pytest --help

For Windows Terminal,

# Ensure `$PWD.Path` is the root directory of the project
$ $env:PYTHONPATH=$PWD.Path; pipenv run python .\python_boilerplate\demo\pandas_usage.py
$ $env:PYTHONPATH=$PWD.Path; pipenv run python .\python_boilerplate\demo\multithread_and_thread_pool_usage.py

# Run the main module
$ $env:PYTHONPATH=$PWD.Path; pipenv run python .\python_boilerplate\__main__.py

Package with PyInstaller

⚠️ WANRING

Need to call freeze_support() immediately when startup in python_boilerplate/__init__.py

from multiprocessing import freeze_support

freeze_support()

• Pyinstaller multiprocessing name of process is always "MainProcess" #3957
• pyinstaller linux binary is getting restarted again and again #4190

Build artifact with macOS or Linux,

$ pipenv run pyinstaller --console \
--add-data "setup.cfg:." \
--add-data "python_boilerplate/resources/*:python_boilerplate/resources" \
--name pandas_usage \
--clean --noconfirm python_boilerplate/demo/pandas_usage.py

On Windows,

$ pipenv run pyinstaller --console `
--add-data "setup.cfg;." `
--add-data "python_boilerplate/resources/*;python_boilerplate/resources" `
--name multithread_and_thread_pool_usage `
--clean --noconfirm python_boilerplate/demo/multithread_and_thread_pool_usage.py

Run Unit Tests

Run with pytest, analyze code coverage, generate HTML code coverage reports, fail the test if coverage percentage is under 90%,

$ pipenv run pytest --cov --cov-report html --cov-fail-under=85 --capture=no --log-cli-level=INFO

Benchmark with pytest,

$ pipenv run pytest --capture=no --log-cli-level=ERROR -n 0 --benchmark-only

Conventional Changelog CLI

1. Install global dependencies (optional if installed):

   $ npm install -g conventional-changelog-cli

2. This will not overwrite any previous changelogs. The above generates a changelog based on commits since the last semver tag that matches the pattern of "Feature", "Fix", "Performance Improvement" or "Breaking Changes".

   $ conventional-changelog -p angular -i CHANGELOG.md -s

3. If this is your first time using this tool and you want to generate all previous changelogs, you could do:

   $ conventional-changelog -p angular -i CHANGELOG.md -s -r 0

Check Versions of Python Packages

$ pipenv run pip list --outdated

Output be like,

Package          Version   Latest     Type
---------------- --------- ---------- -----
black            23.10.1   23.11.0    wheel
certifi          2023.7.22 2023.11.17 wheel
Faker            19.12.0   20.1.0     wheel
identify         2.5.31    2.5.32     wheel
idna             3.4       3.6        wheel
matplotlib       3.8.0     3.8.2      wheel
mypy             1.6.1     1.7.1      wheel
numpy            1.26.1    1.26.2     wheel
pandas           2.1.2     2.1.3      wheel
pip              23.1.2    23.3.1     wheel
platformdirs     3.11.0    4.0.0      wheel
pydantic         2.4.2     2.5.2      wheel
pydantic_core    2.10.1    2.14.5     wheel
pyinstaller      6.1.0     6.2.0      wheel
pyinstrument     4.6.0     4.6.1      wheel
pytest-html      4.0.2     4.1.1      wheel
pytest-xdist     3.3.1     3.5.0      wheel
setuptools       68.2.2    69.0.2     wheel
types-setuptools 68.2.0.0  69.0.0.0   wheel
urllib3          2.0.7     2.1.0      wheel
virtualenv       20.24.6   20.24.7    wheel
wheel            0.41.3    0.42.0     wheel

CI (Continuous Integration)

• GitHub Actions are for building projects and running tests.
• Travis CI is for publishing Docker Hub images of SNAPSHOT and RELEASE.

FAQ

1. ExecutableNotFoundError: Executable /bin/bash not found when using pre-commit

   Solution: delete .git/hooks/pre-commit.legacy and then reinstall, also deleted .git/hooks/pre-commit just to be sure before installing again.

2. Installing all stub type packages,

   $ mypy --install-types

Maintainers

@johnnymillergh.

Contributing

Feel free to dive in! Open an issue.

Contributors

This project exists thanks to all the people who contribute.

• Johnny Miller [@johnnymillergh]
• …

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

Credits

This package was created with Cookiecutter and the sourcery-ai/python-best-practices-cookiecutter project template.

Inspired by How to set up a perfect Python project.

License

Apache License © Johnny Miller

2021—Present