Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support scan_ function hooks #267

Merged
merged 6 commits into from
Aug 4, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion build/functions.j2
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ function {{ "scan_" ~ function if scan else function }}() {
# Dynamically register hooks
for file in /opt/hooks/bin/*.sh; do
command=$(awk -F\: '/register_hook/ { gsub(/ /,""); print $2 }' "${file}")
if [[ "${command}" == "{{ function }}" ]]; then
if [[ "${command}" == "{{ 'scan_' ~ function if scan else function }}" ]]; then
easy_infra_{{ function | replace("-", "_") }}_hooks+=("${file}")
fi
done
Expand Down
4 changes: 3 additions & 1 deletion docs/Hooks/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@ like to hook::
# register_hook: terraform

.. note::
Only packages (or their related aliases, where specified) listed in ``easy_infra.yml`` at build time are supported. This means that, if a package has multiple aliases it will need to be registered against each of those aliases.
Only packages or their related aliases listed in ``easy_infra.yml`` at build time, or the ``scan_`` functions (which only perform the security scans, i.e.
``scan_terraform`` or ``scan_cloudformation``) are supported. This means that, if a package has multiple aliases, or if you'd like to support the ``scan_``
function, you will need to register your hook against each of those.

Example
^^^^^^^
Expand Down
17 changes: 12 additions & 5 deletions docs/Technical Details/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -243,13 +243,20 @@ Internal naming
above), as configured in the ``easy_infra.yml`` file used to build the image.
- Package: The name of a package that can be installed to perform a necessary function. It could be a tool, a security tool, or a generic helper such
as ``fluent-bit`` or ``envconsul``.
- Command: A runtime command, following the use of the term by bash (see the "Command Execution" of this documentation). This could be an alias, a
package, or some other executable on the user's ``PATH``.
- Command: A runtime command, following the use of the term by bash (see `this documentation<https://www.ibm.com/docs/en/aix/7.2?topic=c-command-command>`_).
This could be an alias, a package, or some other executable on the user's ``PATH``.
- Alias: An executable file in the easy_infra and root user's ``PATH`` which executes the installed by a package. While ``aws-cli`` would be a package, ``aws``
would be the associated alias.
- Environment: A supported destination that a tool (see above) may deploy into, such as a cloud provider. An environment constitutes a bundle of
packages.

Scan-only functions
===================

In addition to the Alias and Package functions which are generated, ``scan_`` functions are also created to allow you to run only the security scans (and
related hooks) for a given Package or Alias. So, for instance, if you have a Package of ``terraform``, the related scan function would be ``scan_terraform``.
Whereas, if you have a Package of ``ansible`` which has Aliases of ``ansible`` and ``ansible-playbook``, you will get scan functions of ``scan_ansible`` and
``scan_ansible-playbook``.

High-Level Design of the image build process
============================================
Expand Down Expand Up @@ -280,9 +287,9 @@ build on top of ``Dockerfile.abc``, and it is both expected that in ``Dockerfrag
Runtime user support
====================

By default, ``easy_infra`` runs as the ``easy_infra`` user and should be fully functional, however we also support the
``root`` user due to various file system permission issues that often occur in pipelines when running as non-root users.
Where possible, the ``easy_infra`` user should be used due to the security risks of running containers as ``root``.
By default, ``easy_infra`` runs as the ``easy_infra`` user and should be fully functional, however we also support the ``root`` user due to various file system
permission issues that often occur in pipelines when running as non-root users. Where possible, the ``easy_infra`` user should be used due to the security risks
of running containers as ``root``.

Adding to the project
=====================
Expand Down
2 changes: 1 addition & 1 deletion easy_infra/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ def opinionated_docker_run(
detach: bool = True,
environment: dict = {},
user: str = "",
volumes: dict = {},
volumes: dict | list = {},
working_dir: str = "/iac/",
expected_exit: int = 0,
check_logs: Pattern[str] | None = None,
Expand Down
3 changes: 3 additions & 0 deletions tests/terraform/hooks/scan_terraform/secure.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
terraform {
required_version = "1.5.4"
}
1 change: 1 addition & 0 deletions tests/terraform/hooks/scan_terraform/unwanted_file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
I am an unwanted file, use a hook to exit 230 if it finds me!
8 changes: 8 additions & 0 deletions tests/test-hooks/50-find-unwanted.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
# register_hook: scan_terraform

findings="$(find . -name "unwanted_file")"

if [[ "${findings}" ]]; then
exit 230
fi
39 changes: 35 additions & 4 deletions tests/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -352,13 +352,23 @@ def exec_tests(
image: str,
tests: list[tuple[dict, str, int]],
user: str = "",
volumes: dict,
volumes: dict | list[dict],
network_mode: Union[str, None] = None,
) -> int:
"""Execute the provided tests and return a count of tests run"""
num_tests_ran = 0
config_dir = list(volumes.keys())[0]
working_dir = volumes[config_dir]["bind"]
if isinstance(volumes, dict):
config_dir = list(volumes.keys())[0]
working_dir = volumes[config_dir]["bind"]
final_volumes: dict = volumes
elif isinstance(volumes, list):
# Use the first dict in the list to extract working dir, etc.
config_dir = list(volumes[0].keys())[0]
working_dir = volumes[0][config_dir]["bind"]
final_volumes: list[str] = []
for volume in volumes:
for k, v in volume.items():
final_volumes.append(str(k.absolute()) + ":" + v["bind"])

if not user:
LOG.error("A user must be specified to execute tests!")
Expand All @@ -372,7 +382,7 @@ def exec_tests(
expected_exit=expected_exit,
image=image,
user=user,
volumes=volumes,
volumes=final_volumes,
working_dir=working_dir,
network_mode=network_mode,
)
Expand Down Expand Up @@ -708,6 +718,10 @@ def run_terraform(*, image: str, user: str) -> None:
hooks_config_volumes: dict[Path, dict[str, str]] = {
hooks_config_dir: {"bind": working_dir, "mode": "rw"}
}
hooks_script_dir: Path = TESTS_PATH.joinpath("test-hooks")
hooks_script_volumes: dict[Path, dict[str, str]] = {
hooks_script_dir: {"bind": "/opt/hooks/bin/", "mode": "ro"}
}
fluent_bit_config_host: Path = TESTS_PATH.joinpath("fluent-bit.outputs.conf")
fluent_bit_config_container: str = (
"/usr/local/etc/fluent-bit/fluent-bit.outputs.conf"
Expand Down Expand Up @@ -1001,6 +1015,23 @@ def run_terraform(*, image: str, user: str) -> None:
tests=tests, volumes=secure_volumes, image=image, user=user
)

# Ensure the scan_ hooks work and that you can provide custom hooks via volume mounting at runtime
# Tests is a list of tuples containing the test environment, command, and expected exit code
tests: list[tuple[dict, str, int]] = [ # type: ignore
(
{"DISABLE_SECURITY": "true"},
"scan_terraform",
230,
) # This tests scan_terraform hook registration; if it runs and finds the unwanted file it should exit 230
]

LOG.debug("Testing scan_ hooks and custom hooks volume mounted at runtime")
volumes: list[dict[Path, dict[str, str]]] = [
hooks_config_volumes,
hooks_script_volumes,
]
num_tests_ran += exec_tests(tests=tests, volumes=volumes, image=image, user=user)

# Ensure the easy_infra hooks work as expected when network access is available
# Tests is a list of tuples containing the test environment, command, and expected exit code
tests: list[tuple[dict, str, int]] = [ # type: ignore
Expand Down