Skip to content

Contributing Guidelines

Jump to bottom
Gustavo Silva edited this page Feb 10, 2023 · 3 revisions

Contributing to Surface

The following sections covers a fully detailed list of steps on how to set up and start Surface locally. The first sections are both useful for development but also for users, if they desire to not use Docker.

Setup Surface Locally

  1. Make sure you have Python 3.9 or higher
  2. git clone [email protected]:surface-security/surface.git (ssh config is required - https://docs.github.com/en/authentication/connecting-to-github-with-ssh)
  3. cd ./surface
  4. Install OS dependencies
  5. Upgrade your pip - python3 -m pip install --upgrade pip
  6. Create and activate virtual environment: using default venv, for instance: 
    # MacOS / Linux:
python3 -m venv venv3
source venv3/bin/activate

# Windows:
python3 -m venv venv3
venv3\Scripts\activate.bat
  7. Install requirements: 
    python3 -m pip install -r requirements_dev.txt

    If macOS use export LDFLAGS=-L/usr/local/opt/openssl/lib before the previous the commands above
    Also in macOS, read https://dev.to/ruivieira/fixing-libcrypto-ansible-crashes-on-macos-mm2

  8. Create local env file for variables, usernames, passwords: touch local.env
  9. (Optional) Default Surface configuration will use a SQLite file which is enough for most development purposes. If you'd rather use MySQL in Docker:
    • Create MySQL docker container bash ../dev/1-create_mysql_container.sh
    • Add the mysql config in local.env 
      SURF_DATABASE_URL=mysql://root:@127.0.0.1:3306/surface
SURF_DATABASE_PASSWORD=""
  10. Apply all migrations to the database python3 manage.py migrate
  11. You can create a super user with python3 manage.py createsuperuser
  12. Run the application: python3 manage.py runserver
  13. Log in with your created super user account.

Note: You can exit the virtual environment with deactivate.

Note: there are many virtualenv options out there if you don't like venv, it's just the default one. virtualenvwrapper-win, hatch, pyenv, or any other will do just fine.

VSCode Configuration

Sample .vscode/settings.json that includes:

  • autocomplete, pytest, sonarqube link, autoformatting (black) and some exclusions from tree browser
{
    "python.autoComplete.extraPaths": [
        "surface",
    ],
    "python.analysis.extraPaths": [
        "surface",
    ],
    "python.testing.pytestArgs": [
        "surface",
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.formatting.provider": "black",
    "files.exclude": {
        "**/__pycache__": true,
        "**/.pytest_cache": true,
        "**/mysql.sql": true
    }
}

You can also use this launch.json example that includes Python debug config, Django's runserver and launchers for commands with and without arguments:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Run Command",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/surface/manage.py",
            "args": [
                "${fileBasenameNoExtension}",
            ],
            "django": true,
            "justMyCode": false
        },
        {
            "name": "Python: Run Command With 1 Arg",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/surface/manage.py",
            "args": [
                "${fileBasenameNoExtension}", "${input:someArgs}"
            ],
            "django": true,
            "justMyCode": false
        },
        {
            "name": "Python: Run Command With 2 Args",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/surface/manage.py",
            "args": [
                "${fileBasenameNoExtension}", "${input:someArgs}", "${input:someArgs2}"
            ],
            "django": true,
            "justMyCode": false
        },
        {
            "name": "Python: Django",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/surface/manage.py",
            "args": [
                "runserver"
            ],
            "django": true,
            "justMyCode": false
        },
        {
            "name": "Debug Unit Test",
            "type": "python",
            "request": "test",
            "justMyCode": false,
        }
    ],
    "inputs": [
        {
          "id": "someArgs",
          "type": "promptString",
          "description": "enter argument"
        },
        {
            "id": "someArgs2",
            "type": "promptString",
            "description": "enter argument 2"
          }
    ]
}

Running tests

Surface and their apps have a battery of unit tests what should be running green at all times. Before changing anything, make sure you have an healthy test environment by running the existing tests.

If you're using sqlite in Surface, no need to perform the below steps. For MySQL:

  1. Start MySQL (if you haven't already). With docker, that's as easy as: docker-compose -f dev/docker-compose-dev.yml up mysql.
  2. Make sure you have at least SURF_DATABASE_URL set up in surface/local.env, which will tell Surface to use that database.
  • if using MySQL from Docker: SURF_DATABASE_URL=mysql://root:@127.0.0.1:23306/surface
  1. Run pytest inside the root of the project:
  • Using -n4 (multiprocessing for tests) option with pytest is recommended for a speedier run.

Contributing to Surface (after you setup it locally)

Please be a good citizen and write descriptive, short and concise commit messages. Run pylint locally, otherwise the style checks will fail and that could be avoided effortlessly.

  1. Create a new branch for your work: git checkout -b my-awesome-branch
  2. Write new code or make changes.
    • If you're adding new functionality, use python3 manage.py startapp to automatically generate the correct file tree for you.
    • Add its code dependencies to YOURAPP/requirements.txt even if they're already included by other apps (but make sure to mention same versions otherwise pip will go crazy). These allow us to keep track of each app's dependencies and uses.
    • Update base requirements surface/requirements.txt with -r YOURAPP/requirements.txt.
    • If you do add an app-specific requirements.txt, add that ADD line to docker/Dockerfile in the highlighted section.
  3. Whenever you are editing anything in the models, make sure you run - python3 manage.py makemigrations to update the database tables.
  4. After you have new migrations or you get new changes from Git, make sure you run again python3 manage.py migrate just to update your table and make sure there are no incompatible changes being pushed.
  5. Use a linter to check the quality of the code - we recommend Visual Studio code. Please try to fix the reported issues/warnings.
  6. Follow Code Style Guidelines section.
  7. Add all files you want to add to the repository - git add <filename(s)>
    • Ideally, commits are brief messages explaining what is added to the project and should be easy to read as well. Avoid committing too many files simultaneously and avoid cryptic commit messages. Both make it harder to review.
    • Do not push your local.env or any other sensitive file. In addition, even though mentioned in the Code Style Guidelines, do not push code with secrets in it.
  8. Commit your changes to your branch - git commit -m "MESSAGE"
  9. Push the new created branch to GitHub - git push --set-upstream origin my-awesome-branch (you need to defined the upstream only at the first push)
  10. Create a new Pull Request (PR) in GitHub from source as being your new branch
  11. After the PR is created and the pipeline is passing, assign the core team as reviewer because it is required to have at least one one approval to merge.

Dependencies

Surface can use mulitple stacks of services, like Elasticsearch, dkron, MySQL and the list continues to grow. Plus, you probably have your own. To create a working docker-compose for you, ensure that:

  1. Take a look at dev/docker-compose-in-a-box.yml
  2. Copy it to dev/docker-compose.yml and make your own adjustments: Read the comments, comment or uncomment the services you don't want
  3. Pay attention to the mentioned/required changes in your local.env
  4. Launch it with docker-compose -f dev/docker-compose.yml up

Specific Surface modules

Scanners

Check scanners/CONTRIBUTING.md

Table of Contents

Using

Contributing

Developing

Clone this wiki locally