Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added API checks and updated README. #14

Merged
merged 6 commits into from
Oct 27, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .github/workflows/generate-postman-collection.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: generate-postman-collection

on:
push:
paths:
- "src/openapi.yaml"
- ".github/workflows/generate-postman-collection.yml"
branches:
- '**'
workflow_dispatch:

jobs:
run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Use Node.js
uses: actions/setup-node@v1
with:
node-version: '12'
- name: Install dependencies
run: npm install -g openapi-to-postmanv2
- name: Create tests folder
run: mkdir -p ./tests/postman
- name: Generate Postman collection
run: openapi2postmanv2 -s ./src/openapi.yaml -o ./tests/postman/collection.json --pretty
36 changes: 36 additions & 0 deletions .github/workflows/generate-sdks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: generate-sdks

on:
push:
paths:
- "src/openapi.yaml"
- ".github/workflows/generate-sdks.yml"
branches:
- '**'
workflow_dispatch:

jobs:
run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Use Node.js
uses: actions/setup-node@v1
with:
node-version: '12'
- name: Install dependencies
run: npm install -g @openapitools/openapi-generator-cli
- name: Validate schema
run: openapi-generator-cli validate -i ./src/openapi.yaml
- name: Generate Java client
run: openapi-generator-cli generate -i ./src/openapi.yaml --global-property=modelTests=false,apiTests=false,modelDocs=false,apiDocs=false \
-o ./sdks/java -g java --additional-properties=dateLibrary=java8,java8=true,optionalProjectFile=false,optionalAssemblyInfo=false
- name: Generate .NET Core client
run: openapi-generator-cli generate -i ./src/openapi.yaml --global-property=modelTests=false,apiTests=false,modelDocs=false,apiDocs=false \
-o ./sdks/netcore -g csharp-netcore --additional-properties=optionalProjectFile=false,optionalAssemblyInfo=false
- name: Generate .NET Full Framework client
run: openapi-generator-cli generate -i ./src/openapi.yaml --global-property=modelTests=false,apiTests=false,modelDocs=false,apiDocs=false \
-o ./sdks/net -g csharp --additional-properties=optionalProjectFile=false,optionalAssemblyInfo=false
- name: Generate Python client
run: openapi-generator-cli generate -i ./src/openapi.yaml --global-property=modelTests=false,apiTests=false,modelDocs=false,apiDocs=false \
-o ./sdks/python -g python --additional-properties=optionalProjectFile=false,optionalAssemblyInfo=false+
24 changes: 24 additions & 0 deletions .github/workflows/lint-oas.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name: lint-oas

on:
push:
paths:
- src/openapi.yaml
- .github/workflows/lint-oas.yml
branches:
- '**'
workflow_dispatch:

jobs:
run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Use Node.js
uses: actions/setup-node@v1
with:
node-version: '12'
- name: Install spectral
run: npm install -g @stoplight/spectral
- name: Run OAS linter
run: spectral lint ./src/openapi.yaml
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,8 @@ src/objecttypes/static/css/**/*.css.map

src/objecttypes/static/fonts/
src/objecttypes/static/js/

# Statics generated by Workflows
/tests/postman
/sdks/
openapitools.json
78 changes: 28 additions & 50 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,91 +1,69 @@
# This is a multi-stage build file, which means a stage is used to build
# the backend (dependencies), the frontend stack and a final production
# stage re-using assets from the build stages. This keeps the final production
# image minimal in size.

# Stage 1 - Backend build environment
# includes compilers and build tooling to create the environment
FROM python:3.7-buster AS backend-build
# Stage 1 - Compile needed python dependencies
FROM python:3.7-buster AS build

RUN apt-get update && apt-get install -y --no-install-recommends \
libpq-dev \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /app
RUN mkdir /app/src

# Ensure we use the latest version of pip
RUN pip install pip setuptools -U
COPY ./requirements /app/requirements
RUN pip install pip setuptools -U
RUN pip install -r requirements/production.txt


# Stage 2 - Install frontend deps and build assets
FROM node:13-buster AS frontend-build

RUN apt-get update && apt-get install -y --no-install-recommends \
git \
&& rm -rf /var/lib/apt/lists/*
# Stage 2 - build frontend
FROM mhart/alpine-node:10 AS frontend-build

WORKDIR /app

# copy configuration/build files
COPY ./build /app/build/
COPY ./*.json ./*.js ./.babelrc /app/

# install WITH dev tooling
COPY ./*.json /app/
RUN npm ci

# copy source code
COPY ./src /app/src
COPY ./gulpfile.js ./webpack.config.js ./.babelrc /app/
COPY ./build /app/build/

# build frontend
COPY src/objecttypes/sass/ /app/src/objecttypes/sass/
COPY src/objecttypes/js/ /app/src/objecttypes/js/
RUN npm run build


# Stage 3 - Build docker image suitable for production
FROM python:3.7-buster
# Stage 3 - Build docker image suitable for execution and deployment
FROM python:3.7-buster AS production

# Stage 3.1 - Set up the needed production dependencies
# install all the dependencies for GeoDjango
RUN apt-get update && apt-get install -y --no-install-recommends \
procps \
vim \
postgresql-client \
# lxml deps
# libxslt \
&& rm -rf /var/lib/apt/lists/*

COPY --from=build /usr/local/lib/python3.7 /usr/local/lib/python3.7
COPY --from=build /usr/local/bin/uwsgi /usr/local/bin/uwsgi

# Stage 3.2 - Copy source code
WORKDIR /app
COPY ./bin/docker_start.sh /start.sh
RUN mkdir /app/log
RUN mkdir /app/media

# copy backend build deps
COPY --from=backend-build /usr/local/lib/python3.7 /usr/local/lib/python3.7
COPY --from=backend-build /usr/local/bin/uwsgi /usr/local/bin/uwsgi
COPY --from=backend-build /app/src/ /app/src/
RUN mkdir /app/log /app/config

# copy frontend build statics
COPY --from=frontend-build /app/src/objecttypes/static /app/src/objecttypes/static

# copy source code
COPY --from=frontend-build /app/src/objecttypes/static/css /app/src/objecttypes/static/css
COPY --from=frontend-build /app/src/objecttypes/static/js /app/src/objecttypes/static/js
COPY ./src /app/src

RUN useradd -M -u 1000 maykin
RUN chown -R maykin /app

# drop privileges
USER maykin

ARG COMMIT_HASH
ARG RELEASE
ENV GIT_SHA=${COMMIT_HASH}
ENV RELEASE=${RELEASE}

ENV DJANGO_SETTINGS_MODULE=objecttypes.conf.docker

ARG SECRET_KEY=dummy

# Run collectstatic, so the result is already included in the image
RUN python src/manage.py collectstatic --noinput

LABEL org.label-schema.vcs-ref=$COMMIT_HASH \
org.label-schema.vcs-url="https://github.com/maykinmedia/objecttypes-api" \
org.label-schema.version=$RELEASE \
org.label-schema.name="Objecttypes API"

EXPOSE 8000
CMD ["/start.sh"]
CMD ["/start.sh"]
69 changes: 8 additions & 61 deletions INSTALL.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Prerequisites

You need the following libraries and/or programs:

* `Python`_ 3.6 or above
* `Python`_ 3.7 or above
* Python `Virtualenv`_ and `Pip`_
* `PostgreSQL`_ 10 or above
* `Node.js`_
Expand All @@ -44,7 +44,7 @@ development machine.

.. code-block:: bash

$ git clone [email protected]:maykinmedia/objecttypes.git
$ git clone [email protected]:maykinmedia/objecttypes-api.git
$ cd objecttypes

3. Install all required libraries.
Expand Down Expand Up @@ -169,7 +169,7 @@ The easiest way to get the project started is by using `Docker Compose`_.

.. code-block:: bash

$ git clone [email protected]:maykinmedia/objecttypes.git
$ git clone [email protected]:maykinmedia/objecttypes-api.git
Cloning into 'objecttypes'...
...

Expand All @@ -180,28 +180,28 @@ The easiest way to get the project started is by using `Docker Compose`_.
.. code-block:: bash

$ docker-compose up -d
Starting objecttypes_db_1 ... done
Starting objecttypes_web_1 ... done
Starting objecttypes-api_db_1 ... done
Starting objecttypes-api_web_1 ... done

It can take a while before everything is done. Even after starting the web
container, the database might still be migrating. You can always check the
status with:

.. code-block:: bash

$ docker logs -f objecttypes_web_1
$ docker logs -f objecttypes-api_web_1

3. Create an admin user and load initial data. If different container names
are shown above, use the container name ending with ``_web_1``:

.. code-block:: bash

$ docker exec -it objecttypes_web_1 /app/src/manage.py createsuperuser
$ docker exec -it objecttypes-api_web_1 /app/src/manage.py createsuperuser
Username: admin
...
Superuser created successfully.

$ docker exec -it objecttypes_web_1 /app/src/manage.py loaddata admin_index groups
$ docker exec -it objecttypes-api_web_1 /app/src/manage.py loaddata admin_index groups
Installed 5 object(s) from 2 fixture(s)

4. Point your browser to ``http://localhost:8000/`` to access the project's
Expand Down Expand Up @@ -247,59 +247,6 @@ all settings.

$ docker exec -it objecttypes /app/src/manage.py createsuperuser

Building and publishing the image
---------------------------------

Using ``bin/release-docker-image``, you can easily build and tag the image.

The script is based on git branches and tags - if you're on the ``master``
branch and the current ``HEAD`` is tagged, the tag will be used as
``RELEASE_TAG`` and the image will be pushed. If you want to push the image
without a git tag, you can use the ``RELEASE_TAG`` envvar.

The image will only be pushed if the ``JOB_NAME`` envvar is set. The image
will always be built, even if no envvar is set. The default release tag is
``latest``.

Example usage:

.. code-block:: bash

JOB_NAME=publish RELEASE_TAG=dev ./bin/release-docker-image.sh


Staging and production
======================

Ansible is used to deploy test, staging and production servers. It is assumed
the target machine has a clean `Debian`_ installation.

1. Make sure you have `Ansible`_ installed (globally or in the virtual
environment):

.. code-block:: bash

$ pip install ansible

2. Navigate to the project directory, and install the Maykin deployment
submodule if you haven't already:

.. code-block:: bash

$ git submodule update --init

3. Run the Ansible playbook to provision a clean Debian machine:

.. code-block:: bash

$ cd deployment
$ ansible-playbook <test/staging/production>.yml

For more information, see the ``README`` file in the deployment directory.

.. _Debian: https://www.debian.org/
.. _Ansible: https://pypi.org/project/ansible/


Settings
========
Expand Down
Loading