Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload encryption with WASM #395

Merged
merged 51 commits into from
Jan 12, 2022
Merged

Upload encryption with WASM #395

Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
d87eedc
begin wasmcrypt implementation
sampsapenna Sep 6, 2021
0a54a66
add upload session initialization functions
sampsapenna Oct 5, 2021
33f5c82
add forgotten dereference
sampsapenna Oct 5, 2021
758220b
add initial wasm encryption implementation
sampsapenna Oct 21, 2021
5903c8b
add skeleton structure for encrypted uploads
sampsapenna Oct 22, 2021
786255a
add libupload.js import
sampsapenna Oct 22, 2021
f0e6824
add encryption FS key init
sampsapenna Oct 22, 2021
b9e13c0
fix libupload path
sampsapenna Oct 22, 2021
af3f822
remove unnecessary chdir
sampsapenna Oct 22, 2021
e9d2ad0
add wasm call
sampsapenna Oct 22, 2021
a789061
add initial MEMFS read for uploading
sampsapenna Oct 22, 2021
2b42f8b
fix missing path from libupload.wasm
sampsapenna Oct 22, 2021
10f497c
change exported functoin in wasm build
sampsapenna Oct 22, 2021
31db76f
log privkey
sampsapenna Nov 1, 2021
8f2cb0b
ignore passphrase
sampsapenna Nov 2, 2021
a464ac1
debug upload
sampsapenna Nov 3, 2021
593e8c9
try nftw correct function signature
sampsapenna Nov 4, 2021
c200ee7
fix recv key paths
sampsapenna Nov 4, 2021
451e215
log encrypted files to console
sampsapenna Nov 4, 2021
629fb0e
precalculate and log input file paths
sampsapenna Nov 4, 2021
ceba861
initialize path string memory
sampsapenna Nov 4, 2021
0a62c68
map file name to Blob instance
sampsapenna Nov 4, 2021
3e8a19a
debug incorrect file mapping
sampsapenna Nov 5, 2021
b58b789
fix File access to be async
sampsapenna Nov 5, 2021
af77358
fix incorrect Blob build
sampsapenna Nov 5, 2021
966def8
remove JS side debug messages
sampsapenna Nov 5, 2021
aff4155
test key passphrase input
sampsapenna Nov 5, 2021
bccda45
fix outdated folder structure in README
sampsapenna Nov 8, 2021
2549c37
collect key passphrase from JS side
sampsapenna Nov 10, 2021
22d9cd5
export ccall
sampsapenna Nov 10, 2021
87d722f
add ephemeral encryption
sampsapenna Nov 10, 2021
2237d24
add ephemeral upload js
sampsapenna Nov 10, 2021
84ec26d
add function export underscore
sampsapenna Nov 10, 2021
25c6244
export getRandomValue
sampsapenna Nov 11, 2021
57eeca6
link sodium to production library
sampsapenna Nov 11, 2021
c2b9873
try stirring randomness
sampsapenna Nov 11, 2021
8ee61d0
unexport getRandomValue
sampsapenna Nov 11, 2021
3caefee
strip debug messages
sampsapenna Nov 12, 2021
af9a837
purge unneeded code
sampsapenna Nov 12, 2021
bc4a5d3
integrate encryption to UI
sampsapenna Nov 15, 2021
95a5e52
check checksum
sampsapenna Nov 18, 2021
b8ca4f7
fix newlines vanishing in tags
sampsapenna Nov 24, 2021
10e8889
refactor upload engine for presigned URLs
sampsapenna Dec 30, 2021
18820b2
separate dependency layers to improve build caching
sampsapenna Dec 30, 2021
8496aeb
remove deprecated component
sampsapenna Dec 31, 2021
e09b4c4
fix uploads into pseudofolders with new upload implementation
sampsapenna Dec 31, 2021
96ce7b4
deprecate upload redirection tests
sampsapenna Dec 31, 2021
ab4ccc2
remove default key from general build files
sampsapenna Dec 31, 2021
fc9bbd1
bump openssl 1.1.1m
sampsapenna Jan 4, 2022
df0169b
fix footer issues
sampsapenna Jan 7, 2022
18ddcdd
bump npm, lock git dependencies
sampsapenna Jan 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 145 additions & 0 deletions dockerfiles/Dockerfile-build-crypt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# Build libsodium
FROM emscripten/emsdk:latest AS SODIUM

RUN wget https://download.libsodium.org/libsodium/releases/libsodium-1.0.18-stable.tar.gz \
&& tar xvf libsodium-1.0.18-stable.tar.gz && cd libsodium-stable \
&& dist-build/emscripten.sh --sumo

# Build openssl
FROM emscripten/emsdk:latest AS OPENSSL

RUN wget https://www.openssl.org/source/openssl-1.1.1l.tar.gz \
sampsapenna marked this conversation as resolved.
Show resolved Hide resolved
&& tar xvf openssl-1.1.1l.tar.gz \
sampsapenna marked this conversation as resolved.
Show resolved Hide resolved
&& cd openssl-1.1.1l \
&& emconfigure ./Configure linux-generic64 no-asm no-threads no-engine no-hw no-weak-ssl-ciphers no-dtls no-shared no-dso --prefix=/emsdk/upstream \
&& sed -i 's|^CROSS_COMPILE.*$|CROSS_COMPILE=|g' Makefile \
&& sed -i '/^CFLAGS/ s/$/ -D__STDC_NO_ATOMICS__=1/' Makefile \
&& sed -i '/^CXXFLAGS/ s/$/ -D__STDC_NO_ATOMICS__=1/' Makefile \
&& emmake make -j 2 all \
&& emmake make install

# Build libcrypt4gh
FROM emscripten/emsdk:latest AS LIBCRYPT4GH

COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/include/ /emsdk/upstream/include/
COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/lib/ /emsdk/upstream/lib/

RUN sudo apt-get update \
&& sudo apt-get upgrade -y \
&& sudo apt-get install -y autoconf

RUN git clone https://github.com/silverdaz/libcrypt4gh.git
sampsapenna marked this conversation as resolved.
Show resolved Hide resolved

# We'll skip linking libraries since emcc only produces static libraries
# Linking sodium at this point causes a linker conflict – thus cutting out $(LIBS)
RUN export EMCC_CFLAGS="-I/emsdk/upstream/include -L/emsdk/upstream/lib" \
&& export LDFLAGS="-L/emsdk/upstream/lib" \
&& cd libcrypt4gh \
&& autoreconf \
&& sed -i 's/$(LIBS) //' Makefile.in \
&& emconfigure ./configure --prefix=/emsdk/upstream \
&& emmake make \
&& emmake make install

# Build libcrypt4gh-keys
FROM emscripten/emsdk:latest AS LIBCRYPT4GHKEYS

COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/include/ /emsdk/upstream/include/
COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/lib/ /emsdk/upstream/lib/

COPY --from=OPENSSL /emsdk/upstream/include/ /emsdk/upstream/include/
COPY --from=OPENSSL /emsdk/upstream/lib/ /emsdk/upstream/lib/

RUN sudo apt-get update \
&& sudo apt-get upgrade -y \
&& sudo apt-get install -y autoconf build-essential

RUN git clone https://github.com/silverdaz/libcrypt4gh-keys.git

# We'll skip linking libraries since emcc only produces static libraries
# Linking sodium at this point causes a linker conflict – thus cutting out $(LIBS)
RUN export EMCC_CFLAGS="-I/emsdk/upstream/include -L/emsdk/upstream/lib" \
&& export LDFLAGS="-L/emsdk/upstream/lib" \
&& cd libcrypt4gh-keys \
&& autoreconf \
&& sed -i 's/$(LIBS) //' Makefile.in \
&& emconfigure ./configure --prefix=/emsdk/upstream --with-openssl=/emsdk/upstream \
&& emmake make \
&& emmake make install

# Build wasm encryption
FROM emscripten/emsdk:latest AS WASMCRYPT

COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/include/ /emsdk/upstream/include/
COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/lib/ /emsdk/upstream/lib/

COPY --from=LIBCRYPT4GH /emsdk/upstream/include/ /emsdk/upstream/include/
COPY --from=LIBCRYPT4GH /emsdk/upstream/lib/ /emsdk/upstream/lib/

COPY --from=LIBCRYPT4GHKEYS /emsdk/upstream/include/ /emsdk/upstream/include/
COPY --from=LIBCRYPT4GHKEYS /emsdk/upstream/lib/ /emsdk/upstream/lib/

COPY swift_browser_ui_frontend/wasm/ /src/

RUN export EMCC_CFLAGS="-I/emsdk/upstream/include -L/emsdk/upstream/lib" \
&& export EMCC_FORCE_STDLIBS=libc \
&& emmake make

# Build UI with encryption
FROM node:14.17.1-alpine3.12 as FRONTEND
sampsapenna marked this conversation as resolved.
Show resolved Hide resolved

RUN apk add --update \
&& apk add --no-cache build-base curl-dev linux-headers bash git\
&& rm -rf /var/cache/apk/*

COPY swift_browser_ui_frontend /root/swift_ui/swift_browser_ui_frontend

RUN cd /root/swift_ui/swift_browser_ui_frontend \
&& npm install -g [email protected] \
blankdots marked this conversation as resolved.
Show resolved Hide resolved
sampsapenna marked this conversation as resolved.
Show resolved Hide resolved
&& npm install \
&& npm run build

FROM python:3.8-alpine3.12 as BACKEND
sampsapenna marked this conversation as resolved.
Show resolved Hide resolved

RUN apk add --update \
&& apk add --no-cache build-base curl-dev linux-headers bash git \
&& apk add --no-cache libressl-dev libffi-dev rust cargo \
&& rm -rf /var/cache/apk/*

COPY requirements.txt /root/swift_ui/requirements.txt
COPY setup.py /root/swift_ui/setup.py
COPY swift_browser_ui /root/swift_ui/swift_browser_ui
COPY --from=FRONTEND /root/swift_ui/swift_browser_ui_frontend/dist /root/swift_ui/swift_browser_ui_frontend/dist
COPY --from=WASMCRYPT /src/src/libupload.js /root/swift_ui/swift_browser_ui_frontend/dist/js/libupload.js
COPY --from=WASMCRYPT /src/src/libupload.wasm /root/swift_ui/swift_browser_ui_frontend/dist/js/libupload.wasm

RUN pip install --upgrade pip && \
pip install -r /root/swift_ui/requirements.txt && \
pip install /root/swift_ui

FROM python:3.8-alpine3.12

RUN apk add --no-cache --update bash

LABEL maintainer "CSC Developers"
LABEL org.label-schema.schema-version="1.0"
LABEL org.label-schema.vcs-url="https://github.com/CSCfi/swift-browser-ui"

COPY --from=BACKEND /usr/local/lib/python3.8/ /usr/local/lib/python3.8/

COPY --from=BACKEND /usr/local/bin/gunicorn /usr/local/bin/

COPY --from=BACKEND /usr/local/bin/swift-browser-ui /usr/local/bin/

RUN mkdir -p /app

WORKDIR /app

COPY ./deploy/app.sh /app/app.sh

RUN chmod +x /app/app.sh

RUN adduser --disabled-password --no-create-home swiftui
USER swiftui

ENTRYPOINT ["/bin/sh", "-c", "/app/app.sh"]
147 changes: 147 additions & 0 deletions dockerfiles/Dockerfile-build-crypt-devel
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Build libsodium
FROM emscripten/emsdk:latest AS SODIUM

RUN wget https://download.libsodium.org/libsodium/releases/libsodium-1.0.18-stable.tar.gz \
&& tar xvf libsodium-1.0.18-stable.tar.gz && cd libsodium-stable \
&& dist-build/emscripten.sh --sumo

# Build openssl
FROM emscripten/emsdk:latest AS OPENSSL

RUN wget https://www.openssl.org/source/openssl-1.1.1l.tar.gz \
&& bash -c 'if [[ $(sha256sum < openssl-1.1.1l.tar.gz) != *$(curl https://www.openssl.org/source/openssl-1.1.1l.tar.gz.sha256)* ]]; then echo $(sha256sum < openssl-1.1.1l.tar.gz) $(curl https://www.openssl.org/source/openssl-1.1.1l.tar.gz.sha256); echo Downloaded file checksum does not match. ; exit 1; fi' \
&& tar xvf openssl-1.1.1l.tar.gz \
&& cd openssl-1.1.1l \
&& emconfigure ./Configure linux-generic64 no-asm no-threads no-engine no-hw no-weak-ssl-ciphers no-dtls no-shared no-dso --prefix=/emsdk/upstream \
&& sed -i 's|^CROSS_COMPILE.*$|CROSS_COMPILE=|g' Makefile \
&& sed -i '/^CFLAGS/ s/$/ -D__STDC_NO_ATOMICS__=1/' Makefile \
&& sed -i '/^CXXFLAGS/ s/$/ -D__STDC_NO_ATOMICS__=1/' Makefile \
&& emmake make -j 2 all \
&& emmake make install

# Build libcrypt4gh
FROM emscripten/emsdk:latest AS LIBCRYPT4GH

COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/include/ /emsdk/upstream/include/
COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/lib/ /emsdk/upstream/lib/

RUN sudo apt-get update \
&& sudo apt-get upgrade -y \
&& sudo apt-get install -y autoconf

RUN git clone https://github.com/silverdaz/libcrypt4gh.git

# We'll skip linking libraries since emcc only produces static libraries
# Linking sodium at this point causes a linker conflict – thus cutting out $(LIBS)
RUN export EMCC_CFLAGS="-I/emsdk/upstream/include -L/emsdk/upstream/lib" \
&& export LDFLAGS="-L/emsdk/upstream/lib" \
&& cd libcrypt4gh \
&& autoreconf \
&& sed -i 's/$(LIBS) //' Makefile.in \
&& emconfigure ./configure --prefix=/emsdk/upstream \
&& emmake make \
&& emmake make install

# Build libcrypt4gh-keys
FROM emscripten/emsdk:latest AS LIBCRYPT4GHKEYS

COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/include/ /emsdk/upstream/include/
COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/lib/ /emsdk/upstream/lib/

COPY --from=OPENSSL /emsdk/upstream/include/ /emsdk/upstream/include/
COPY --from=OPENSSL /emsdk/upstream/lib/ /emsdk/upstream/lib/

RUN sudo apt-get update \
&& sudo apt-get upgrade -y \
&& sudo apt-get install -y autoconf build-essential

RUN git clone https://github.com/silverdaz/libcrypt4gh-keys.git

# We'll skip linking libraries since emcc only produces static libraries
# Linking sodium at this point causes a linker conflict – thus cutting out $(LIBS)
RUN export EMCC_CFLAGS="-I/emsdk/upstream/include -L/emsdk/upstream/lib" \
&& export LDFLAGS="-L/emsdk/upstream/lib" \
&& cd libcrypt4gh-keys \
&& autoreconf \
&& sed -i 's/$(LIBS) //' Makefile.in \
&& emconfigure ./configure --prefix=/emsdk/upstream --with-openssl=/emsdk/upstream \
&& emmake make \
&& emmake make install

# Build wasm encryption
FROM emscripten/emsdk:latest AS WASMCRYPT

COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/include/ /emsdk/upstream/include/
COPY --from=SODIUM /src/libsodium-stable/libsodium-js-sumo/lib/ /emsdk/upstream/lib/

COPY --from=LIBCRYPT4GH /emsdk/upstream/include/ /emsdk/upstream/include/
COPY --from=LIBCRYPT4GH /emsdk/upstream/lib/ /emsdk/upstream/lib/

COPY --from=LIBCRYPT4GHKEYS /emsdk/upstream/include/ /emsdk/upstream/include/
COPY --from=LIBCRYPT4GHKEYS /emsdk/upstream/lib/ /emsdk/upstream/lib/

COPY swift_browser_ui_frontend/wasm/ /src/

RUN export EMCC_CFLAGS="-I/emsdk/upstream/include -L/emsdk/upstream/lib" \
&& export EMCC_FORCE_STDLIBS=libc \
&& emmake make

# Build UI with encryption
FROM node:14.17.1-alpine3.12 as FRONTEND

RUN apk add --update \
&& apk add --no-cache build-base curl-dev linux-headers bash git\
&& rm -rf /var/cache/apk/*

COPY swift_browser_ui_frontend /root/swift_ui/swift_browser_ui_frontend

RUN cd /root/swift_ui/swift_browser_ui_frontend \
&& npm install -g [email protected] \
&& npm install
RUN cd /root/swift_ui/swift_browser_ui_frontend \
&& npm run build-devel

FROM python:3.8-alpine3.12 as BACKEND

RUN apk add --update \
&& apk add --no-cache build-base curl-dev linux-headers bash git \
&& apk add --no-cache libressl-dev libffi-dev rust cargo \
&& rm -rf /var/cache/apk/*

COPY requirements.txt /root/swift_ui/requirements.txt
COPY setup.py /root/swift_ui/setup.py
COPY swift_browser_ui /root/swift_ui/swift_browser_ui
COPY --from=FRONTEND /root/swift_ui/swift_browser_ui_frontend/dist /root/swift_ui/swift_browser_ui_frontend/dist
COPY --from=WASMCRYPT /src/src/libupload.js /root/swift_ui/swift_browser_ui_frontend/dist/js/libupload.js
COPY --from=WASMCRYPT /src/src/libupload.wasm /root/swift_ui/swift_browser_ui_frontend/dist/js/libupload.wasm

RUN pip install --upgrade pip && \
pip install -r /root/swift_ui/requirements.txt
RUN pip install /root/swift_ui

FROM python:3.8-alpine3.12

RUN apk add --no-cache --update bash

LABEL maintainer "CSC Developers"
LABEL org.label-schema.schema-version="1.0"
LABEL org.label-schema.vcs-url="https://github.com/CSCfi/swift-browser-ui"

COPY --from=BACKEND /usr/local/lib/python3.8/ /usr/local/lib/python3.8/

COPY --from=BACKEND /usr/local/bin/gunicorn /usr/local/bin/

COPY --from=BACKEND /usr/local/bin/swift-browser-ui /usr/local/bin/

RUN mkdir -p /app

WORKDIR /app

COPY ./deploy/app.sh /app/app.sh

RUN chmod +x /app/app.sh

RUN adduser --disabled-password --no-create-home swiftui
USER swiftui

ENTRYPOINT ["/bin/sh", "-c", "/app/app.sh"]
7 changes: 4 additions & 3 deletions dockerfiles/Dockerfile-ui-devel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ COPY swift_browser_ui_frontend /root/swift_ui/swift_browser_ui_frontend

RUN cd /root/swift_ui/swift_browser_ui_frontend \
&& npm install -g [email protected] \
&& npm install \
&& npm install
RUN cd /root/swift_ui/swift_browser_ui_frontend \
&& npm run build-devel

FROM python:3.8-alpine3.15 as BACKEND
Expand All @@ -24,8 +25,8 @@ COPY swift_browser_ui /root/swift_ui/swift_browser_ui
COPY --from=FRONTEND /root/swift_ui/swift_browser_ui_frontend/dist /root/swift_ui/swift_browser_ui_frontend/dist

RUN pip install --upgrade pip && \
pip install -r /root/swift_ui/requirements.txt && \
pip install /root/swift_ui
pip install -r /root/swift_ui/requirements.txt
RUN pip install /root/swift_ui

FROM python:3.8-alpine3.15

Expand Down
Loading