Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow developers to run API in docker, and select between mutable and immutable modes #1888

Open
wants to merge 194 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
194 commits
Select commit Hold shift + click to select a range
2acfbf4
feat: Make the test environment data immutable so tests are repeatable
redaphid Apr 11, 2022
6583f5c
fix: add a tmp/ folder to .gitignore
redaphid Apr 11, 2022
4f31e5f
token state manual testing for now
redaphid Apr 11, 2022
cce6f7a
continuing to write manual tests for state
redaphid Apr 11, 2022
c4991ec
add simple, manual tests to see if state is being preserved between runs
redaphid Apr 11, 2022
56e44ae
Docker can run db-init script. It doesn't work, but it can run it
redaphid Apr 11, 2022
3e25945
fix docker image to be the same as the sql we're using
redaphid Apr 11, 2022
88eae2e
volume mount db_init script to test things out
redaphid Apr 11, 2022
025c8bf
put scripts I've been running to a script file so I don't have to rem…
redaphid Apr 11, 2022
d56ed3e
Miniflare listens in the docker container if you do this secret magic…
redaphid Apr 12, 2022
4c63f56
remove tcp from host port exposure
redaphid Apr 12, 2022
c99b45a
move the helper scripts I use for testing into a scripts/ directory
redaphid Apr 12, 2022
6d06599
fancier, more correct shebangs
redaphid Apr 12, 2022
cff7a67
working on getting environment vars into miniflare
redaphid Apr 12, 2022
3dc4cb4
make script that uses env vars in miniflare cli
redaphid Apr 12, 2022
9d2d6f4
debugging env config; get env vars from root .env file
redaphid Apr 12, 2022
24c0e47
clean start print docker-compose config for debugging. run-miniflare.…
redaphid Apr 13, 2022
2ec2e5a
send env vars from .env->compose->miniflare. lol
redaphid Apr 13, 2022
28e44ca
cat env vars from docker container to .env file
redaphid Apr 13, 2022
a8a8cca
enough env vars for miniflare to work!
redaphid Apr 13, 2022
1505c47
.env.tpl file updated to use docker-compose dns
redaphid Apr 13, 2022
8c25067
remove unneccessary port exposure as per @dashcraft's request :)
redaphid Apr 13, 2022
8152e18
prepping to execute db-sql.js directly
redaphid Apr 13, 2022
05f6fe3
revert to our hacked cli.js script so this is left in a working state
redaphid Apr 13, 2022
2ee6d63
no volume mounts; newer esbuild
redaphid Apr 13, 2022
ba5f9c2
add .js to an import
redaphid Apr 13, 2022
1e484ec
just run the sql files directly for db init
redaphid Apr 14, 2022
59c0686
working poc of constants injection via esbuild
redaphid Apr 15, 2022
0460f1a
working poc of constants injection via esbuild that *only* templates …
redaphid Apr 15, 2022
9d9adf1
merge w/main so we don't diverge too far. #this branch is living too …
redaphid Apr 18, 2022
6b2f228
move constants file into a dedicated folder, so we can scope variable…
redaphid Apr 18, 2022
232faaf
update constants import all around different files
redaphid Apr 18, 2022
c73811a
move constants.js even further away from the src/ dir to reinforce it…
redaphid Apr 18, 2022
d6ebcc1
actually move constants stuff to the server
redaphid Apr 18, 2022
248fbe7
back out of native es2020 imports because it's a nightmare
redaphid Apr 18, 2022
4e947df
require shim for the mixing and matching we have to do because of com…
redaphid Apr 18, 2022
15a5087
dockerignore dist directory. constants-injection will work in alpine.
redaphid Apr 18, 2022
7eb463a
It builds (again!). What a ride. With the latest magic-link sdk break…
redaphid Apr 18, 2022
706aa9a
name ipfs cluster consistently so other services can find it
redaphid Apr 18, 2022
ec404d6
Did you know underscores aren't allowed in URLs? Ya learn something n…
redaphid Apr 18, 2022
cbd5555
debug timing issues w/db init. Remove commented out old code
redaphid Apr 22, 2022
58df797
remove sleep from db_init entrypoint. Add an example (for me, at leas…
redaphid Apr 23, 2022
0c6cecd
latest version of supabase's postgrest client supports global fetch! …
redaphid Apr 23, 2022
0a5b247
it woooorks with the latest supabase client!
redaphid Apr 23, 2022
b9f0cd0
move constants.js back to the location in main. merge w/main
redaphid Apr 23, 2022
3a68220
volume mount api src to watch for changes
redaphid Apr 26, 2022
c476df5
watch on src/ and rebuild api works!
redaphid Apr 26, 2022
0e2f819
add db_data to .gitignore. Also: add .gitignore for package.
redaphid Apr 26, 2022
5198e65
keepin' up with main
redaphid Apr 29, 2022
07af2a9
reorganizing pre-PR. adding some basic comments
redaphid Apr 29, 2022
ce438c7
make variables for compose file, env file locations etc. because they…
redaphid Apr 29, 2022
1da42a6
fix wrangler.toml, context for docker-compose
redaphid Apr 29, 2022
049b073
clean-start->start.sh. Because I'm ambitious
redaphid Apr 29, 2022
8c61b67
start docker script.
redaphid Apr 29, 2022
154bf3b
Can't decide on what to call this script. It's ok, we're moving it an…
redaphid Apr 29, 2022
8fa8139
trying unsucessfully to use host env vars.
redaphid Apr 29, 2022
61cff27
debugging constants injection from env vars
redaphid Apr 29, 2022
ade0482
add .envrc to .gitignore for direnv (if you don't know about it, it's…
redaphid Apr 29, 2022
546ef14
organize env vars for debugging vs testing
redaphid Apr 29, 2022
c023f75
Trying different ways of organizing the constants injection while I w…
redaphid Apr 29, 2022
a6a7546
debugging constants injection
redaphid Apr 29, 2022
4e35dc0
move all service keys in docker-compose.yml to dashes, because unders…
redaphid Apr 29, 2022
95871e5
update env template file. Sort env vars where appropriate to make deb…
redaphid Apr 30, 2022
d072ef4
Merge remote-tracking branch 'origin/main' into feat/organize-docker-…
redaphid May 3, 2022
ca84fda
pull cli.js from main
redaphid May 3, 2022
6b94719
moving docker-related scripts to ./docker dir (to try to isolate from…
redaphid May 3, 2022
95cc504
make COPY statments more explicit (to avoid cli.js), fix constants-in…
redaphid May 3, 2022
161cafb
remove commented out volumes key in docker-compose
redaphid May 3, 2022
8b58e27
start-dev-persist scripts
redaphid May 3, 2022
8e8373a
update development docs
redaphid May 3, 2022
98af2e0
fix typos in .env.tpl and .gitignore
redaphid May 3, 2022
19acaea
restore package.json scripts from origin/main
redaphid May 3, 2022
100d199
remove env vars in wrangler.toml for dev. Putting al the env vars in …
redaphid May 3, 2022
d5bc754
remove out-of-date running single service example
redaphid May 3, 2022
41dcfea
yarn dev now uses the docker method. Ignore database data
redaphid May 4, 2022
c717523
make db persistence save outside the docker context to prevent folder…
redaphid May 4, 2022
4c7f1f0
explore running playwright tests in docker
redaphid May 4, 2022
0c35371
make psql give us a non-0 exit code on failure
redaphid May 4, 2022
b0d007b
allow testing user to pin
redaphid May 4, 2022
7d3a12a
expose ports for playwright testing
redaphid May 4, 2022
ed6801b
add playwright-specific db url
redaphid May 4, 2022
e18d670
expose all the ports for playwright
redaphid May 4, 2022
8c81f43
trying to match the .env.tpl file to the constants defined for playwr…
redaphid May 4, 2022
b7f913b
add functions to db init
redaphid May 4, 2022
b485471
debugging nft storing
redaphid May 4, 2022
7f7db7f
make Dockerfile image comments accurate
redaphid May 5, 2022
645ae98
Merge remote-tracking branch 'origin/main' into feat/make-api-running…
redaphid May 5, 2022
067850e
add a temporary test:docker to package.json just to test that things …
redaphid May 5, 2022
b8cb9c6
I guess Dockerfiles have a HEALTHCHECK option now. Using that for the…
redaphid May 5, 2022
372e56a
Factoring the dev-specific stuff out to a separate file to prep for t…
redaphid May 5, 2022
77c0619
fix docker-compose.persist.yml's persistent db data location and init
redaphid May 5, 2022
70798f6
small comment fix
redaphid May 5, 2022
321b746
add comment re: why the db_data had to be where it is. So I don't cha…
redaphid May 5, 2022
e7329de
stay up to date w/master. Add snippets that are helpful to me while c…
redaphid May 11, 2022
38f615b
add typescript to test dockerfile. Narrow test script to only run pla…
redaphid May 11, 2022
bdec45c
minor syntax fixes; run my modified test scripts in docker so I don't…
redaphid May 11, 2022
6d87890
use main's yarn.lock
redaphid May 11, 2022
a7aaaf4
make playwright docker tests depend on api. This + the HEALTHCHECK Do…
redaphid May 11, 2022
17259d8
remove port exposure in base docker-compose.yml to allow testing at t…
redaphid May 11, 2022
72201b6
put the constant-injection build step in the dockerfile for now
redaphid May 11, 2022
103716b
use v1.20.0 of playwright base image
yusefnapora May 13, 2022
54f92e3
add playwright run script so we can put the constants injection in th…
redaphid May 15, 2022
5013376
fix typo in test.Dockerfile
redaphid May 15, 2022
5d97c1d
add tsc to the playwright script
redaphid May 17, 2022
6c07889
playwright runs in docker!
redaphid May 18, 2022
96b545f
most of the playwright tests run in docker now!
redaphid May 18, 2022
ceefe58
tsc before bundling seems to help things
redaphid May 18, 2022
0aadacf
remove unnecessary injections, commented out code
redaphid May 19, 2022
e4fb007
use latest docker-compose syntax. 'test' in package.json now runs it …
redaphid May 19, 2022
100304a
use docker compose's profiles to simplify running different contexts …
redaphid May 19, 2022
120b8df
use latest docker compose syntax to simplify execution. NOTE: start-t…
redaphid May 19, 2022
fd5a0a5
use the test profile
redaphid May 19, 2022
56ad251
move start-dev.sh to new docker compose syntax
redaphid May 19, 2022
462fdb9
standardize start script syntax and make test stuff clear
redaphid May 19, 2022
63a2966
get exit code from playwright; this doesn't quite work as the db-init…
redaphid May 19, 2022
abd6139
return exit code from playwright in tests; other immutability improve…
redaphid May 19, 2022
2215b4d
renaming the project in the start-test.sh allows us to run both dev a…
redaphid May 19, 2022
c7097c0
prevent dev from interfering w/test, and vice versa
redaphid May 19, 2022
a11960b
make the persist environment not care if the db-init script fails
redaphid May 19, 2022
4c92b5c
make persist docker-compose do less, use docker volumes for db, etc
redaphid May 19, 2022
ff96394
remove snippets.fish
redaphid May 20, 2022
00fb9f2
add --remove-orphans to be safe to docker scripts. Debugging 'failed …
redaphid May 22, 2022
27c10f0
Merge remote-tracking branch 'origin/main' into feat/make-api-running…
redaphid May 24, 2022
820b351
attempt to get constants for tests from process.env
redaphid May 24, 2022
4dd20fc
remove playwright-specific db env from template
redaphid May 24, 2022
4534772
ipfs->ipfs-cluster in ipfs url. Docker healthcheck much shorter for t…
redaphid May 24, 2022
6a4ca59
remove console.log of process.env. add S3_* env vars to .env.tpl
redaphid May 24, 2022
9f2b293
alphabetize env vars
redaphid May 24, 2022
d47fcda
add metaplex token to constants injection
redaphid May 24, 2022
c430674
ipfs-cluster:latest !== ipfs-cluster:v1.0.0-rc4
redaphid May 24, 2022
5068b2e
use latest nft.storage client
redaphid May 24, 2022
654bc29
add test and test:watch scripts to package.json, as I've been finding…
redaphid May 24, 2022
ea4ab1e
add nodemon to devdependencies
redaphid May 24, 2022
8d29a28
undo commit of it.only
redaphid May 24, 2022
66d0a29
more test:watch fixes. Add all possible api env vars to docker-compos…
redaphid May 25, 2022
84cb321
found some more constants that needed injecting
redaphid May 25, 2022
eaa33a1
all the playwright tests pass :)
redaphid May 25, 2022
3780af0
remove now-unnecessary clean script. Remove --debug flag from cf to p…
redaphid May 25, 2022
b0095e8
see if github actions can run this new compose test setup
redaphid May 25, 2022
023ea0a
seeing if we can get Github Actions to run the whole thing
redaphid May 25, 2022
b38b38f
forgot psql is necessary; readded to Dockerfile
redaphid May 25, 2022
b3ca51f
add playwright attaching
redaphid May 25, 2022
59082db
make the .env.tpl `source`able for now, to test if CI doesn't have co…
redaphid May 25, 2022
cc7f3a4
testing sourcing the .env file
redaphid May 25, 2022
1d00208
debugging playwright tests in the Github CI environment
redaphid May 25, 2022
5646559
make the start-test return the playwright exit code; it might let Git…
redaphid May 25, 2022
b94ca4d
remove old volume mount
redaphid May 25, 2022
ea08a80
debug possible timeout while waiting for playwright exit code
redaphid May 25, 2022
3c3b573
test docker layer caching to speed up tests
redaphid May 25, 2022
e290b7c
let the db init container hang around so it doesn't trigger docker-co…
redaphid May 25, 2022
55768d6
removing docker caching on Github CI for now. It doesn't seem to spee…
redaphid May 25, 2022
010ead3
get log output of service test failures from Github CI
redaphid May 25, 2022
6a6b217
fix omission of ipfs-cluster for api dependencies; use 'docker compos…
redaphid May 25, 2022
d2ae2e2
DRY up the start-test.sh script
redaphid May 25, 2022
592dc8f
remove unnecessary cli option
redaphid May 25, 2022
55dd671
remove s3 variables from running the api in dev
redaphid May 25, 2022
372e2ad
remove unnecessary sourcing of env vars
redaphid May 25, 2022
9e22911
try using wrangler.toml from main to see if we can deploy to dev
redaphid May 25, 2022
85ce475
remove unnecessary wrangler.toml vars, and the localtunnel-based dev …
redaphid May 25, 2022
d722db0
make sure docker removes the container after run
redaphid May 25, 2022
c47119a
try the previous syntax for playwright testing and see if it works in…
redaphid May 25, 2022
35d0ef1
remove specific DATABASE_URL/DATABASE_TOKEN defines in the playwright…
redaphid May 25, 2022
2ed41c2
make start-test remove orphans; clean up test output
redaphid May 25, 2022
ed45a13
Merge remote-tracking branch 'origin/main' into feat/make-api-running…
redaphid May 26, 2022
51a9f61
make a docker-compose file for exposing all dependant service ports
redaphid May 26, 2022
a86f7f8
initial commits for running playwright scripts locally and accessing …
redaphid May 26, 2022
6e6ae7f
add port exposures for running playwright outside of docker
redaphid May 26, 2022
f2ff90b
rename local playwright script name
redaphid May 26, 2022
78494eb
add working local playwright execution against docker-compose
redaphid May 26, 2022
bfee0c3
make project name for the persistent dev environment different than t…
redaphid May 31, 2022
c33105a
merge main
yusefnapora Jun 2, 2022
55e2aba
get things working with new config setup
yusefnapora Jun 2, 2022
d242e09
Merge branch 'main' into feat/make-api-running-immutable
yusefnapora Jun 2, 2022
df55213
inject playwright config globals from process.env
yusefnapora Jun 2, 2022
dffc744
rm console logs
yusefnapora Jun 2, 2022
82c73d4
rm commented out code
yusefnapora Jun 2, 2022
4737976
now with 100% less constants injection :)
redaphid Jun 2, 2022
9929366
add a dev:clean script to clean persistence
redaphid Jun 2, 2022
3d272d8
make a healthcheck for the database, so we don't rely on `db-init`'s…
redaphid Jun 2, 2022
0a979ee
move default `yarn:test` -> local. Add `yarn:test:docker` for using d…
redaphid Jun 2, 2022
9998341
rm debug log
yusefnapora Jun 3, 2022
b043d10
Merge branch 'main' into feat/make-api-running-immutable
redaphid Jun 13, 2022
669e3e9
Fix postres Dockerfile downgrade
redaphid Jun 13, 2022
8a46de3
rename post-rest->postgrest
redaphid Jun 13, 2022
06eadfe
post-rest->postgrest, add cross-fetch stuff back into build
redaphid Jun 13, 2022
887a944
rename scripts to reflect their respective `yarn run` commands
redaphid Jun 13, 2022
d1da8c4
re-add deploy-dev to github ci
redaphid Jun 13, 2022
0970904
Merge branch 'feat/make-api-running-immutable' of github.com:nftstora…
redaphid Jun 14, 2022
a090356
re-add deploy-dev. I missed the actual key of the test name
redaphid Jun 16, 2022
bd82712
line up env vars so the change is more obvious in this file
redaphid Jun 16, 2022
ec8e6ce
line up env vars so the change is more obvious in this file
redaphid Jun 16, 2022
69c748d
use main's yarn.lock
redaphid Jun 16, 2022
9b0584c
minor whitespace change
redaphid Jun 16, 2022
6708db9
add slack constant in to the env vars approach
redaphid Jun 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rm commented out code
yusefnapora committed Jun 2, 2022

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 82c73d4f5eb3b02e73f247447f4c8a779bf22f84
1 change: 0 additions & 1 deletion packages/api/pw-test.config.cjs
Original file line number Diff line number Diff line change
@@ -62,7 +62,6 @@ const config = {
inject: [
path.join(__dirname, './scripts/node-globals.js'),
injectGlobalsTempfile.path,
// path.join(__dirname, './test/scripts/globals.js'),
],
define: {
NFT_STORAGE_VERSION: JSON.stringify('0.1.0'),