[WIP] feat(docker): enable single curl of docker-compose.yml

[sebastianliebscher]

SUMMARY

As the title says, this PR tries to address #24987

• instead of mounting the docker/ directory into the container, moved the scripts in docker/ into the docker image
• moved default (environment) variables from .env into superset_config.py

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[sebastianliebscher]

Current blocker:

• the Postgres examples-init.sh

[sfirke]

@sebastianliebscher Thank you for this! This will be a big win. What do you think is needed to unblock the examples-init.sh piece? Is there support you need or options that we should discuss?

[sebastianliebscher]

@sfirke Both, I'm not sure which is the best way to initlialize the Postgres db, without the docker/docker-entrypoint-initdb.d/examples-init.sh.

I'm thinking of the following possibilities:

1. provide another docker image tag e.g. superset:examples-db. the docker-compose-non-dev.yml would get the examples db as its own postgres service like I mentioned here Increase superset userbase by allowing upload csv out of the box #23399 (comment)
2. move the logic of examples-init.sh into the cli command superset load-examples. for this you would have to detect if the example-db is postgres
3. copy examples-init.sh into the superset-init container and add logic to run the script. to do this you would need to detect if the examples-db is postgres and the superset-init container would also need the postgres-client package

I think 3. is not a good idea as it is too much customization. 2. I am not familiar with superset load-examples. I would go for 1.

What do you think?

[sfirke]

Ooh I also like #1. Another bonus: I loaded the example data into what became my production instance and it was hard to get it out b/c it was intermingled with the metadata (unless I'm confused). If we had this paradigm you propose in #1 I could have just dropped the examples container (or dropped the data source in Superset), easy peasy.

Would the pip workflow continue as is, if we make this modification to docker?

[sfirke]

Thinking more about this, and the fact that Superset 4.0 will not support SQLite as a metadata db -- would we need a whole separate Postgres container for the examples, or just a separate db (and maybe user) in the same Postgres container?

[sebastianliebscher]

Would the pip workflow continue as is, if we make this modification to docker?

Yes, the pip workflow still uses SQLite by default.

would we need a whole separate Postgres container for the examples, or just a separate db (and maybe user) in the same Postgres container?

Ideally we should just use one Postgres service and have Superset's metadata separated from the examples, yes. This was already achieved and merged by #25003. But that requires an initialization script (examples-init-sh) to set up the other database and user. To still be able to curl and start just the single docker-compose-non-dev.yml, this init script logic has to be removed/moved without losing the separation between metadata and examples. That's what I tried to express with my comment above #25546 (comment).

The current solution in this PR uses two separate Postgres services with examples enabled by default. If a user decides not to use examples, it is as easy as deleting the examples service + volume and SUPERSET_LOAD_EXAMPLES config.

[rusackas]

Is this still needed, or should we close it out?