Use a Multi stage Dockerfile to eliminate the need for our own base image

[godber]

I think we should at least try to rejigger our Teraslice Dockerfile to try and eliminate the need for maintaining our own base image. I think the approach would be to use a Multistage Dockerfile (https://docs.docker.com/build/building/multi-stage/) whose first stage creates a temporary "dev image" (not pushed anywhere) that has all the node-gyp C dependencies and builds Teraslice. Then the second stage will build a smaller final image from the output of the dev image.

Roughly speaking:

• The image should be FROM one of these: https://hub.docker.com/_/node
  • Note we can start with specific versions for now, e.g. https://hub.docker.com/_/node/tags?page=1&name=18.18.2
  • I would start with bookworm, then bullseye, then alpine
  • The dev image should be bookworm, and the final image should be bookworm-slim (adjust debian release name as appropriate)
• The first stage should do "mostly this stuff":
  • https://github.com/terascope/base-docker-image/blob/master/Dockerfile
• The second stage should do "mostly this stuff":
• https://github.com/terascope/teraslice/blob/master/Dockerfile

I think we can still parameterize the Node version the same way we are now. Our main goal here is to isolate the changes as much as possible in the Dockerfile and making sure what we build passes e2e tests.

Ref:

When considering naming schemes for our Docker images in the issue below, I thought its probably worth trying out:

https://github.com/terascope/debian-base/issues/1#issuecomment-1883904854

[godber]

We attempted to roll out this change in the PR above in release:

https://github.com/terascope/teraslice/releases/tag/v0.91.0

But the resulting image showed significantly higher memory usage in large scale Kafka jobs so we rolled it back in this release:

https://github.com/terascope/teraslice/releases/tag/v0.92.0

For now this effort is on hold.