Configurable s3 endpoint for download

[aaperis]

Related issue(s) and PR(s)
This PR closes #750.

Description
With this PR download can either be deployed as serving either unencrypted or encrypted files. The serveUnencryptedData boolean is removed as well as the logic for download to always generate an internal c4gh key-pair (that would allow the service to serve unencrypted files in case serveUnencryptedData=true).

Main changes:

• download can now be configured to only serve unencrypted files if a filepath to a c4gh private key file is provided. Internally, the code checks the validity of the provided key by using the supplied passphrase to retrieve the corresponding c4gh public key. The service will not start if this procedure fails and the service will serve unencrypted data only if the retrieved public key is non-empty. This ensures that there can be no misconfiguration by e.g. providing malfunctioning keys etc. For example, the crypt4gh library and therefore reencrypt will use any string provided to it and encrypt a file but with this mechanism we avoid such a scary scenario.
• if no filepath and passphrase are provided, then download will serve only encrypted files. This is the default behavior.

Other changes include:

• go testsuite updates and some refactoring along the way
• update of dev environment: in order to be able to test both setups at once, the docker compose files have been modified to include two instances of download listening at different ports, one serving encrypted and the other serving unencrypted files
• update of integration tests, including a separate bash script for error messages returned to the user (according to the issue description)
• update of helm chart

How to test
Integration tests pass.

[jbygdell]

Crypt4gh config already exists, there is not need to add another one since the app will only need one.

Either reuse the existing one or move everything up under the app sub heading.

[jbygdell]

Suggested change

	C4GH_PASSPHRASE=$(grep -F passphrase config.yaml | tail -1 | sed -e 's/.* //' -e 's/"//g')
	C4GH_PASSPHRASE=$(grep -F passphrase config.yaml | sed -e 's/.* //' -e 's/"//g')

Shouldn't be needed since the config file only will have one cryptgh config block

[jbygdell]

If you want to extract a specific entry using yq is the simplest solution.

yq '.app.c4gh.passphrase' config.yaml

[aaperis]

changed all similar to using yq

[aaperis]

I unresolved the initial the conversations that I set as resolved (these are answered) so that @jbygdell can take a look and resolve. According to how we decided to do things on our last meeting about PRs.

[nanjiangshu]

This is a bit confusing for providing a bad_token at the first glance.

[MalinAhlberg]

Looks like good and thorough work..!
I left some questions, but otherwise I think it looks good!

[aaperis]

I believe I have addressed all relevant comments and suggest we leave the renaming of variables for a refinement issue, especially since these are suggestions over changes made in the 1st round in response to 1st round renaming suggestions for the same variables.

[MalinAhlberg]

I've tested this manually, and the s3 endpoint seems to work as expected ⭐ . However, the files endpoint is not supported when the service is in encrypted mode, the answer is always downloading unencrypted data is not supported. Maybe that's intended..? The docs says you can provide the header Public-Key, but I cannot get that to work. I also tried the Client-Public-Key header.
If this behavior is intended (or if someone can point out what I'm doing wrong), I will approve :)

[aaperis]

I've tested this manually, and the s3 endpoint seems to work as expected ⭐ . However, the files endpoint is not supported when the service is in encrypted mode, the answer is always downloading unencrypted data is not supported. Maybe that's intended..? The docs says you can provide the header Public-Key, but I cannot get that to work. I also tried the Client-Public-Key header. If this behavior is intended (or if someone can point out what I'm doing wrong), I will approve :)

Good point @MalinAhlberg, just tried the same thing that you describe with the latest version from main and the behavior is the same. So the current behavior is at least consistent.

Not sure /files ever served encrypted data, from what I see in #695 and #761 all work for re-encryption is done on the s3 side only. For example see here. The c.Param("type") is only set to encryptedif as3` endpoint is called. Otherwise, I believe the default case in this switch statement seems to stream decrypted data, see here.

All in all, we need to discuss about this with the team.
Good catch with the docs!

[nanjiangshu]

Looks good!

[jbygdell]

Suggested change

	- name: APP_C4GH_PRIVATEKEYPATH
	value: {{ template "c4ghPath" . }}/{{ .Values.global.download.serveDecrypted.c4ghKeyFile }}
	- name: APP_C4GH_PASSPHRASE
	- name: C4GH_TRANSIENTPRIVATEKEYPATH
	value: {{ template "c4ghPath" . }}/{{ .Values.global.download.serveDecrypted.c4ghKeyFile }}
	- name: C4GH_TRANSIENTPASSPHRASE

[jbygdell]

Suggested change

	keyPath := viper.GetString("app.c4gh.privateKeyPath")
	passphrase := viper.GetString("app.c4gh.passphrase")
	keyPath := viper.GetString("c4gh.transientKeyPath")
	passphrase := viper.GetString("c4gh.transientPassphrase")

[jbygdell]

Suggested change

	if viper.GetString("app.c4gh.privateKeyPath") != "" {
	if !viper.IsSet("app.c4gh.passphrase") {
	return errors.New("app.c4gh.passphrase is not set")
	}
	c.App.Crypt4GHPrivateKey, c.App.Crypt4GHPublicKeyB64, err = GetC4GHKeys()
	if err != nil {
	return err
	}
	if viper.GetString("c4gh.transientKeyPath") != "" {
	if !viper.IsSet("c4gh.transientPassphrase") {
	return errors.New("c4gh.transientPassphrase is not set")
	}
	c.C4GH.PrivateKey, c.C4GH.PublicKeyB64, err = GetC4GHKeys()
	if err != nil {
	return err
	}

If this is done something like this, we wont end up with to much of a conflict when we merge download int the rest of sda. transient might not be the best prefix but as long as it is clear it should be fine.

tl;dr;
Create an entry in the Map struct for the C4GH keypair and remove it from the AppConfig entry above. Since the AppConfig holds the server config, something that will be shared by all our APIs in the sda folder (curently there are some duplicaitons that needs to be removed there.)

[jbygdell]

Suggested change

	viper.Set("app.c4gh.PrivateKeyPath", privateKeyFile.Name())
	viper.Set("app.c4gh.passphrase", "password")
	viper.Set("c4gh.transientKeyPath", privateKeyFile.Name())
	viper.Set("c4gh.transientPassphrase", "password")

[jbygdell]

Suggested change

	viper.Set("app.c4gh.privateKeyPath", "some/nonexistent.key")
	viper.Set("c4gh.transientKeyPath", "some/nonexistent.key")

[jbygdell]

Suggested change

	viper.Set("app.c4gh.privateKeyPath", privateKeyFilePath)
	viper.Set("app.c4gh.passphrase", "password")
	config.Config.App.Crypt4GHPrivateKey, config.Config.App.Crypt4GHPublicKeyB64, err = config.GetC4GHKeys()
	viper.Set("c4gh.transientKeyPath", privateKeyFilePath)
	viper.Set("c4gh.transientPassphrase", "password")
	config.Config.C4GH.PrivateKey, config.Config.C4GH.PublicKeyB64, err = config.GetC4GHKeys()

[jbygdell]

Suggested change

	originalServeUnencryptedDataTrigger := config.Config.App.Crypt4GHPublicKeyB64
	originalC4ghPrivateKeyFilepath := config.Config.App.Crypt4GHPrivateKey
	originalServeUnencryptedDataTrigger := config.Config.C4GH.PublicKeyB64
	originalC4ghPrivateKeyFilepath := config.Config.C4GH.PrivateKey

[jbygdell]

Suggested change

	originalServeUnencryptedDataTrigger := config.Config.App.Crypt4GHPublicKeyB64
	config.Config.App.Crypt4GHPublicKeyB64 = ""
	originalServeUnencryptedDataTrigger := config.Config.C4GH.PublicKeyB64
	config.Config.C4GH.PublicKeyB64 = ""

[jbygdell]

Suggested change

	if config.Config.App.Crypt4GHPublicKeyB64 == "" {
	if config.Config.C4GH.PublicKeyB64 == "" {

[jbygdell]

Suggested change

	if c.Param("type") != "encrypted" && config.Config.App.Crypt4GHPublicKeyB64 == "" {
	if c.Param("type") != "encrypted" && config.Config.CGH.PublicKeyB64 == "" {