[20.09] pluggable infrastructure for file uploads (dropbox) broken for default users

[bgruening]

In #9888 a pluggable infrastructure for file uploads has been added. If this is setup for Dropbox, with credentials from the user-preferences, this works great for people that entered credentials. For people that have not put in credentials the general upload via URL is not working anymore and results in the following traceback.

Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: Traceback (most recent call last):
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/util/__init__.py", line 1040, in unicodify
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: value = str(value)
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/venv/lib64/python3.6/site-packages/Cheetah/Template.py", line 1053, in __unicode__
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: return getattr(self, mainMethName)()
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "cheetah_DynamicallyCompiledCheetahTemplate_1604111380_182973_40258.py", line 86, in respond
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: KeyError: 'dropbox|access_token'
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: During handling of the above exception, another exception occurred:
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: Traceback (most recent call last):
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/util/__init__.py", line 1042, in unicodify
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: value = str(value)
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/venv/lib64/python3.6/site-packages/Cheetah/Template.py", line 1053, in __unicode__
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: return getattr(self, mainMethName)()
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "cheetah_DynamicallyCompiledCheetahTemplate_1604111380_182973_40258.py", line 86, in respond
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: KeyError: 'dropbox|access_token'
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: During handling of the above exception, another exception occurred:
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: Traceback (most recent call last):
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 236, in prepare_job
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: job_wrapper.prepare()
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/jobs/__init__.py", line 1112, in prepare
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: self.command_line, self.extra_filenames, self.environment_variables = tool_evaluator.build()
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/tools/evaluation.py", line 450, in build
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: raise e
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/tools/evaluation.py", line 446, in build
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: self.__build_config_files()
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/tools/evaluation.py", line 511, in __build_config_files
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: config_text, is_template = self.__build_config_file_text(content)
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/tools/evaluation.py", line 600, in __build_config_file_text
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: file_sources_dict = self.app.file_sources.to_dict(for_serialization=True, user_context=user_context)
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/files/__init__.py", line 144, in to_dict
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: 'file_sources': self.plugins_to_dict(for_serialization=for_serialization, user_context=user_context),
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/files/__init__.py", line 138, in plugins_to_dict
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: el = file_source.to_dict(for_serialization=for_serialization, user_context=user_context)
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/files/sources/__init__.py", line 97, in to_dict
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: rval.update(self._serialization_props(user_context=user_context))
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/files/sources/_pyfilesystem2.py", line 69, in _serialization_props
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: effective_props[key] = self._evaluate_prop(val, user_context=user_context)
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/files/sources/__init__.py", line 132, in _evaluate_prop
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: rval = fill_template(prop_val, context=template_context, futurized=True)
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/util/template.py", line 127, in fill_template
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: raise first_exception or e
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/util/template.py", line 81, in fill_template
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: return unicodify(t)
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/server/lib/galaxy/util/__init__.py", line 1048, in unicodify
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: msg = "Value '{}' could not be coerced to Unicode: {}('{}')".format(value, type(e).__name__, e)
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "/opt/galaxy/venv/lib64/python3.6/site-packages/Cheetah/Template.py", line 1053, in __unicode__
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: return getattr(self, mainMethName)()
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: File "cheetah_DynamicallyCompiledCheetahTemplate_1604111380_182973_40258.py", line 86, in respond
Oct 31 10:38:52 sn04.bi.uni-freiburg.de python[1936614]: KeyError: 'dropbox|access_token'

[bgruening]

The following configuration avoids the problem.

- type: dropbox
  id: dropbox
  label: Your Dropbox Files
  doc: Your Dropbox files - configure an access token via the user preferences
  accessToken: ${user.preferences.get('dropbox|access_token')}

Please note the ${user.preferences.get('dropbox|access_token')} vs ${user.preferences['dropbox|access_token']) thanks to @mvdbeek for the idea.

Imho this is a too large problem caused by a misconfiguration and Galaxy should bail out in nicer way than aborting all uploads.
I will leave this issue open.

[mvdbeek]

Ah, even better might be

- type: dropbox
  id: dropbox
  label: Your Dropbox Files
  doc: Your Dropbox files - configure an access token via the user preferences
  accessToken: ${user.preferences.get('dropbox|access_token') if user else ''}

otherwise that may break uploads for anonymous users.

[wm75]

accessToken: ${user.preferences.get('dropbox|access_token') if user else ''}

Unfortunately, this doesn't work.
The way this gets parsed as Cheetah does not seem to be compatible with any checks for a logged in user.
Apparently, user.preferences exists even for an anonymous user, but doesn't have a get method, and the allowed syntax seems too limited to handle this situation (at least I couldn't come up with a solution).

:(

[mvdbeek]

I've finally gotten around to test this locally,
${user.preferences.get('dropbox|access_token') if $user.preferences else ''} would work, but #10747 will restore the ability to set ${user.preferences['dropbox|access_token']).