[release testing] Visualizations not loading.

[guerler]

Describe the bug
This seems to be a re-occuring issue, visualizations are not loading properly. Observed on Main.

Galaxy Version and/or server at which you observed the bug
Galaxy Version: 22.05

To Reproduce
Steps to reproduce the behavior:

1. Select a tabular file
2. Click on the Visualization icon
3. Select a NVD3 bar diagram
4. See error

Expected behavior
Visualization should load properly.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[dannon]

Open the console and note the 404 in loading; the actual files for the viz don't exist. Exploration on main:

[g2main@galaxy-web-05 nvd3_bar]$ pwd
/cvmfs/main.galaxyproject.org/galaxy/static/plugins/visualizations/nvd3/nvd3_bar
[g2main@galaxy-web-05 nvd3_bar]$ ls -lR
.:
total 5
drwxr-xr-x. 2 cvmfs cvmfs 4096 Jun 17 10:11 static

./static:
total 2173
-rw-r--r--. 1 cvmfs cvmfs    7073 May 19  2020 logo.png
-rw-r--r--. 1 cvmfs cvmfs    8551 Jun 16 14:12 nvd3.css
-rw-r--r--. 1 cvmfs cvmfs 2208724 Feb  4 14:15 nvd3.js.map
-rw-r--r--. 1 cvmfs cvmfs      41 Feb  4 14:15 plugin_build_hash.txt
[g2main@galaxy-web-05 nvd3_bar]$ cat static/plugin_build_hash.txt
c1f7799f98f0b9d3d3b629e3a908482aa529903c

Which is weird. The build hash is from Feb 4, lining up with most of those timestamps. But the .css is newer. Walking through what might have happened, a fresh build was triggered, wrote out the css, and then failed after deleting script.js but before emitting new artifacts? I did fix a bug about 3 months ago (#13493) that would have written the client build hash even in the event of a failure, but that doesn't explain that June 17 timestamp existing and this thing not retrying a build.

For right now on main we can use the new force plugin rebuild task to clean all this up with fresh builds. I think an additional step I can take would be to, on attempted rebuild, move the previous plugin_build_hash to plugin_build_hash_previous or something (so we know when it was last actually successfully built) but don't end up in a state like this where we have a 'declared' successful build with artifacts mixed from two builds.

[mvdbeek]

OK, @dannon and I rebuilt the visualizations on cvmfs0. Was a bit tricky due to completely outdated build deps on the cvmfs machine, we finally did it using docker: https://gist.github.com/mvdbeek/b3ebf4f58080a9788a8754fa977eb3de