Merge pull request #4606 from galaxyproject/hale-mercat

Add variables for usegalaxy.star servers
galaxyproject · Dec 20, 2023 · 4b9d087 · 4b9d087
2 parents 4995efe + 229669b
commit 4b9d087
Show file tree

Hide file tree

Showing 25 changed files with 120 additions and 74 deletions.
diff --git a/Makefile b/Makefile
@@ -233,6 +233,9 @@ _site/%/slides_CAT_ES.pdf: _site/%/slides_CAT_ES.html
 video: ## Build all videos
 	bash bin/ari-make.sh
 
+metadata/public-server-tools.json:
+	python ./bin/supported-fetch.py
+
 annotate: ## annotate the tutorials with usable Galaxy instances
 	${ACTIVATE_ENV} && \
 	wget https://github.com/hexylena/toolshed-version-database/raw/main/guid-rev.json -O metadata/toolshed-revisions.json && \

diff --git a/_layouts/workflow-list.html b/_layouts/workflow-list.html
@@ -48,9 +48,10 @@ <h6 class="card-subtitle mb-2 text-muted">
   </button>
   <div class="dropdown-menu">
 			<a href="{{ site.url }}{{ site.baseurl }}/{{ workflow.path }}" class="dropdown-item">Download Galaxy Workflow.ga</a>
-			<a href="https://usegalaxy.eu/workflows/trs_import?run_form=true&trs_url={{ workflow.trs_endpoint }}" class="dropdown-item hide-when-galaxy-proxy-active">UseGalaxy.eu</a>
-			<a href="https://usegalaxy.org/workflows/trs_import?run_form=true&trs_url={{ workflow.trs_endpoint }}" class="dropdown-item hide-when-galaxy-proxy-active">UseGalaxy.org</a>
-			<a href="https://usegalaxy.org.au/workflows/trs_import?run_form=true&trs_url={{ workflow.trs_endpoint }}" class="dropdown-item hide-when-galaxy-proxy-active">UseGalaxy.org.au</a>
+			{% assign servers = list_usegalaxy_servers_shuffle %}
+			{% for server in servers %}
+			<a href="{{ server.url }}/workflows/trs_import?run_form=true&trs_url={{ workflow.trs_endpoint }}" class="dropdown-item hide-when-galaxy-proxy-active">{{ server.human }}</a>
+			{% endfor %}
 			<a href="https://my.galaxy.training/?path=/workflows/trs_import%3Frun_form=true%26trs_url={{ workflow.trs_endpoint }}"
 			   class="dropdown-item hide-when-galaxy-proxy-active">Import to another server (≥23.0+ only!)</a>
   </div>

diff --git a/_plugins/gtn.rb b/_plugins/gtn.rb
@@ -10,6 +10,7 @@
 require './_plugins/gtn/scholar'
 require './_plugins/gtn/supported'
 require './_plugins/gtn/toolshed'
+require './_plugins/gtn/usegalaxy'
 require './_plugins/jekyll-topic-filter'
 require 'time'
 
@@ -375,6 +376,14 @@ def get_version_number(page)
       Gtn::ModificationTimes.obtain_modification_count(page['path'])
     end
 
+    def list_usegalaxy_servers(_site)
+      Gtn::Usegalaxy.servers.map { |x| x.transform_keys(&:to_s) }
+    end
+
+    def list_usegalaxy_servers_shuffle(_site)
+      Gtn::Usegalaxy.servers.map { |x| x.transform_keys(&:to_s) }.shuffle
+    end
+
     def topic_name_from_page(page, site)
       if page.key? 'topic_name'
         site.data[page['topic_name']]['title']

diff --git a/_plugins/gtn/supported.rb b/_plugins/gtn/supported.rb
@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require './_plugins/gtn/usegalaxy'
+
 module Gtn
   # Handle tool support queries
   module Supported
@@ -21,11 +23,11 @@ def self.calculate(data, tool_list)
       if data.nil? || data.empty? || tool_list.empty? || tool_list.nil?
         return {
           'exact' => [],
-          'inexact' => [
-            { 'name' => 'UseGalaxy.eu', 'url' => 'https://usegalaxy.eu', 'usegalaxy' => true },
-            { 'name' => 'UseGalaxy.org', 'url' => 'https://usegalaxy.org', 'usegalaxy' => true },
-            { 'name' => 'UseGalaxy.org.au', 'url' => 'https://usegalaxy.org.au', 'usegalaxy' => true }
-          ],
+          'inexact' => Gtn::Usegalaxy.servers.map do |x|
+                         x = x.transform_keys(&:to_s)
+                         x['usegalaxy'] = true
+                         x
+                       end
         }
       end
 
@@ -70,13 +72,19 @@ def self.calculate(data, tool_list)
       # generate a 'false' value when merging sets.
       inexact_support -= exact_support
 
+      usegalaxy_server_urls = Gtn::Usegalaxy.servers.map { |x| x[:url] }
+
       {
         'exact' => (exact_support || []).map do |id|
           data['servers'][id].update(
-            { 'usegalaxy' => data['servers'][id]['url'].downcase.include?('usegalaxy.').to_s }
+            { 'usegalaxy' => usegalaxy_server_urls.include?(data['servers'][id]['url']) }
           )
         end,
-        'inexact' => (inexact_support || []).map { |id| data['servers'][id] }
+        'inexact' => (inexact_support || []).map do |id|
+          data['servers'][id].update(
+            { 'usegalaxy' => usegalaxy_server_urls.include?(data['servers'][id]['url']) }
+          )
+        end
       }
     end
   end

diff --git a/_plugins/gtn/usegalaxy.rb b/_plugins/gtn/usegalaxy.rb
@@ -0,0 +1,15 @@
+module Gtn
+  # Data about the UseGalaxy.* servers
+  module Usegalaxy
+    @@SERVERS = [
+      { name: 'UseGalaxy.eu', url: 'https://usegalaxy.eu', id: 'eu', human: 'Galaxy Europe' },
+      { name: 'UseGalaxy.org', url: 'https://usegalaxy.org', id: 'us', human: 'Galaxy Main' },
+      { name: 'UseGalaxy.org.au', url: 'https://usegalaxy.org.au', id: 'au', human: 'Galaxy Australia' },
+      { name: 'UseGalaxy.fr', url: 'https://usegalaxy.fr', id: 'fr', human: 'Galaxy France' }
+    ]
+
+    def self.servers
+      @@SERVERS
+    end
+  end
+end
diff --git a/bin/fetch-categories.rb b/bin/fetch-categories.rb
@@ -6,7 +6,7 @@
 
 # Get the list of toolcats
 def fetch_toolcats(server)
-  uri = URI.parse("#{server}")
+  uri = URI.parse(server.to_s)
   request = Net::HTTP::Get.new(uri)
   req_options = {
     use_ssl: uri.scheme == 'https',
@@ -45,16 +45,11 @@ def fetch_toolcats(server)
   values = [eu, org, aus].compact
   # values = [org].compact
 
-
   # Majority answer wins
   # set that value to toolcats[k]
   # If there is no majority, pick one.
   # print("#{k} - #{values.length} => #{values.uniq.compact.length}\n")
-  if values.length.positive?
-    toolcats[k] = values.max_by { |v| v['count'] }
-  else
-    toolcats[k] = nil
-  end
+  toolcats[k] = (values.max_by { |v| v['count'] } if values.length.positive?)
 end
 
 # Write the list to a file

diff --git a/bin/lint.rb b/bin/lint.rb
@@ -1110,7 +1110,9 @@ def self.enumerate_symlinks
   end
 
   def self.enumerate_lintable
-    enumerate_type(/bib$/) + enumerate_type(/md$/) + enumerate_type(/md$/, root_dir: 'faqs') + enumerate_type(/md$/, root_dir: 'news')
+    enumerate_type(/bib$/) + enumerate_type(/md$/) + enumerate_type(/md$/,
+                                                                    root_dir: 'faqs') + enumerate_type(/md$/,
+                                                                                                       root_dir: 'news')
   end
 
   def self.enumerate_all

diff --git a/bin/workflow-test.rb b/bin/workflow-test.rb
@@ -1,12 +1,17 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
+require './_plugins/gtn/usegalaxy'
+
 require 'open3'
 require 'json'
 
-GALAXIES = {
-  eu: { url: 'https://usegalaxy.eu', key: ENV.fetch('GALAXY_EU_KEY', 'NONE') },
-}
+GALAXIES = Gtn::Usegalaxy.servers.select { |s| s[:id] == 'eu' }.to_h do |server|
+  [
+    server[:id],
+    { url: server[:url], key: ENV.fetch("GALAXY_#{server[:id].upcase}_KEY", 'NONE') }
+  ]
+end
 
 def test_workflow(workflow_file, galaxy_id)
   directory = File.dirname(workflow_file)

diff --git a/bin/workflows-fetch.rb b/bin/workflows-fetch.rb
@@ -3,6 +3,7 @@
 require 'net/http'
 require 'uri'
 require 'yaml'
+require './_plugins/gtn/usegalaxy'
 
 def request(url)
   uri = URI.parse(url)
@@ -11,10 +12,9 @@ def request(url)
   req_options = {
     use_ssl: uri.scheme == 'https',
   }
-  response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
+  Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
     http.request(request)
   end
-  response
 end
 
 # Get the list of workflows
@@ -31,34 +31,32 @@ def fetch_workflows(server)
   end
 end
 
-def fetch_workflowhub()
-  projects = JSON.parse(request("https://workflowhub.eu/projects").body)
-  project_mapping = projects['data'].map{|p| [p['id'], p['attributes']['title']]}.to_h
+def fetch_workflowhub
+  projects = JSON.parse(request('https://workflowhub.eu/projects').body)
+  project_mapping = projects['data'].to_h { |p| [p['id'], p['attributes']['title']] }
 
-  response = request("https://workflowhub.eu/workflows?filter[workflow_type]=galaxy")
+  response = request('https://workflowhub.eu/workflows?filter[workflow_type]=galaxy')
   data = JSON.parse(response.body)
   if !data['links']['next'].nil?
-    puts "ERROR: Cannot yet handle multiple pages"
+    puts 'ERROR: Cannot yet handle multiple pages'
     exit 42
   end
   puts "INFO: Fetching #{data['data'].length} workflows from WorkflowHub"
-  data['data'].map.with_index { |w, i|
+  data['data'].map.with_index do |w, _i|
     # {"id"=>"14", "type"=>"workflows", "attributes"=>{"title"=>"Cheminformatics - Docking"}, "links"=>{"self"=>"/workflows/14"}}
     wf_info = JSON.parse(request("https://workflowhub.eu#{w['links']['self']}").body)
     creator_list = []
 
     creator0 = wf_info['data']['attributes']['creators'][0]
-    owner = ""
-    if !creator0.nil?
-      # Primary
-      creator_list.push(creator0['given_name'] + " " + creator0['family_name'])
-    else
+    if creator0.nil?
       # Other creators
       other = wf_info['data']['attributes']['other_creators']
       if !other.nil? && other.length.positive?
-        creator_list.push(wf_info['data']['attributes']['other_creators'].split(',').map{|x| x.strip})
-      else
+        creator_list.push(wf_info['data']['attributes']['other_creators'].split(',').map(&:strip))
       end
+    else
+      # Primary
+      creator_list.push("#{creator0['given_name']} #{creator0['family_name']}")
     end
     # Projects
     wf_info['data']['relationships']['projects']['data'].each do |p|
@@ -74,33 +72,30 @@ def fetch_workflowhub()
         'number_of_steps' => wf_info['data']['attributes']['internals']['steps'].length,
         'server' => 'https://workflowhub.eu',
         'id' => wf_info['data']['id'],
-        'tags' => wf_info['data']['attributes']['tags'].map{|t| t.gsub(/^name:/, '')},
+        'tags' => wf_info['data']['attributes']['tags'].map { |t| t.gsub(/^name:/, '') },
         'update_time' => wf_info['data']['attributes']['updated_at'],
       }
-    rescue
+    rescue StandardError
       r = nil
     end
     r
-  }.reject{|x| x.nil? }
+  end.compact
 end
 
-
 # Parse the response
-workflows_eu = fetch_workflows('https://usegalaxy.eu')
-puts "INFO: Fetched #{workflows_eu.length} workflows from EU"
-workflows_org = fetch_workflows("https://usegalaxy.org")
-puts "INFO: Fetched #{workflows_org.length} workflows from ORG"
-workflows_aus = fetch_workflows("https://usegalaxy.org.au")
-puts "INFO: Fetched #{workflows_aus.length} workflows from AUS"
-workflows = workflows_eu + workflows_org + workflows_aus
+workflows = Gtn::Usegalaxy.servers.map do |server|
+  workflows = fetch_workflows(server[:url])
+  puts "INFO: Fetched #{workflows.length} workflows from #{server[:name]}"
+  workflows
+end.flatten
 
 # Cleanup the list
 workflows.filter! do |w|
   w['published'] == true && w['importable'] == true && w['deleted'] == false && w['hidden'] == false
 end
 
 # Add in WFHub workflows
-workflows += fetch_workflowhub()
+workflows += fetch_workflowhub
 
 # Group by name + owner
 cleaned = workflows.group_by { |w| "#{w['name']}<WFID>#{w['owner']}" }

diff --git a/faqs/galaxy/account_create.md b/faqs/galaxy/account_create.md
@@ -9,7 +9,11 @@ contributors: [jennaj, bernandez, samanthaanjei, wm75]
 
 1. To create an account at any public Galaxy instance, choose your server from the available [list of Galaxy Platforms](https://galaxyproject.org/use/).
 
-    There are 3 main public Galaxy servers: [UseGalaxy.org](https://usegalaxy.org/) (**US**), [UseGalaxy.eu](https://usegalaxy.eu/) (**EU**), and [UseGalaxy.org.au](https://usegalaxy.org.au/) (**AU**).
+    There are several UseGalaxy servers:
+    {% assign servers = nil | list_usegalaxy_servers_shuffle %}
+    {% for server in servers %}
+    - [{{ server.name }}]({{ server.url }}) (**{{ server.id | upcase }}**)
+    {% endfor %}
 
 2. Click on **"Login or Register"** in the masthead on the server.
 

diff --git a/faqs/galaxy/support_admins.md b/faqs/galaxy/support_admins.md
@@ -13,4 +13,5 @@ If you suspect there is something wrong with the server, or would like to reques
   - **Galaxy US:** [Email](mailto:[email protected]), [Gitter channel](https://gitter.im/galaxyproject/Lobby)
   - **Galaxy EU:** [Gitter channel](https://gitter.im/usegalaxy-eu/Lobby), [Request TIaaS](https://usegalaxy.eu/tiaas/)
   - **Galaxy AU:** [Email](mail:[email protected]), [Request a tool](https://request.usegalaxy.org.au/), [Request Data Quota](https://docs.google.com/forms/d/e/1FAIpQLSeiw6ajmkezLCwbXc3OFQEU3Ai9hGnBd967u9YbQ8ANPgvatA/viewform)
+  - **Galaxy FR:** [Request TIaaS](https://usegalaxy.fr/tiaas/)
   - Other Galaxy servers? Check the homepage for more information.
diff --git a/faqs/gtn/contributors_getting_started.md b/faqs/gtn/contributors_getting_started.md
@@ -15,7 +15,7 @@ Below is a checklist of things to look out for to help you get started. If you f
 Basic:
 
 - **Test** the tutorial on a running Galaxy instance
-   - For example [Galaxy Main](https://usegalaxy.org), [Galaxy Europe](https://usegalaxy.eu), or [Galaxy Australia](https://usegalaxy.org.au)
+   - For example {% assign servers = list_usegalaxy_servers_shuffle %}{% for server in servers %}[{{ server.name }}]({{ server.url }}){% if forloop.last %}{% else %}, {% endif %}{% endfor %}
    - Report any issues you run into
 - **Language** editing
   - Fix spelling and grammar mistakes

diff --git a/faqs/gtn/contributors_tours.md b/faqs/gtn/contributors_tours.md
@@ -9,7 +9,7 @@ contributors: [shiltemann]
 Perhaps you've been asked to review an interactive tour, or maybe you just want to try one out. The easiest way to run an interactive tour is to use the [Tour builder](https://tailordev.fr/blog/2017/07/19/the-galaxy-tour-builder-extension/) browser extension.
 
 1. Install the Tour Builder extension to your browser ([Chrome Web Store](https://chrome.google.com/webstore/detail/galaxy-tour-builder/mdfbapknmcpnbmggahhaegehbbbmhmgg), [Firefox add-on](https://addons.mozilla.org/en-US/firefox/addon/galaxy-tour-builder/)).
-2. Navigate to a Galaxy instance supporting the tutorial. To find which Galaxy instances support each tutorial, please see the dropdown menu next to the tutorial on the training website. Using one of the usegalaxy.\* instances ([Galaxy Main](https://usegalaxy.org), [Galaxy Europe](https://usegalaxy.eu), or [Galaxy Australia](https://usegalaxy.org.au)
+2. Navigate to a Galaxy instance supporting the tutorial. To find which Galaxy instances support each tutorial, please see the dropdown menu next to the tutorial on the training website. Using one of the usegalaxy.\* instances ({% assign servers = list_usegalaxy_servers_shuffle %}{% for server in servers %}[{{ server.name }}]({{ server.url }}){% if forloop.last %}{% else %}, {% endif %}{% endfor %})
 ) is usually a good bet.
 3. Start the Tour Builder plugin by clicking on the icon in your browser menu bar
 4. Copy the contents of the `tour.yaml` file into the Tour builder editor window

diff --git a/short/galaxy.html b/short/galaxy.html
@@ -25,9 +25,10 @@ <h2 class="preferred-set">Go!</h2>
 			<details id="set-preferred" open>
 				<summary>Select your Preferred Galaxy Server</summary>
 				<p class="card-text">
-					<a class="clickthrough btn btn-primary" href="https://usegalaxy.eu">UseGalaxy.eu 🌍🇪🇺</a>
-					<a class="clickthrough btn btn-primary" href="https://usegalaxy.org">UseGalaxy.org 🌎🇺🇸</a>
-					<a class="clickthrough btn btn-primary" href="https://usegalaxy.org.au">UseGalaxy.org.au 🌏</a>
+					{% assign servers = list_usegalaxy_servers_shuffle %}
+					{% for server in servers %}
+					<a class="clickthrough btn btn-primary" href="{{ server.url }}">{{ server.human }} ({{ server.id | upcase }})</a>
+					{% endfor %}
 					<button id="set-other" type="button" class="btn btn-secondary" data-toggle="modal" data-target="#exampleModal">Other</button>
 				</p>
 			</details>

diff --git a/topics/admin/tutorials/introduction/slides.html b/topics/admin/tutorials/introduction/slides.html
@@ -33,8 +33,10 @@
 
 # Where can Galaxy run?
 
+{% assign servers = list_usegalaxy_servers_shuffle %}
+
 * Cloud (SaaS)
-  - A usegalaxy.* site [Main](https://usegalaxy.org) [Europe](https://usegalaxy.eu) [Australia](https://usegalaxy.org.au/)
+  - A usegalaxy.* site: {% for server in servers %}[{{ server.human }}]({{ server.url }}){% if forloop.last %}{% else %}, {% endif %}{% endfor %}
   - [Public Galaxy Servers](https://galaxyproject.org/use)
   - [Amazon EC2 or MS Azure](https://launch.usegalaxy.org/)
   - Semi-private cloud (e.g.: [NeCTAR](https://nectar.org.au/), [Jetstream](http://jetstream-cloud.org/))

diff --git a/topics/galaxy-interface/tutorials/ncbi-sarf/tutorial.md b/topics/galaxy-interface/tutorials/ncbi-sarf/tutorial.md
@@ -104,12 +104,13 @@ We will import the JSON files into Galaxy to query them directory, however the f
 >
 {: .comment}
 
+{% assign servers = list_usegalaxy_servers_shuffle %}
 
 > <hands-on-title>Loading SRA Aligned Read Format (SARF) Object Metadata URLs into Galaxy</hands-on-title>
 >
 >This step needs to be repeated at the beginning of an analysis to refresh the metadata to the latest daily version.
 >
-> 1. Go to your Galaxy instance of choice such as one of the [usegalaxy.org](https://usegalaxy.org/), [usegalaxy.eu](https://usegalaxy.eu), [usegalaxy.org.au](https://usegalaxy.org.au) or any other.
+> 1. Go to your Galaxy instance of choice such as one of the {% for server in servers %}[{{ server.name }}]({{ server.url }}){% if forloop.last %}{% else %}, {% endif %}{% endfor %} or any other.
 >
 > 2. Create a new history
 >

diff --git a/topics/introduction/tutorials/galaxy-intro-ngs-data-managment/tutorial.md b/topics/introduction/tutorials/galaxy-intro-ngs-data-managment/tutorial.md
@@ -28,8 +28,6 @@ contributors:
 
 In this section we will look at practical aspects of manipulation of next-generation sequencing data. We will start with the FASTQ format produced by most sequencing machines and will finish with the SAM/BAM format representing mapped reads. The cover image above shows a screen dump of a SAM dataset.
 
-# Introduction to sequencing data
-
 ## FASTQ manipulation and quality control
 
 [FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) is not a very well defined format. In the beginning various manufacturers of sequencing instruments were free to interpret FASTQ as they saw fit, resulting in a multitude of FASTQ flavors. This variation stemmed primarily from different ways of encoding quality values as described [on the Wikipedia article for FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) (below you will find an explanation of quality scores and their meaning). Today, the [FASTQ Sanger](https://www.ncbi.nlm.nih.gov/pubmed/20015970) version of the format is considered to be the standard form of FASTQ. Galaxy is using FASTQ Sanger as the only legitimate input for downstream processing tools and provides [a number of utilities for converting FASTQ files](https://www.ncbi.nlm.nih.gov/pubmed/20562416) into this form (see **FASTQ Quality Control** section of Galaxy tools).
@@ -73,7 +71,7 @@ It is common to prepare pair-end and mate-pair sequencing libraries. This is hig
 
 Thus in both cases (paired-end and mate-pair) a single physical piece of DNA (or RNA in the case of RNA-seq) is sequenced from two ends and so generates two reads. These can be represented as separate files (two FASTQ files with first and second reads) or a single file were reads for each end are interleaved. Here are examples:
 
-#### Two single files
+### Two single files
 
 File 1
 
@@ -105,7 +103,7 @@ CACTACCGGGGTATCTAATCCTGTTCGCTCCCCACGCTTTCGTCCATC
 > Note that read IDs are **identical** in two files and they are listed in **the same** order. In some cases read IDs in the first and second file may be appended with `/1` and `/2` tags, respectively.
 {: .comment}
 
-#### Interleaved file
+### Interleaved file
 
 ```
 @1/1
@@ -448,7 +446,7 @@ Now that we have downloaded this file we can go to a Galaxy instance and start p
 
 > <hands-on-title>Upload `SraRunInfo.csv` file into Galaxy</hands-on-title>
 >
-> 1. Go to your Galaxy instance of choice such as one of the [usegalaxy.org](https://usegalaxy.org/), [usegalaxy.eu](https://usegalaxy.eu), [usegalaxy.org.au](https://usegalaxy.org.au) or any other. (This tutorial uses usegalaxy.org).
+> 1. Go to your Galaxy instance of choice such as one of the [UseGalaxy.* instances](https://galaxyproject.org/usegalaxy/) or any other. (This tutorial uses usegalaxy.org).
 > 1. Click *Upload Data* button:
 > ![Data upload button](../../images/upload_button.png)
 > 1. In the dialog box that would appear click "*Choose local files*" button: