New Tutorial: Building and Manipulating Collections with Rules (#676)

* Rules tutorial. * rewrite tutorial formatting
galaxyproject · May 25, 2018 · 30f6ebb · 30f6ebb
1 parent 8ace551
commit 30f6ebb
Show file tree

Hide file tree

Showing 77 changed files with 864 additions and 1 deletion.
diff --git a/topics/introduction/images/rules/PRJNA355367_spreadsheet.png b/topics/introduction/images/rules/PRJNA355367_spreadsheet.png
diff --git a/topics/introduction/images/rules/ena_PRJDB3920_download.png b/topics/introduction/images/rules/ena_PRJDB3920_download.png
diff --git a/topics/introduction/images/rules/ena_table_download.png b/topics/introduction/images/rules/ena_table_download.png
diff --git a/topics/introduction/images/rules/rule_builder_filters.png b/topics/introduction/images/rules/rule_builder_filters.png
diff --git a/...troduction/images/rules/rules_apply_rules_example_4_10_apply_rules_inverted.png b/...troduction/images/rules/rules_apply_rules_example_4_10_apply_rules_inverted.png
diff --git a/topics/introduction/images/rules/rules_apply_rules_example_4_11_inverted.png b/topics/introduction/images/rules/rules_apply_rules_example_4_11_inverted.png
diff --git a/...introduction/images/rules/rules_apply_rules_example_4_12_apply_rules_filter.png b/...introduction/images/rules/rules_apply_rules_example_4_12_apply_rules_filter.png
diff --git a/topics/introduction/images/rules/rules_apply_rules_example_4_13_filtered.png b/topics/introduction/images/rules/rules_apply_rules_example_4_13_filtered.png
diff --git a/...images/rules/rules_apply_rules_example_4_14_apply_rules_filtered_and_nested.png b/...images/rules/rules_apply_rules_example_4_14_apply_rules_filtered_and_nested.png
diff --git a/...ntroduction/images/rules/rules_apply_rules_example_4_15_filtered_and_nested.png b/...ntroduction/images/rules/rules_apply_rules_example_4_15_filtered_and_nested.png
diff --git a/topics/introduction/images/rules/rules_apply_rules_example_4_1_input_paste.png b/topics/introduction/images/rules/rules_apply_rules_example_4_1_input_paste.png
diff --git a/topics/introduction/images/rules/rules_apply_rules_example_4_2_input_rules.png b/topics/introduction/images/rules/rules_apply_rules_example_4_2_input_rules.png
diff --git a/topics/introduction/images/rules/rules_apply_rules_example_4_3_input_ready.png b/topics/introduction/images/rules/rules_apply_rules_example_4_3_input_ready.png
diff --git a/topics/introduction/images/rules/rules_apply_rules_example_4_4_input_list.png b/topics/introduction/images/rules/rules_apply_rules_example_4_4_input_list.png
diff --git a/...introduction/images/rules/rules_apply_rules_example_4_5_apply_rules_landing.png b/...introduction/images/rules/rules_apply_rules_example_4_5_apply_rules_landing.png
diff --git a/...troduction/images/rules/rules_apply_rules_example_4_6_apply_rules_init_flat.png b/...troduction/images/rules/rules_apply_rules_example_4_6_apply_rules_init_flat.png
diff --git a/...troduction/images/rules/rules_apply_rules_example_4_7_apply_rules_add_depth.png b/...troduction/images/rules/rules_apply_rules_example_4_7_apply_rules_add_depth.png
diff --git a/topics/introduction/images/rules/rules_apply_rules_example_4_8_nested.png b/topics/introduction/images/rules/rules_apply_rules_example_4_8_nested.png
diff --git a/...oduction/images/rules/rules_apply_rules_example_4_9_apply_rules_init_nested.png b/...oduction/images/rules/rules_apply_rules_example_4_9_apply_rules_init_nested.png
diff --git a/topics/introduction/images/rules/rules_example_1_1_rules_landing.png b/topics/introduction/images/rules/rules_example_1_1_rules_landing.png
diff --git a/topics/introduction/images/rules/rules_example_1_2_paste.png b/topics/introduction/images/rules/rules_example_1_2_paste.png
diff --git a/topics/introduction/images/rules/rules_example_1_3_initial_rules.png b/topics/introduction/images/rules/rules_example_1_3_initial_rules.png
diff --git a/topics/introduction/images/rules/rules_example_1_4_filter_header.png b/topics/introduction/images/rules/rules_example_1_4_filter_header.png
diff --git a/topics/introduction/images/rules/rules_example_1_5_mapping_edit.png b/topics/introduction/images/rules/rules_example_1_5_mapping_edit.png
diff --git a/topics/introduction/images/rules/rules_example_1_6_mapping_set.png b/topics/introduction/images/rules/rules_example_1_6_mapping_set.png
diff --git a/topics/introduction/images/rules/rules_example_1_7_extension_set.png b/topics/introduction/images/rules/rules_example_1_7_extension_set.png
diff --git a/topics/introduction/images/rules/rules_example_2_1_inputs.png b/topics/introduction/images/rules/rules_example_2_1_inputs.png
diff --git a/topics/introduction/images/rules/rules_example_2_2_initial_rules.png b/topics/introduction/images/rules/rules_example_2_2_initial_rules.png
diff --git a/topics/introduction/images/rules/rules_example_2_3_rules.png b/topics/introduction/images/rules/rules_example_2_3_rules.png
diff --git a/topics/introduction/images/rules/rules_example_2_4_name.png b/topics/introduction/images/rules/rules_example_2_4_name.png
diff --git a/topics/introduction/images/rules/rules_example_3_10_regex_paired.png b/topics/introduction/images/rules/rules_example_3_10_regex_paired.png
diff --git a/topics/introduction/images/rules/rules_example_3_11_has_paired_id.png b/topics/introduction/images/rules/rules_example_3_11_has_paired_id.png
diff --git a/topics/introduction/images/rules/rules_example_3_12_swap_columns.png b/topics/introduction/images/rules/rules_example_3_12_swap_columns.png
diff --git a/topics/introduction/images/rules/rules_example_3_13_swapped_columns.png b/topics/introduction/images/rules/rules_example_3_13_swapped_columns.png
diff --git a/topics/introduction/images/rules/rules_example_3_14_paired_identifier_set.png b/topics/introduction/images/rules/rules_example_3_14_paired_identifier_set.png
diff --git a/topics/introduction/images/rules/rules_example_3_1_inputs.png b/topics/introduction/images/rules/rules_example_3_1_inputs.png
diff --git a/topics/introduction/images/rules/rules_example_3_2_initial_rules.png b/topics/introduction/images/rules/rules_example_3_2_initial_rules.png
diff --git a/topics/introduction/images/rules/rules_example_3_3_old_rules.png b/topics/introduction/images/rules/rules_example_3_3_old_rules.png
diff --git a/topics/introduction/images/rules/rules_example_3_4_regex.png b/topics/introduction/images/rules/rules_example_3_4_regex.png
diff --git a/topics/introduction/images/rules/rules_example_3_6_with_regexes.png b/topics/introduction/images/rules/rules_example_3_6_with_regexes.png
diff --git a/topics/introduction/images/rules/rules_example_3_7_removed_column.png b/topics/introduction/images/rules/rules_example_3_7_removed_column.png
diff --git a/topics/introduction/images/rules/rules_example_3_8_split_columns.png b/topics/introduction/images/rules/rules_example_3_8_split_columns.png
diff --git a/topics/introduction/images/rules/rules_example_3_9_columns_are_split.png b/topics/introduction/images/rules/rules_example_3_9_columns_are_split.png
diff --git a/topics/introduction/images/rules/rules_example_4_1_inputs.png b/topics/introduction/images/rules/rules_example_4_1_inputs.png
diff --git a/topics/introduction/images/rules/rules_example_4_2_initial_rules.png b/topics/introduction/images/rules/rules_example_4_2_initial_rules.png
diff --git a/topics/introduction/images/rules/rules_example_4_3_basic_rules.png b/topics/introduction/images/rules/rules_example_4_3_basic_rules.png
diff --git a/topics/introduction/images/rules/rules_example_4_4_sort.png b/topics/introduction/images/rules/rules_example_4_4_sort.png
diff --git a/topics/introduction/images/rules/rules_example_4_5_url.png b/topics/introduction/images/rules/rules_example_4_5_url.png
diff --git a/topics/introduction/images/rules/rules_example_4_6_url_built.png b/topics/introduction/images/rules/rules_example_4_6_url_built.png
diff --git a/topics/introduction/images/rules/rules_example_4_7_mapping_extension_and_name.png b/topics/introduction/images/rules/rules_example_4_7_mapping_extension_and_name.png
diff --git a/topics/introduction/images/rules/rules_example_4_8_source.png b/topics/introduction/images/rules/rules_example_4_8_source.png
diff --git a/topics/introduction/images/rules/rules_example_4_8_text.txt b/topics/introduction/images/rules/rules_example_4_8_text.txt
@@ -0,0 +1,53 @@
+{
+  "rules": [
+    {
+      "type": "add_filter_count",
+      "count": "1",
+      "which": "first",
+      "invert": false
+    },
+    {
+      "type": "remove_columns",
+      "target_columns": [
+        1,
+        2,
+        4,
+        5,
+        6
+      ]
+    },
+    {
+      "type": "sort",
+      "target_column": 0,
+      "numeric": false
+    },
+    {
+      "type": "add_column_regex",
+      "target_column": 0,
+      "expression": ".*",
+      "replacement": "http://www.uniprot.org/uniprot/$&.fasta"
+    }
+  ],
+  "mapping": [
+    {
+      "type": "info",
+      "columns": [
+        1
+      ]
+    },
+    {
+      "type": "list_identifiers",
+      "columns": [
+        0
+      ],
+      "editing": false
+    },
+    {
+      "type": "url",
+      "columns": [
+        2
+      ]
+    }
+  ],
+  "extension": "csfasta"
+}
diff --git a/topics/introduction/images/rules/rules_example_5_1_inputs.png b/topics/introduction/images/rules/rules_example_5_1_inputs.png
diff --git a/topics/introduction/images/rules/rules_example_5_2_source.png b/topics/introduction/images/rules/rules_example_5_2_source.png
diff --git a/topics/introduction/images/rules/rules_example_5_3_initial_rules.png b/topics/introduction/images/rules/rules_example_5_3_initial_rules.png
diff --git a/topics/introduction/images/rules/rules_example_5_4_fasta.png b/topics/introduction/images/rules/rules_example_5_4_fasta.png
diff --git a/topics/introduction/images/rules/rules_example_5_5_url.png b/topics/introduction/images/rules/rules_example_5_5_url.png
diff --git a/topics/introduction/images/rules/rules_example_5_6_presplit.png b/topics/introduction/images/rules/rules_example_5_6_presplit.png
diff --git a/topics/introduction/images/rules/rules_example_5_7_split_columns.png b/topics/introduction/images/rules/rules_example_5_7_split_columns.png
diff --git a/topics/introduction/images/rules/rules_example_5_8_split.png b/topics/introduction/images/rules/rules_example_5_8_split.png
diff --git a/topics/introduction/images/rules/rules_example_5_9_mapping.png b/topics/introduction/images/rules/rules_example_5_9_mapping.png
diff --git a/topics/introduction/images/rules/rules_example_6_1_paste.png b/topics/introduction/images/rules/rules_example_6_1_paste.png
diff --git a/topics/introduction/images/rules/rules_example_6_2_rules_landing.png b/topics/introduction/images/rules/rules_example_6_2_rules_landing.png
diff --git a/topics/introduction/images/rules/rules_example_6_3_end_of_table.png b/topics/introduction/images/rules/rules_example_6_3_end_of_table.png
diff --git a/topics/introduction/images/rules/rules_example_6_4_regex.png b/topics/introduction/images/rules/rules_example_6_4_regex.png
diff --git a/topics/introduction/images/rules/rules_example_6_5_multiple_identifiers_edit.png b/topics/introduction/images/rules/rules_example_6_5_multiple_identifiers_edit.png
diff --git a/topics/introduction/images/rules/rules_example_6_6_multiple_identifiers.png b/topics/introduction/images/rules/rules_example_6_6_multiple_identifiers.png
diff --git a/topics/introduction/images/rules/rules_example_6_7_named.png b/topics/introduction/images/rules/rules_example_6_7_named.png
diff --git a/topics/introduction/images/rules/uniprot_fasta.png b/topics/introduction/images/rules/uniprot_fasta.png
diff --git a/topics/introduction/images/rules/uniprot_fasta_url.png b/topics/introduction/images/rules/uniprot_fasta_url.png
diff --git a/topics/introduction/images/rules/uniprot_url.png b/topics/introduction/images/rules/uniprot_url.png
diff --git a/topics/introduction/metadata.yaml b/topics/introduction/metadata.yaml
@@ -298,6 +298,24 @@ material:
     key_points:
     contributors:
       - nekrut
+  -
+    title: "Rule Based Uploader"
+    type: "tutorial"
+    name: "galaxy-intro-rules"
+    zenodo_link: ""
+    workflows: no
+    galaxy_tour: no
+    hands_on: yes
+    slides: no
+    questions:
+      - "How to use the rule based uploader to create complex collections"
+    objectives:
+      - "Learn about the Rule Based Uploader"
+    time_estimation: "30 min"
+    key_points:
+    contributors:
+      - jmchilton
+      - erasche
 
 maintainers:
   - jennaj

diff --git a/topics/introduction/tutorials/galaxy-intro-rules/PRJNA355367.csv b/topics/introduction/tutorials/galaxy-intro-rules/PRJNA355367.csv
diff --git a/topics/introduction/tutorials/galaxy-intro-rules/PRJNA355367.tsv b/topics/introduction/tutorials/galaxy-intro-rules/PRJNA355367.tsv
diff --git a/topics/introduction/tutorials/galaxy-intro-rules/gen_images.sh b/topics/introduction/tutorials/galaxy-intro-rules/gen_images.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+# Usage:
+#    bash topics/introduction/tutorials/galaxy-intro-rules/gen_images.sh /path/to/galaxy
+#
+# Description:
+# This is meant to show how the images were generated more than for end user consumption
+# so things are a bit messy and sub-optimal.
+
+export CWD=`pwd`
+export THIS_DIR="$(dirname "$0")"
+
+GALAXY_ROOT="$1"
+cd $GALAXY_ROOT
+
+# in case THIS_DIR is relative go back to where we started
+make client
+GALAXY_SKIP_CLIENT_BUILD=1 GALAXY_RUN_WITH_TEST_TOOLS=1 GALAXY_CONFIG_OVERRIDE_DATABASE_CONNECTION=postgres://postgres:mysecretpassword@localhost:5432/galaxy sh run.sh &
+
+sleep 60
+
+# Set Galaxy Selenium tests to generate screenshots into a clean directory.
+rm -rf screens
+export GALAXY_TEST_SCREENSHOTS_DIRECTORY=`pwd`/screens
+
+. .venv/bin/activate
+
+# Run Galaxy Selenium tests corresponding to the examples.
+GALAXY_TEST_EXTERNAL=http://localhost:8080/ nosetests test/selenium_tests/test_uploads.py:UploadsTestCase.test_rules_example_1_datasets
+GALAXY_TEST_EXTERNAL=http://localhost:8080/ nosetests test/selenium_tests/test_uploads.py:UploadsTestCase.test_rules_example_2_list
+GALAXY_TEST_EXTERNAL=http://localhost:8080/ nosetests test/selenium_tests/test_uploads.py:UploadsTestCase.test_rules_example_3_list_pairs
+GALAXY_TEST_EXTERNAL=http://localhost:8080/ nosetests test/selenium_tests/test_uploads.py:UploadsTestCase.test_rules_example_4_accessions
+GALAXY_TEST_EXTERNAL=http://localhost:8080/ nosetests test/selenium_tests/test_uploads.py:UploadsTestCase.test_rules_example_5_matching_collections
+GALAXY_TEST_EXTERNAL=http://localhost:8080/ nosetests test/selenium_tests/test_uploads.py:UploadsTestCase.test_rules_example_6_nested_lists
+
+cd "$CWD"
+cd "$THIS_DIR/../.."
+rm -f images/rules/rules_example*
+cp "$GALAXY_ROOT"/screens/rules_example* images/rules
diff --git a/topics/introduction/tutorials/galaxy-intro-rules/tutorial.md b/topics/introduction/tutorials/galaxy-intro-rules/tutorial.md
diff --git a/topics/introduction/tutorials/processing-many-samples-at-once/tutorial.md b/topics/introduction/tutorials/processing-many-samples-at-once/tutorial.md
@@ -39,7 +39,7 @@ Here we will show Galaxy features designed to help with the analysis of large nu
 
 These datasets represent genomic DNA (enriched for mitochondria via a long range PCR) isolated from blood and cheek (buccal swab) of mother (`M117`) and her child (`M117C1`) that was sequenced on an Illumina miSeq machine as paired-read library (250-bp reads; see our [2014](http://www.pnas.org/content/111/43/15474.abstract) manuscript for **Methods**).
 
-## 1. Creating a paired dataset collection
+## 1. Creating a list of paired datasets
 
 If you imported [history]( https://test.galaxyproject.org/u/anton/h/collections-1) as described [above](https://github.com/nekrut/galaxy/wiki/Processing-many-samples-at-once#0-getting-data), your screen will look something like this: