Feature jobs update (#108)

* start updating jebs * pipeline previously jobs page updated * remove old jobs page * add anchors for links from pipeline page * add anchors for the pipeline page
vestalisvirginis · Jul 25, 2024 · c1dc89c · c1dc89c
1 parent 1284e29
commit c1dc89c
Show file tree

Hide file tree

Showing 38 changed files with 417 additions and 222 deletions.
diff --git a/docs/configurations.md b/docs/configurations.md
@@ -2,6 +2,7 @@
 
 Using Dagster to run synphage pipeline allows the user to easily configure the jobs at run-time. 
 
+<a id="dir-config"></a>
 
 ## Plotting config options  
 

diff --git a/docs/images/pipeline/Job_step_1a_get_user_data.svg b/docs/images/pipeline/Job_step_1a_get_user_data.svg
diff --git a/docs/images/pipeline/Job_step_1b_download.svg b/docs/images/pipeline/Job_step_1b_download.svg
diff --git a/docs/images/pipeline/Job_step_2_make_validation.svg b/docs/images/pipeline/Job_step_2_make_validation.svg
diff --git a/docs/images/pipeline/Job_step_2_make_validation_reloads.svg b/docs/images/pipeline/Job_step_2_make_validation_reloads.svg
diff --git a/docs/images/pipeline/Job_step_3a_make_blastn.svg b/docs/images/pipeline/Job_step_3a_make_blastn.svg
diff --git a/docs/images/pipeline/Job_step_3b_make_blastp.svg b/docs/images/pipeline/Job_step_3b_make_blastp.svg
diff --git a/docs/images/pipeline/Job_step_3c_make_all_blast.svg b/docs/images/pipeline/Job_step_3c_make_all_blast.svg
diff --git a/docs/images/pipeline/Job_step_4_make_plot.svg b/docs/images/pipeline/Job_step_4_make_plot.svg
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.25.22.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.25.22.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.25.48.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.25.48.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.27.13.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.27.13.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.27.26.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.27.26.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.27.47.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.27.47.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.28.31.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.28.31.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.29.11.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.29.11.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.29.59.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.29.59.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.30.15.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.30.15.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.31.10.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.31.10.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.31.26.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.31.26.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.31.38.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.31.38.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.32.21.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.32.21.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-24 at 22.33.23.png b/docs/images/pipeline/Screenshot 2024-07-24 at 22.33.23.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-25 at 10.13.33.png b/docs/images/pipeline/Screenshot 2024-07-25 at 10.13.33.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-25 at 10.13.54.png b/docs/images/pipeline/Screenshot 2024-07-25 at 10.13.54.png
diff --git a/docs/images/pipeline/Screenshot 2024-07-25 at 10.14.41.png b/docs/images/pipeline/Screenshot 2024-07-25 at 10.14.41.png
diff --git a/docs/images/pipeline/dag_jobs.png b/docs/images/pipeline/dag_jobs.png
diff --git a/docs/images/pipeline/dag_landing_page.png b/docs/images/pipeline/dag_landing_page.png
diff --git a/docs/images/pipeline/navigating_job_logs.png b/docs/images/pipeline/navigating_job_logs.png
diff --git a/docs/images/pipeline/navigating_left_panel.png b/docs/images/pipeline/navigating_left_panel.png
diff --git a/docs/images/pipeline/step1b_launchpad.png b/docs/images/pipeline/step1b_launchpad.png
diff --git a/docs/images/pipeline/step1b_open_launchpad.png b/docs/images/pipeline/step1b_open_launchpad.png
diff --git a/docs/installation.md b/docs/installation.md
@@ -123,7 +123,7 @@ pip install --upgrade pip
 
 2. Data Input and Output
 
-    1. Data Input
+    1. Data Input  <a id="data_input_pip"></a>
 
         The input data are the GenBank files located in the `INPUT_DIR`. However paths to other data location can be passed at run time for loading data from another directory.
 
@@ -135,7 +135,7 @@ pip install --upgrade pip
             `.gb`and `.gbk` are both valid extension for genbank files
 
 
-    2. Data Output
+    2. Data Output  <a id="data_output_pip"></a>
 
         All output data are located in the `OUTPUT_DIR` set by the user.  
         This directory can be reused in future runs if the user needs to process additional sequences or simply generate additional synteny diagrams. 
@@ -147,7 +147,7 @@ pip install --upgrade pip
 
 
         ???+ tip
-            The current data directory can be checked in the [config panel](jobs.md#dir-config) of the jobs.  
+            The current data directory can be checked in the [config panel](pipeline.md#dir-config) of the jobs.  
 
 
 3. Start synphage via dagster web-based interface
@@ -243,7 +243,6 @@ The following dependency needs to be installed in order to run synphage Docker I
 
 ### Run `synphage` container  <a id="run-synphage-container"></a>
 
-#### Start the container  
 
 === ":simple-docker: Docker Desktop"
     1. Start the container
@@ -266,7 +265,7 @@ The following dependency needs to be installed in order to run synphage Docker I
 
     4. Set the `Volumes`
 
-        1. Data Output  
+        1. Data Output  <a id="data_output_dd"></a>
             All output data are located in the `/data` directory of the container.  
             The output data can be copied after the run from the `/data` folder or they can be stored in a `Docker Volume` that can be mounted to a new Docker Container and reused in subsequent runs if the user needs to process additional sequences or simply generate additional synteny diagrams.
 
@@ -293,7 +292,7 @@ The following dependency needs to be installed in order to run synphage Docker I
             If no Volume is mounted to the /data directory and the user do not save the data, data will be lost.
 
 
-    5. Set the environment variables (optional)  
+    5. Set the environment variables (optional)   <a id="env_dockerdesktop"></a>
         `synphage` uses the following environment variables:  
         - `EMAIL` (optional): for connecting to the NCBI database.  
         - `API_KEY` (optional): for connecting to the NCBI database and download files.  
@@ -309,7 +308,7 @@ The following dependency needs to be installed in order to run synphage Docker I
         Your container is now running.
 
 
-    7. Import local GenBank files (optional)  
+    7. Import local GenBank files (optional)  <a id="data_input_dockerdesktop"></a>
         `/user_files` is the directory that received users' GenBank files.  
         For using locally stored GenBank files, the files can be imported or dragged and dropped (depending on your system) into the `/user_files` directory.
    
@@ -340,7 +339,7 @@ The following dependency needs to be installed in order to run synphage Docker I
 
 === ":octicons-terminal-16:  Bash"
 
-    1. Environment variables
+    1. Environment variables  <a id="env_docker"></a>
 
         `synphage` uses the following environment variables:  
             - `EMAIL` (optional): for connecting to the NCBI database.  
@@ -384,7 +383,7 @@ The following dependency needs to be installed in order to run synphage Docker I
 
     3. Set the `Volumes`
 
-        1. Data Output  
+        1. Data Output  <a id="data_output_docker"></a>
             All output data are located in the `/data` directory of the container.  
             The output data can be copied after the run from the `/data` folder or they can be stored in a `Docker Volume` that can be mounted to a new Docker Container and reused in subsequent run if the user needs to process additional sequences or simply generate additional synteny diagrams.
 
@@ -423,7 +422,7 @@ The following dependency needs to be installed in order to run synphage Docker I
             Volume names must be unique. You canot set two volumes wit the same name.
 
 
-    1. Import local GenBank files (optional)  
+    1. Import local GenBank files (optional)  <a id="data_input_docker"></a>
         `/user_files` is the directory that received users GenBank files.  
         For using locally stored GenBank files, the files can be copied into the `/user_files` directory.  
         ```bash

diff --git a/docs/jobs.md b/docs/jobs.md
diff --git a/docs/output.md b/docs/output.md
@@ -0,0 +1,39 @@
+# Synphage output data
+
+All the data generated during synphage runs are stored in the data directory. (see Data Output setup for [pip](installation.md#data_output_pip), [docker desktop](installation.md#data_output_dd) or [docker](installation.md#data_output_docker) users)  
+
+synphage's output consists of four to six main parquet files (depending if blastn and blastp were both executed) and the synteny graphic. However all the data generated by the synphage pipeline are made available in your data directory.
+
+
+## Generated data architecture
+
+The folders and files are organised as follow inside the `data directory`:  
+
+```
+.
+├── <path_to_synphage_folder>/
+│   ├── download/
+│   ├── fs/
+│   ├── genbank/
+│   ├── gene_identity/
+│   │   ├── fasta_n/
+│   │   ├── blastn_database/
+│   │   └── blastn/
+│   ├── protein_identity/
+│   │   ├── fasta_p/
+│   │   ├── blastp_database/
+│   │   └── blastp/
+│   ├── tables/
+│   │   ├── genbank_db.parquet
+│   │   ├── processed_genbank_df.parquet
+│   │   ├── blastn_summary.parquet
+│   │   ├── blastp_summary.parquet
+│   │   ├── gene_uniqueness.parquet
+│   │   └── protein_uniqueness.parquet
+│   ├── sequences.csv
+│   └── synteny/
+│      ├── colour_table.parquet
+│      ├── synteny_graph.png
+│      └── synteny_graph.svg
+└── ...
+```
diff --git a/docs/phages.md b/docs/phages.md
@@ -21,20 +21,20 @@ To get familiar with `synphage` capabilities, you can reproduce the step-by-step
         pip install synphage
         dagster dev -h 0.0.0.0 -p 3000 -m synphage
         ```
-        For more details, see [installation instruction](https://vestalisvirginis.github.io/synphage/installation/#via-pip) or [how to run the software](https://vestalisvirginis.github.io/synphage/installation/#pull-synphage-image).
+        For more details, see [installation instruction](installation.md#pip-install) or [how to run the software](installation.md/#run-synphage-pip).
     === "docker"
         ``` bash
         docker pull vestalisvirginis/synphage:<tag>
         docker run --rm --name my-synphage-container -p 3000 vestalisvirginis/synphage:<tag>
         ```
-        For more details, see [installation instruction](https://vestalisvirginis.github.io/synphage/installation/#via-docker) or [how to run the software](https://vestalisvirginis.github.io/synphage/installation/#pull-synphage-image).
+        For more details, see [installation instruction](installation.md/#docker-install) or [how to run the software](installation.md/#run-synphage-container).
     === "Docker Desktop"
         <iframe width="560" height="315"
         src="../images/phages/docker_desktop_movie.mp4" 
         frameborder="0" 
         allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" 
         allowfullscreen></iframe>  
-        For more details, see [installation instruction](https://vestalisvirginis.github.io/synphage/installation/#via-docker) or [how to run the software](https://vestalisvirginis.github.io/synphage/installation/#pull-synphage-image).
+        For more details, see [installation instruction](installation.md/#docker-install) or [how to run the software](installation.md/#run-synphage-container).
 
 
 <figure markdown="span">
@@ -102,10 +102,10 @@ When the job finishes its execution, we can move to `step 2` to run the validati
 </figure>
 
 
-### Step 2: Run data quality checks on the dataset
+### Step 2: Run data quality checks on the dataset  <a id="step2-phage-example"></a>  
 
 ???+ note
-    For more detailed information on this step, check the [Validation](jobs.md#step2-validation-job).  
+    For more detailed information on this step, check the [Validation](pipeline.md#step2-validation-job).  
 
 Go to Dagster_home -> Jobs -> make_validations.  
 <figure markdown="span">
@@ -144,7 +144,7 @@ The metadata attached to the last step of the validation, render an overview of
 </figure>  
 
 
-### Step 3: Run the blast
+### Step 3: Run the blast  <a id="step3-phage-example"></a>  
 
 For this example, blastn was run on the dataset.  
 
@@ -224,7 +224,7 @@ This file can be read and manipulated with any DataFrame API the user choose, su
     The unified dataset generated through the blastp pipeline is saved under `protein_uniqueness.parquet`.
 
 
-### Step 4: Generate the plot
+### Step 4: Generate the plot  <a id="step4-phage-example"></a>  
 
 Go to Dagster_home -> Jobs -> make_plot
 <figure markdown="span">
Original file line number	Diff line number	Diff line change
Expand Up		@@ -2,6 +2,7 @@

		Using Dagster to run synphage pipeline allows the user to easily configure the jobs at run-time.

		<a id="dir-config"></a>

		## Plotting config options

Expand Down