Adding mia examples #50

SHillman836 · 2024-07-15T08:27:33Z

So this is the newest EBI notebooks section. There were the following issues/questions that came up -

The newer getTop() mia function wasn't working. So I had to use the GetTopFeatures() method. I'm on version 1.12.0 which I believe is the latest one
Similarly, the newer rarefyAssay() function wasn't found, so I used subsampleCounts() instead.
In the section titled "Comparative metagenomics at community level: Beta diversity" you need a mountford distance matrix for the adonis2 and betadisper methods. And when using the runMDS() method, you can't then convert the matrix that method produces to a dist object as it throws an error that it's not a square matrix. Or at least I wasn't sure how to do this. So I called the vegan package directly.

By the way, this PR includes the changes that Noah had made, I corrected one thing on one of his files, but other than that assumed his PR was correct.

@TuomasBorman @antagomir

…ic diversity

…ed notebook even inside shiny proxy

… with deep-linked variables. style and docs updates.

…jupyter Fixes for running inside Shiny Proxy

…ve_metagenomics Comparative metagenomics

…BI-Metagenomics#10) * docs: update README.md [skip ci] * docs: create .all-contributorsrc [skip ci] Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

…-Metagenomics#9) * docs: update README.md [skip ci] * docs: create .all-contributorsrc [skip ci] Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com> Co-authored-by: Sandy Rogers <[email protected]>

* docs: update README.md [skip ci] * docs: update .all-contributorsrc [skip ci] Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

* docs: update README.md [skip ci] * docs: update .all-contributorsrc [skip ci] Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com> Co-authored-by: Sandy Rogers <[email protected]>

…ent (EBI-Metagenomics#17) * docs: update .all-contributorsrc [skip ci] * docs: update README.md [skip ci] Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com> Co-authored-by: Sandy Rogers <[email protected]>

…BI-Metagenomics#18) * docs: update README.md [skip ci] * docs: update .all-contributorsrc [skip ci] Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

Fixing the commit line

TuomasBorman

Unfortunately, I do not have now time to review this in detail. In general, this looks good. However, some operations are done little bit too complicated. I commented at least some of them, check the whole code if there is room to simplify. As this is overview to MGnifyR and mia, we should make the examples rather simple. By that I mean that we should use methods found in SE ecosystem if possible as they make things easier.

About mia version issue. It would be best to have the most updated version of mia so that we do not have to update function names etc in the future. Of course that depends also on the server requirements and stuff like that.

You can find info on installing development version of mia from here https://microbiome.github.io/OMA/docs/devel/pages/06_packages.html#package-installation

The easiest is to install it from GitHub. (However, installing devel version from Bioconductor is usually preferred).

src/notebooks/R Mia Examples/Fetch-Analyses-metadata-for-a-Study.qmd

.../R Mia Examples/Fetch-Analyses-metadata-for-a-Study_files/libs/bootstrap/bootstrap-icons.css

src/notebooks/R Mia Examples/_resources/mgnifyr_help.md

src/notebooks/R Mia Examples/Comparative-Metagenomics.qmd

antagomir

Some comments for starters.

src/notebooks/R Mia Examples/Fetch-Analyses-metadata-for-a-Study.qmd

src/notebooks/R Mia Examples/_resources/mgnifyr_help.md

antagomir · 2024-07-16T08:09:34Z

How does this PR relate to #49? Can we close #49 if it has been merged with this?

SHillman836 · 2024-07-16T08:12:04Z

How does this PR relate to #49? Can we close #49 if it has been merged with this?

Sorry didn't mean to close this have just reoopened it now. Yes we can close the other PR

antagomir

Ok herewego! Some more suggestions. The essential thing is to add more support for using the (Tree)SE ecosystem tools

src/notebooks/R Mia Examples/Comparative-Metagenomics.qmd

SHillman836 · 2024-07-16T08:53:43Z

Some comments for starters.

@antagomir I was wondering - the "Fetch analyses metadata for a study" file that I pulled from the other PR - should I just remove it completely?

I think there was some confusion. It's basically a replica of this notebook - https://docs.mgnify.org/src/notebooks/R%20Examples/Fetch%20Analyses%20metadata%20for%20a%20Study.html

But the only difference is the last section where we convert to mia vs phyloseq. But I don't really see it being needed, because we have to do that anyway in the comparative metagenomics file.

I think Noah may have done the wrong notebook. Or maybe we're meant to do a mia version of both the comparative metagenomics guide and the fetching metadata guide? Let me know and then I'll make those grammar changes

I used what he did for section 1 of this guide - https://docs.mgnify.org/src/notebooks/R%20Examples/Comparative%20Metagenomics.html - but maybe that was the part we needed not the fetch metadata guide

SHillman836 · 2024-07-16T08:55:55Z

Ok herewego! Some more suggestions. The essential thing is to add more support for using the (Tree)SE ecosystem tools

Great thanks will do

antagomir · 2024-07-16T10:46:27Z

@TuomasBorman might add some comments but basically the notebook that you linked is based on the old version of MGnifyR. We recently rewrote the package to clean the code base and to add support for TreeSE/MAE.

Some function names might have changed, and there may be some other issues (argument names, changes in output formats how they are being processed). Make sure that the new workflow is tested with the latest Bioconductor development version of MGnifyR.

We support switching from phyloseq to TreeSE/MAE framework. Therefore it is justified to update (or rather, create a new version) of this notebook to show how to do this for TreeSE/MAE with the new upgraded MGnifyR package.

Imo. it is useful to have the two notebooks: this one that shows how to fetch data with MGnifyR and just get started with TreeSE/MAE (instead of phyloseq). And then the other one that focuses on the actual downstream analyses.

However if it starts to feel that there is too much overlap and these are better merged, we could think about that.

SHillman836 · 2024-07-19T09:56:25Z

@antagomir - I've just pushed a full updated - should be very close to finished now, it's quite polished. I'm sure there'll be a few changes though.

antagomir

Good! Just some remarks and suggestions still.

src/notebooks/R Mia Examples/Fetch-Analyses-metadata-for-a-Study.qmd

src/notebooks/R Mia Examples/Comparative-Metagenomics.qmd

antagomir · 2024-07-22T12:49:03Z

src/notebooks/R Mia Examples/Comparative-Metagenomics.qmd

+# Prepare the cross-validation data frame.
+# Transpose the assay matrix to have samples as rows and features (taxa) as columns
+assay <- t(assay)
+
+# Convert the transposed matrix to a data frame for easier manipulation and analysis
+df <- as.data.frame(assay)
+
+# Extract the geographic location labels from the colData of the tse object
+labels <- colData(filtered_tse)$sample_geographic.location..country.and.or.sea.region.
+
+# Simplify the labels to make them valid R variable names
+labels <- gsub(" ", "_", labels)
+labels <- gsub("[:.]", "", labels)
+
+# Convert the geographic location labels to a factor (categorical variable)
+labels <- as.factor(labels)
+
+# Add the geographic location labels as a new column in the data frame
+df$geo_location <- labels


Could you use mia::meltSE() for the same outcome?

meltSE puts data into long format. If I read this correctly, the data here is in wide format (there are as many columns as there are features)

meltSE would be more standard for this given purpose?

should I definitely use meltSE and convert to long format? I think it adds a few extra steps - the original may be easier

I thought that it would help to make this shorter?

doesn't mikropml take wide format data though?

src/notebooks/R Mia Examples/Fetch-Analyses-metadata-for-a-Study.qmd

TuomasBorman

Nice work!

notebooks.Rproj

dependencies/conda

.Rprofile

src/notebooks/R Mia Examples/_resources/mgnifyr_help.md

src/notebooks/R Mia Examples/Fetch-Analyses-metadata-for-a-Study.qmd

src/notebooks/R Mia Examples/Comparative-Metagenomics.qmd

TuomasBorman · 2024-07-23T21:26:37Z

src/notebooks/R Mia Examples/Comparative-Metagenomics.qmd

+# Prepare the cross-validation data frame.
+# Transpose the assay matrix to have samples as rows and features (taxa) as columns
+assay <- t(assay)
+
+# Convert the transposed matrix to a data frame for easier manipulation and analysis
+df <- as.data.frame(assay)
+
+# Extract the geographic location labels from the colData of the tse object
+labels <- colData(filtered_tse)$sample_geographic.location..country.and.or.sea.region.
+
+# Simplify the labels to make them valid R variable names
+labels <- gsub(" ", "_", labels)
+labels <- gsub("[:.]", "", labels)
+
+# Convert the geographic location labels to a factor (categorical variable)
+labels <- as.factor(labels)
+
+# Add the geographic location labels as a new column in the data frame
+df$geo_location <- labels


meltSE puts data into long format. If I read this correctly, the data here is in wide format (there are as many columns as there are features)

antagomir

good! some remarks..

src/notebooks/R Mia Examples/Comparative-Metagenomics.qmd

src/notebooks/R Mia Examples/Comparative-Metagenomics.rmarkdown

antagomir

Just change from qmd to ipynb as discussed in #49

TuomasBorman · 2024-09-16T18:18:25Z

notebooks.Rproj

This file should be removed

SandyRogers and others added 30 commits February 7, 2022 17:32

Initial commit

bec04a9

adds docker setups for local and shinyproxy; first notebooks

d8bced8

updates container config for quay.io

a757cc1

updates R notebooks: cheat sheet; output removal; cross-study taxonom…

2c8c1c7

…ic diversity

use upstream jupyter/datascience-notebook layer instead of shiny-proxy's

af7a2dc

pins some dependencies for a more reproducible build

604da99

adds a custom jupyter lab extension to redirect jupyterlab to specifi…

3332498

…ed notebook even inside shiny proxy

adds support for setting ENV VARs via query params. updates notebooks…

634b9d9

… with deep-linked variables. style and docs updates.

Merge pull request EBI-Metagenomics#1 from EBI-Metagenomics/upstream-…

fcbbe13

…jupyter Fixes for running inside Shiny Proxy

cleanup of jl extension: subsume license and remove GHA

5ce5715

Adds integration tests (EBI-Metagenomics#2)

1bd99fb

adds integration status badge

0f7e0ac

bioconda SIAMCAT install

cfbdd60

Update environment.yml

91d6090

Install metagenomeseq

f02a5e4

Merge pull request EBI-Metagenomics#4 from EBI-Metagenomics/comparati…

ff0bca7

…ve_metagenomics Comparative metagenomics

Comparative metagenomics (EBI-Metagenomics#5)

b7cd231

docs: add SandyRogers as a contributor for code, example, and 3 more (E…

98aa836

…BI-Metagenomics#10) * docs: update README.md [skip ci] * docs: create .all-contributorsrc [skip ci] Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

Comparative metagenomics siamcat (EBI-Metagenomics#6)

2f61427

adds jupyter-lab extension with MGnify help (EBI-Metagenomics#12)

b4e7686

updates comparative metagenomics notebook for lib upgrades

ad64119

docs: add bebatut as a contributor for infra (EBI-Metagenomics#15)

6f9c305

* docs: update README.md [skip ci] * docs: update .all-contributorsrc [skip ci] Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

fixes all-contributors config

dea36fc

docs: add mberacochea as a contributor for ideas, code, and 2 more (E…

7dd1c3b

…BI-Metagenomics#18) * docs: update README.md [skip ci] * docs: update .all-contributorsrc [skip ci] Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

rationalizing docker images and speeding up cache population

b89fe4a

updates shinyproxy on GHA tests

c93867b

fixes shinyproxy version in tests config

7301308

SHillman836 added 4 commits July 10, 2024 11:02

added .Rdata to gitignore

6dbcc07

finished part 2

cf0c292

finished notebook draft

479e65c

Merge remote-tracking branch 'upstream/main' into adding-mia-examples

ef696f3

Fixing the commit line

TuomasBorman requested changes Jul 15, 2024

View reviewed changes

TuomasBorman reviewed Jul 15, 2024

View reviewed changes

src/notebooks/R Mia Examples/Comparative-Metagenomics.qmd Outdated Show resolved Hide resolved

TuomasBorman reviewed Jul 15, 2024

View reviewed changes

src/notebooks/R Mia Examples/Comparative-Metagenomics.qmd Outdated Show resolved Hide resolved

TuomasBorman reviewed Jul 15, 2024

View reviewed changes

src/notebooks/R Mia Examples/Comparative-Metagenomics.qmd Outdated Show resolved Hide resolved

TuomasBorman reviewed Jul 15, 2024

View reviewed changes

src/notebooks/R Mia Examples/Comparative-Metagenomics.qmd Outdated Show resolved Hide resolved

antagomir reviewed Jul 16, 2024

View reviewed changes

SHillman836 closed this Jul 16, 2024

SHillman836 reopened this Jul 16, 2024

antagomir suggested changes Jul 16, 2024

View reviewed changes

updated changes

7a5b6db

antagomir suggested changes Jul 22, 2024

View reviewed changes

TuomasBorman requested changes Jul 23, 2024

View reviewed changes

updated changes

806fa4d

antagomir suggested changes Jul 24, 2024

View reviewed changes

updated changes

88d8d75

antagomir mentioned this pull request Jul 25, 2024

added mgnifyR workflow that uses mia #49

Closed

antagomir suggested changes Jul 25, 2024

View reviewed changes

changed file format

0dca210

TuomasBorman reviewed Sep 16, 2024

View reviewed changes

notebooks.Rproj Outdated

Copy link

TuomasBorman Sep 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file should be removed

Delete notebooks.Rproj

994e0f1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding mia examples #50

Adding mia examples #50

SHillman836 commented Jul 15, 2024 •

edited

Loading

TuomasBorman left a comment

antagomir left a comment

antagomir commented Jul 16, 2024

SHillman836 commented Jul 16, 2024 •

edited

Loading

antagomir left a comment

SHillman836 commented Jul 16, 2024 •

edited

Loading

SHillman836 commented Jul 16, 2024

antagomir commented Jul 16, 2024

SHillman836 commented Jul 19, 2024

antagomir left a comment

antagomir Jul 22, 2024

TuomasBorman Jul 23, 2024

antagomir Jul 24, 2024

SHillman836 Jul 24, 2024 •

edited

Loading

antagomir Jul 24, 2024

SHillman836 Jul 24, 2024 •

edited

Loading

TuomasBorman left a comment

TuomasBorman Jul 23, 2024

antagomir left a comment

antagomir left a comment

TuomasBorman Sep 16, 2024

Adding mia examples #50

Are you sure you want to change the base?

Adding mia examples #50

Conversation

SHillman836 commented Jul 15, 2024 • edited Loading

TuomasBorman left a comment

Choose a reason for hiding this comment

antagomir left a comment

Choose a reason for hiding this comment

antagomir commented Jul 16, 2024

SHillman836 commented Jul 16, 2024 • edited Loading

antagomir left a comment

Choose a reason for hiding this comment

SHillman836 commented Jul 16, 2024 • edited Loading

SHillman836 commented Jul 16, 2024

antagomir commented Jul 16, 2024

SHillman836 commented Jul 19, 2024

antagomir left a comment

Choose a reason for hiding this comment

antagomir Jul 22, 2024

Choose a reason for hiding this comment

TuomasBorman Jul 23, 2024

Choose a reason for hiding this comment

antagomir Jul 24, 2024

Choose a reason for hiding this comment

SHillman836 Jul 24, 2024 • edited Loading

Choose a reason for hiding this comment

antagomir Jul 24, 2024

Choose a reason for hiding this comment

SHillman836 Jul 24, 2024 • edited Loading

Choose a reason for hiding this comment

TuomasBorman left a comment

Choose a reason for hiding this comment

TuomasBorman Jul 23, 2024

Choose a reason for hiding this comment

antagomir left a comment

Choose a reason for hiding this comment

antagomir left a comment

Choose a reason for hiding this comment

TuomasBorman Sep 16, 2024

Choose a reason for hiding this comment

SHillman836 commented Jul 15, 2024 •

edited

Loading

SHillman836 commented Jul 16, 2024 •

edited

Loading

SHillman836 commented Jul 16, 2024 •

edited

Loading

SHillman836 Jul 24, 2024 •

edited

Loading

SHillman836 Jul 24, 2024 •

edited

Loading