Akoumparouli/fix get tokenizer list #10596

akoumpa · 2024-09-24T07:38:54Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

akoumpa · 2024-09-24T07:40:29Z

once merged need to cherrypick to 24.09/r2.0.0

nemo/collections/nlp/modules/common/tokenizer_utils.py

nemo/collections/nlp/modules/common/lm_utils.py

Signed-off-by: Alexandros Koumparoulis <[email protected]>

Signed-off-by: akoumpa <[email protected]>

Signed-off-by: Alexandros Koumparoulis <[email protected]>

nemo/collections/nlp/modules/common/lm_utils.py

Signed-off-by: Alexandros Koumparoulis <[email protected]>

Signed-off-by: akoumpa <[email protected]>

nemo/collections/nlp/modules/common/lm_utils.py

github-actions · 2024-09-27T17:09:27Z

[🤖]: Hi @akoumpa 👋,

I just wanted to let you know that, you know, a CICD pipeline for this PR just finished successfully ✨

So it might be time to merge this PR or like to get some approvals 🚀

But I'm just a 🤖 so I'll leave it you what to do next.

Have a great day!

//cc @ko3n1g

github-actions · 2024-09-30T17:28:58Z

[🤖]: Hi @akoumpa 👋,

I just wanted to let you know that, you know, a CICD pipeline for this PR just finished successfully ✨

So it might be time to merge this PR or like to get some approvals 🚀

But I'm just a 🤖 so I'll leave it you what to do next.

Have a great day!

//cc @ko3n1g

* import fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * add get_tokenizer_list import test Signed-off-by: Alexandros Koumparoulis <[email protected]> * cycle import fix attempt Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix2 Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix3 Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * avoid import Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove relative imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]>

* import fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * add get_tokenizer_list import test Signed-off-by: Alexandros Koumparoulis <[email protected]> * cycle import fix attempt Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix2 Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix3 Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * avoid import Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove relative imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]>

* import fix * add get_tokenizer_list import test * cycle import fix attempt * fix2 * fix3 * Apply isort and black reformatting * avoid import * remove relative imports * Apply isort and black reformatting --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: akoumpa <[email protected]> Co-authored-by: Eric Harper <[email protected]>

* import fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * add get_tokenizer_list import test Signed-off-by: Alexandros Koumparoulis <[email protected]> * cycle import fix attempt Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix2 Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix3 Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * avoid import Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove relative imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]>

* import fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * add get_tokenizer_list import test Signed-off-by: Alexandros Koumparoulis <[email protected]> * cycle import fix attempt Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix2 Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix3 Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * avoid import Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove relative imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]> Signed-off-by: Lifu Zhang <[email protected]>

* import fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * add get_tokenizer_list import test Signed-off-by: Alexandros Koumparoulis <[email protected]> * cycle import fix attempt Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix2 Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix3 Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * avoid import Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove relative imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

* import fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * add get_tokenizer_list import test Signed-off-by: Alexandros Koumparoulis <[email protected]> * cycle import fix attempt Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix2 Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix3 Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * avoid import Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove relative imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]>

akoumpa requested review from pablo-garay and ko3n1g as code owners September 24, 2024 07:38

akoumpa added the Run CICD label Sep 24, 2024

github-actions bot added NLP CI labels Sep 24, 2024

github-advanced-security bot found potential problems Sep 24, 2024

View reviewed changes

nemo/collections/nlp/modules/common/tokenizer_utils.py Fixed Show fixed Hide fixed

ko3n1g added the r2.0.0 label Sep 24, 2024

akoumpa marked this pull request as draft September 24, 2024 08:01

akoumpa force-pushed the akoumparouli/fix_get_tokenizer_list branch from 004d5d7 to e1b2b0f Compare September 24, 2024 08:02

github-advanced-security bot found potential problems Sep 24, 2024

View reviewed changes

nemo/collections/nlp/modules/common/tokenizer_utils.py Fixed Show fixed Hide fixed

akoumpa marked this pull request as ready for review September 24, 2024 08:10

akoumpa force-pushed the akoumparouli/fix_get_tokenizer_list branch from c98c6cf to 7f9343b Compare September 24, 2024 08:18

github-actions bot added the Multi Modal label Sep 24, 2024

akoumpa force-pushed the akoumparouli/fix_get_tokenizer_list branch 2 times, most recently from 10b0774 to 86d1557 Compare September 24, 2024 08:21

github-actions bot removed the Multi Modal label Sep 24, 2024

github-advanced-security bot found potential problems Sep 24, 2024

View reviewed changes

nemo/collections/nlp/modules/common/lm_utils.py Fixed Show fixed Hide fixed

akoumpa force-pushed the akoumparouli/fix_get_tokenizer_list branch from ea42341 to 9f58b16 Compare September 24, 2024 08:30

akoumpa added Run CICD and removed Run CICD labels Sep 24, 2024

akoumpa force-pushed the akoumparouli/fix_get_tokenizer_list branch 2 times, most recently from ccc76c9 to c1fd869 Compare September 24, 2024 08:53

github-advanced-security bot found potential problems Sep 24, 2024

View reviewed changes

nemo/collections/nlp/modules/common/lm_utils.py Fixed Show fixed Hide fixed

akoumpa added 5 commits September 24, 2024 12:16

import fix

a6bcb83

Signed-off-by: Alexandros Koumparoulis <[email protected]>

add get_tokenizer_list import test

3b70d25

Signed-off-by: Alexandros Koumparoulis <[email protected]>

cycle import fix attempt

03c61fd

Signed-off-by: Alexandros Koumparoulis <[email protected]>

fix2

5e83458

Signed-off-by: Alexandros Koumparoulis <[email protected]>

fix3

80e1578

Signed-off-by: Alexandros Koumparoulis <[email protected]>

akoumpa force-pushed the akoumparouli/fix_get_tokenizer_list branch from a73edc7 to 80e1578 Compare September 24, 2024 19:16

akoumpa and others added 2 commits September 24, 2024 19:17

Apply isort and black reformatting

113c09d

Signed-off-by: akoumpa <[email protected]>

avoid import

01a8e9a

Signed-off-by: Alexandros Koumparoulis <[email protected]>

github-advanced-security bot found potential problems Sep 26, 2024

View reviewed changes

nemo/collections/nlp/modules/common/lm_utils.py Fixed Show fixed Hide fixed

remove relative imports

42b5244

Signed-off-by: Alexandros Koumparoulis <[email protected]>

akoumpa added Run CICD and removed Run CICD labels Sep 27, 2024

Apply isort and black reformatting

d745aff

Signed-off-by: akoumpa <[email protected]>

github-advanced-security bot found potential problems Sep 27, 2024

View reviewed changes

nemo/collections/nlp/modules/common/lm_utils.py Dismissed Show dismissed Hide dismissed

akoumpa added Run CICD and removed Run CICD labels Sep 27, 2024

ko3n1g approved these changes Sep 30, 2024

View reviewed changes

akoumpa merged commit 9913441 into main Sep 30, 2024
299 of 307 checks passed

akoumpa deleted the akoumparouli/fix_get_tokenizer_list branch September 30, 2024 17:32

ko3n1g mentioned this pull request Sep 30, 2024

Cherry pick Akoumparouli/fix get tokenizer list (10596) into r2.0.0 #10684

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Akoumparouli/fix get tokenizer list #10596

Akoumparouli/fix get tokenizer list #10596

akoumpa commented Sep 24, 2024

akoumpa commented Sep 24, 2024

github-actions bot commented Sep 27, 2024

github-actions bot commented Sep 30, 2024

Akoumparouli/fix get tokenizer list #10596

Akoumparouli/fix get tokenizer list #10596

Conversation

akoumpa commented Sep 24, 2024

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

akoumpa commented Sep 24, 2024

github-actions bot commented Sep 27, 2024

github-actions bot commented Sep 30, 2024