Natural language prompting for Canary #8833

pzelasko · 2024-04-05T15:32:50Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Piotr Żelasko <[email protected]>

…ontrol the effective audio vs text size (requires latest lhotse) Signed-off-by: Piotr Żelasko <[email protected]>

Signed-off-by: Piotr Żelasko <[email protected]>

Signed-off-by: w4-jinhyeonkim <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

* yttm deprecation init commit Signed-off-by: AlexGrinch <[email protected]> * removed tests Signed-off-by: AlexGrinch <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * bug fix Signed-off-by: AlexGrinch <[email protected]> * path fix Signed-off-by: AlexGrinch <[email protected]> * fixing path Signed-off-by: AlexGrinch <[email protected]> * updated tests to spm Signed-off-by: AlexGrinch <[email protected]> * updated Jenkinsfile Signed-off-by: AlexGrinch <[email protected]> * new model with spm in tests Signed-off-by: AlexGrinch <[email protected]> * yttm removed Signed-off-by: AlexGrinch <[email protected]> * updated aayn config Signed-off-by: AlexGrinch <[email protected]> --------- Signed-off-by: AlexGrinch <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Piotr Żelasko <[email protected]>

* Added copy import to rnnt_decoding.py Signed-off-by: Isaac McFadyen <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Isaac McFadyen <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Piotr Żelasko <[email protected]>

Signed-off-by: smajumdar <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

* Fixed cp HtoD sync Signed-off-by: Selvaraj Anandaraj <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Selvaraj Anandaraj <[email protected]> Co-authored-by: Selvaraj Anandaraj <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Piotr Żelasko <[email protected]>

Signed-off-by: Chen Cui <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

* Fix triton import guards Signed-off-by: Michal Futrega <[email protected]> * Update attention.py Signed-off-by: Michal Futrega <[email protected]> --------- Signed-off-by: Michal Futrega <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

Signed-off-by: Michal Futrega <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

Signed-off-by: Chen Cui <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

* enable layernorm output gathered Signed-off-by: Chen Cui <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Chen Cui <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Piotr Żelasko <[email protected]>

…pport (#8564) * POC using bucketing in transcribe_speech.py Signed-off-by: Piotr Żelasko <[email protected]> * extend to multi task aed Signed-off-by: Piotr Żelasko <[email protected]> * fixes for aed multi task text/lang field selectors Signed-off-by: Piotr Żelasko <[email protected]> * remove assert Signed-off-by: Piotr Żelasko <[email protected]> * fix Signed-off-by: Piotr Żelasko <[email protected]> * expose option for bucket buffer size Signed-off-by: Piotr Żelasko <[email protected]> * fixes, ctc support Signed-off-by: Piotr Żelasko <[email protected]> * support pre-sorting manifests in transcribe_speech.py Signed-off-by: Piotr Żelasko <[email protected]> * cleanup Signed-off-by: Piotr Żelasko <[email protected]> * reorder transcriptions back to original manifest order Signed-off-by: Piotr Żelasko <[email protected]> * remove bucketing entirely Signed-off-by: Piotr Żelasko <[email protected]> * code review changes Signed-off-by: Piotr Żelasko <[email protected]> * code review changes--amend Signed-off-by: Piotr Żelasko <[email protected]> * refactor text_field/lang_field passing Signed-off-by: Piotr Żelasko <[email protected]> * Fix reordering bug; disable presorting for multi task for now Signed-off-by: Piotr Żelasko <[email protected]> * Add support for presort + multi task model Signed-off-by: Piotr Żelasko <[email protected]> * Code reviews Signed-off-by: Piotr Żelasko <[email protected]> * Fix jenkins tests, add user-friendly error msg for canary Signed-off-by: Piotr Żelasko <[email protected]> --------- Signed-off-by: Piotr Żelasko <[email protected]>

Signed-off-by: Piotr Żelasko <[email protected]>

…atures Signed-off-by: Piotr Żelasko <[email protected]>

Signed-off-by: Piotr Żelasko <[email protected]>

Signed-off-by: pzelasko <[email protected]>

nemo/collections/asr/parts/mixins/transcription.py

@@ -17,7 +17,7 @@
 import tempfile
 from abc import ABC, abstractmethod
 from collections.abc import Iterable
-from dataclasses import dataclass
+from dataclasses import dataclass, is_dataclass


github-actions · 2024-05-31T01:47:14Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

github-actions · 2024-06-07T01:48:40Z

This PR was closed because it has been inactive for 7 days since being marked as stale.

pzelasko and others added 28 commits March 7, 2024 14:43

wip

596330d

Signed-off-by: Piotr Żelasko <[email protected]>

Partially working config groups

f54dc1c

Signed-off-by: Piotr Żelasko <[email protected]>

Working test with abasic group in the input config

dd064ab

Signed-off-by: Piotr Żelasko <[email protected]>

Working test with nested groups in input config

073f582

Signed-off-by: Piotr Żelasko <[email protected]>

Working test with specifying a YAML path for input_cfg

ad6df3f

Signed-off-by: Piotr Żelasko <[email protected]>

a very rough example of text dataloading via lhotse

a929f0a

Signed-off-by: Piotr Żelasko <[email protected]>

Cleaner integration of multimodal audio/text loading that allows to c…

a7ab811

…ontrol the effective audio vs text size (requires latest lhotse) Signed-off-by: Piotr Żelasko <[email protected]>

remove obsolete test

05efa1b

Signed-off-by: Piotr Żelasko <[email protected]>

Fix an import in export_utils.py (#8571)

d48ecd9

Signed-off-by: w4-jinhyeonkim <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

Fix bug in RNNT Joint WER calculation for fused batch (#8587)

bff40aa

Signed-off-by: smajumdar <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

change default and add key to config files (#8594)

032a17f

Signed-off-by: Chen Cui <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

Add config key for dropout position in LoRA adapter (#8583)

c8ffcc6

Signed-off-by: Michal Futrega <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

fix ia3 mlp infused adapter (#8597)

29c1225

Signed-off-by: Chen Cui <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

fix tests

8f0c9de

Signed-off-by: Piotr Żelasko <[email protected]>

Bump min required lhotse version

06ab317

Signed-off-by: Piotr Żelasko <[email protected]>

Add some documentation about this config format and the multimodal fe…

fc0f589

…atures Signed-off-by: Piotr Żelasko <[email protected]>

Merge branch 'main' into extended-nemo-data-config

3e9b6fa

Add caution about multiple shards

5e96d4a

Signed-off-by: Piotr Żelasko <[email protected]>

Merge branch 'main' into extended-nemo-data-config

fbd3efa

Address Tom's code review

c921e8b

Signed-off-by: Piotr Żelasko <[email protected]>

Add copyright header

41d246e

Signed-off-by: Piotr Żelasko <[email protected]>

Fix (hopefully) issue with forced ascii encoding in CI

d2cb979

Signed-off-by: Piotr Żelasko <[email protected]>

github-actions bot added ASR common labels Apr 5, 2024

pzelasko and others added 5 commits April 5, 2024 11:35

Support resolving input_cfg path into config contents

4543cde

Signed-off-by: Piotr Żelasko <[email protected]>

Register natural prompt fn for canary

078e649

Signed-off-by: Piotr Żelasko <[email protected]>

Fixes

a5d574c

Signed-off-by: Piotr Żelasko <[email protected]>

Fixes for WER

86dab55

Signed-off-by: Piotr Żelasko <[email protected]>

Remove prompts from prediction and scoring

5daffc0

Signed-off-by: Piotr Żelasko <[email protected]>

pzelasko force-pushed the canary-natural-prompt branch from 52cb05c to 5daffc0 Compare April 5, 2024 15:46

github-actions bot removed the common label Apr 5, 2024

Support stripping PAD and EOS from MultiTask AED beam search hypotheses

d68406e

Signed-off-by: Piotr Żelasko <[email protected]>

Base automatically changed from extended-nemo-data-config to main April 16, 2024 13:18

pzelasko and others added 9 commits April 23, 2024 14:49

Merge branch 'main' into canary-natural-prompt

9379a42

Signed-off-by: Piotr Żelasko <[email protected]>

fixes

63ba238

Signed-off-by: Piotr Żelasko <[email protected]>

Merge remote-tracking branch 'origin/main' into canary-natural-prompt

b61f8e2

support agg tokenizer for natural prompts + add unit tests

e46462d

Signed-off-by: Piotr Żelasko <[email protected]>

Merge remote-tracking branch 'origin/main' into canary-natural-prompt

41de3a2

Support specifying prompt_lang for CanaryTokenizer + natural prompts

7caa888

Signed-off-by: Piotr Żelasko <[email protected]>

Merge remote-tracking branch 'origin/main' into canary-natural-prompt

41e36c4

Merge remote-tracking branch 'origin/main' into canary-natural-prompt

dc35cf3

Apply isort and black reformatting

3b121e8

Signed-off-by: pzelasko <[email protected]>

github-advanced-security bot found potential problems May 16, 2024

View reviewed changes

github-actions bot added the stale label May 31, 2024

github-actions bot closed this Jun 7, 2024

pzelasko mentioned this pull request Jul 31, 2024

Fix Canary not stripping prompt from reference + more test coverage #9987

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Natural language prompting for Canary #8833

Natural language prompting for Canary #8833

pzelasko commented Apr 5, 2024

github-actions bot commented May 31, 2024

github-actions bot commented Jun 7, 2024

Natural language prompting for Canary #8833

Natural language prompting for Canary #8833

Conversation

pzelasko commented Apr 5, 2024

What does this PR do ?

Changelog

Usage

Jenkins CI

Before your PR is "Ready for review"

Who can review?

Additional Information

github-actions bot commented May 31, 2024

github-actions bot commented Jun 7, 2024