Add Lita, Vila and Vita TRTLLM export #9734

xuanzic · 2024-07-15T17:19:25Z

What does this PR do ?

modify the neva_model to remove hardcoded 'vision_encoder" name.
add trtllm export support for lita, vila and vita .nemo model.
add run example for running lita, vila and vita inference.
add triton deployment for these models.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Vivian Chen <[email protected]>

examples/multimodal/multimodal_llm/neva/neva_export.py

Signed-off-by: Vivian Chen <[email protected]>

Signed-off-by: xuanzic <[email protected]>

nemo/export/multimodal/run.py

+        return image
+
+    def process_image(self, image_file, image_processor, nemo_config, image_folder):
+        image_processor = image_processor


nemo/export/multimodal/run.py

 from torchvision import transforms
-from transformers import CLIPImageProcessor
+from transformers import AutoModel, AutoProcessor, CLIPImageProcessor


nemo/export/multimodal/build.py

        dtype,
        image_size=image_size,
+        num_frames=lita_num_frames if model_type == "lita" or model_type == 'vita' else None,


nemo/export/multimodal/run.py

-        else:
-            image = image.expand(batch_size, -1, -1, -1).contiguous()
+        if self.model_type not in ['vila', 'lita', 'vita']:
+            if image.dim() == 5:


Signed-off-by: Vivian Chen <[email protected]>

Signed-off-by: xuanzic <[email protected]>

nemo/export/multimodal/run.py

+        elif isinstance(video_path, np.ndarray):
+            frames = torch.tensor(video_path, dtype=torch.float32)
+
+        return self.preprocess_frames(frames, config, processor)


nemo/export/multimodal/run.py

+                .unsqueeze(0)
+                .to(self.device, dtype=torch.bfloat16)
+            )
+        return image


Signed-off-by: Vivian Chen <[email protected]>

meatybobby

LGTM

* modify vision encoder config Signed-off-by: Vivian Chen <[email protected]> * add lita, vila engine build support and fix export api bugs Signed-off-by: Vivian Chen <[email protected]> * add run example for vila, lita and vita Signed-off-by: Vivian Chen <[email protected]> * couple of changes for exporter Signed-off-by: Vivian Chen <[email protected]> * Apply isort and black reformatting Signed-off-by: xuanzic <[email protected]> * address code scanning issues Signed-off-by: Vivian Chen <[email protected]> * add triton deployment for lita/vila/vita Signed-off-by: Vivian Chen <[email protected]> * Apply isort and black reformatting Signed-off-by: xuanzic <[email protected]> * fix code scan Signed-off-by: Vivian Chen <[email protected]> --------- Signed-off-by: Vivian Chen <[email protected]> Signed-off-by: Vivian Chen <[email protected]> Signed-off-by: xuanzic <[email protected]> Co-authored-by: Vivian Chen <[email protected]> Co-authored-by: xuanzic <[email protected]>

* modify vision encoder config Signed-off-by: Vivian Chen <[email protected]> * add lita, vila engine build support and fix export api bugs Signed-off-by: Vivian Chen <[email protected]> * add run example for vila, lita and vita Signed-off-by: Vivian Chen <[email protected]> * couple of changes for exporter Signed-off-by: Vivian Chen <[email protected]> * Apply isort and black reformatting Signed-off-by: xuanzic <[email protected]> * address code scanning issues Signed-off-by: Vivian Chen <[email protected]> * add triton deployment for lita/vila/vita Signed-off-by: Vivian Chen <[email protected]> * Apply isort and black reformatting Signed-off-by: xuanzic <[email protected]> * fix code scan Signed-off-by: Vivian Chen <[email protected]> --------- Signed-off-by: Vivian Chen <[email protected]> Signed-off-by: Vivian Chen <[email protected]> Signed-off-by: xuanzic <[email protected]> Co-authored-by: Vivian Chen <[email protected]> Co-authored-by: xuanzic <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]>

* modify vision encoder config Signed-off-by: Vivian Chen <[email protected]> * add lita, vila engine build support and fix export api bugs Signed-off-by: Vivian Chen <[email protected]> * add run example for vila, lita and vita Signed-off-by: Vivian Chen <[email protected]> * couple of changes for exporter Signed-off-by: Vivian Chen <[email protected]> * Apply isort and black reformatting Signed-off-by: xuanzic <[email protected]> * address code scanning issues Signed-off-by: Vivian Chen <[email protected]> * add triton deployment for lita/vila/vita Signed-off-by: Vivian Chen <[email protected]> * Apply isort and black reformatting Signed-off-by: xuanzic <[email protected]> * fix code scan Signed-off-by: Vivian Chen <[email protected]> --------- Signed-off-by: Vivian Chen <[email protected]> Signed-off-by: Vivian Chen <[email protected]> Signed-off-by: xuanzic <[email protected]> Co-authored-by: Vivian Chen <[email protected]> Co-authored-by: xuanzic <[email protected]>

* modify vision encoder config Signed-off-by: Vivian Chen <[email protected]> * add lita, vila engine build support and fix export api bugs Signed-off-by: Vivian Chen <[email protected]> * add run example for vila, lita and vita Signed-off-by: Vivian Chen <[email protected]> * couple of changes for exporter Signed-off-by: Vivian Chen <[email protected]> * Apply isort and black reformatting Signed-off-by: xuanzic <[email protected]> * address code scanning issues Signed-off-by: Vivian Chen <[email protected]> * add triton deployment for lita/vila/vita Signed-off-by: Vivian Chen <[email protected]> * Apply isort and black reformatting Signed-off-by: xuanzic <[email protected]> * fix code scan Signed-off-by: Vivian Chen <[email protected]> --------- Signed-off-by: Vivian Chen <[email protected]> Signed-off-by: Vivian Chen <[email protected]> Signed-off-by: xuanzic <[email protected]> Co-authored-by: Vivian Chen <[email protected]> Co-authored-by: xuanzic <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

* modify vision encoder config Signed-off-by: Vivian Chen <[email protected]> * add lita, vila engine build support and fix export api bugs Signed-off-by: Vivian Chen <[email protected]> * add run example for vila, lita and vita Signed-off-by: Vivian Chen <[email protected]> * couple of changes for exporter Signed-off-by: Vivian Chen <[email protected]> * Apply isort and black reformatting Signed-off-by: xuanzic <[email protected]> * address code scanning issues Signed-off-by: Vivian Chen <[email protected]> * add triton deployment for lita/vila/vita Signed-off-by: Vivian Chen <[email protected]> * Apply isort and black reformatting Signed-off-by: xuanzic <[email protected]> * fix code scan Signed-off-by: Vivian Chen <[email protected]> --------- Signed-off-by: Vivian Chen <[email protected]> Signed-off-by: Vivian Chen <[email protected]> Signed-off-by: xuanzic <[email protected]> Co-authored-by: Vivian Chen <[email protected]> Co-authored-by: xuanzic <[email protected]>

Vivian Chen and others added 2 commits July 11, 2024 16:42

modify vision encoder config

8a67982

Signed-off-by: Vivian Chen <[email protected]>

add lita, vila engine build support and fix export api bugs

3e65e6a

Signed-off-by: Vivian Chen <[email protected]>

github-actions bot added the Multi Modal label Jul 15, 2024

xuanzic changed the title ~~Add Lita, Vila and Vita TRTLLM export~~ [DRAFT] Add Lita, Vila and Vita TRTLLM export Jul 15, 2024

xuanzic marked this pull request as draft July 15, 2024 17:27

add run example for vila, lita and vita

d87455e

Signed-off-by: Vivian Chen <[email protected]>

meatybobby reviewed Jul 16, 2024

View reviewed changes

examples/multimodal/multimodal_llm/neva/neva_export.py Show resolved Hide resolved

couple of changes for exporter

df4228f

Signed-off-by: Vivian Chen <[email protected]>

xuanzic changed the title ~~[DRAFT] Add Lita, Vila and Vita TRTLLM export~~ Add Lita, Vila and Vita TRTLLM export Jul 17, 2024

xuanzic marked this pull request as ready for review July 17, 2024 03:21

xuanzic and others added 2 commits July 16, 2024 20:21

Merge branch 'main' into vchen/vila-infer

7d17754

Apply isort and black reformatting

d1f05fd

Signed-off-by: xuanzic <[email protected]>

github-advanced-security bot found potential problems Jul 17, 2024

View reviewed changes

Vivian Chen and others added 3 commits July 17, 2024 04:39

address code scanning issues

f33bf64

Signed-off-by: Vivian Chen <[email protected]>

add triton deployment for lita/vila/vita

c0427d9

Signed-off-by: Vivian Chen <[email protected]>

Apply isort and black reformatting

62b5fa7

Signed-off-by: xuanzic <[email protected]>

github-advanced-security bot found potential problems Jul 18, 2024

View reviewed changes

fix code scan

a1ea8cc

Signed-off-by: Vivian Chen <[email protected]>

meatybobby approved these changes Jul 18, 2024

View reviewed changes

meatybobby added the Run CICD label Jul 18, 2024

xuanzic added 2 commits July 18, 2024 08:13

Merge branch 'main' into vchen/vila-infer

e176675

Merge branch 'main' into vchen/vila-infer

ab66b47

meatybobby added Run CICD and removed Run CICD labels Jul 19, 2024

meatybobby merged commit ab8988e into NVIDIA:main Jul 19, 2024
206 of 207 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Lita, Vila and Vita TRTLLM export #9734

Add Lita, Vila and Vita TRTLLM export #9734

xuanzic commented Jul 15, 2024 •

edited

Loading

meatybobby left a comment

Add Lita, Vila and Vita TRTLLM export #9734

Add Lita, Vila and Vita TRTLLM export #9734

Conversation

xuanzic commented Jul 15, 2024 • edited Loading

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

meatybobby left a comment

Choose a reason for hiding this comment

xuanzic commented Jul 15, 2024 •

edited

Loading