Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Graph freezing and unfreezing #614

Merged
merged 106 commits into from
May 7, 2020
Merged

Graph freezing and unfreezing #614

merged 106 commits into from
May 7, 2020

Conversation

tkornuta-nvidia
Copy link
Contributor

This PR introduces two graph "actions":

  • A method to freeze the weights of (all/selected) trainable modules in a graph
  • A method to unfreeze the weights of (all/selected) trainable modules in a graph

In order to make it work “in an elegant” way they PR introduces a “back-end independent module type” (ModuleType enum).

tkornuta-nvidia and others added 30 commits March 24, 2020 13:39
@tkornuta-nvidia
Manual rebase of dev-module-composition on top of current master
de587bd 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
Adding neural graph examples
15b3b11 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
formatting fixes
2756a66 
Signed-off-by: Tomasz Kornuta <[email protected]>
@blisc
new proposal
5bad9c9 
Signed-off-by: Jason <[email protected]>
@tkornuta-nvidia
Merge branch 'master' of github.com:NVIDIA/NeMo into fix-sign
e26a906
@tkornuta-nvidia
Merge branch 'fix-sign' of github.com:NVIDIA/NeMo into fix-sign
85bb729
@tkornuta-nvidia
format fix of Jason's example
0576ce7 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
Added modules recording, with retrieval of module name
3f1ac01 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
keyerror comment
a57b34d 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
Introduced name property of modules
fc3deac 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
implemented NameRegistry basic functionality, test5 working, unit tes…
5221fef 
…ts failing

Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
example 5 working, ObjectRegistry finished, app_state moved to utils
dd9af75 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
test app_state fixed
ab35621 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
TokenClassification self.name fix
bbd3589 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
ObjectRegistry tests
56efff5 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
cleanup of neural graphs class, NG manager inheriting from ObjectRegi…
c838f13 
…stry, starting to work on graph unit tests

Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
fixes to graph and manager, NeMo tests passing (locally)
1153001 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
code formatting
9df6568 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
turning examples into unit/integration tests, small fixes here and th…
868ec8a 
…ere, work in progress

Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
Graph and module nesting - operation mode injected, implemented funct…
51e852c 
…ionality with unit/integration tests

Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
formatting fix
e2ed33a 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
merged with master
069dd10 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
cleanup of output port logic in __call__, other cleanups, LGTM fixes
a054311 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
Further cleanup of neural_modules.py
c0e5f0b 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
LGTM cleanups
98cfe43 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
style fix:]
d109b5c 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
micro cleanup
6487a65 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
micro cleanup2
6c16c6e 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
punctuation module name fix
b7d3f1c 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
Working on NmTensor extensions
5548495 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
minor cleanup, unit test: a simple graph with a loop (not passing)
bd6c12e 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
style fix
7463dd8 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
Refactored the whole solution, switching from module_name:port_name t…
17e4dd9 
…o step_number:module_name:port_name (or in case of tensors: step_number:port_name). In short: enabled neural graphs to handle loops. Polished the code, unit tests and examples working. Added neural tensor type export along each connection between modules

Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
format fix
7b94c57 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
Added neural type export to config file for both graph bound input an…
e5917fe 
…d output ports, Polishes, cleanups, unit tests working

Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
jasper polished
35b31c9 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
PR polish, added type hints to all classes, added graph summary
5375055 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
removed 'name' in for in actions.py
9f6ee98 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
NG-related integration tests
45dacd3 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
LGTM fix
356e0c2 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
bind description updated
cd1e23e 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
removed app_state from jasper
37ba923 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
graph input_ports and output_ports are now immutable, added some unit…
e28979e 
… tests that check that, minor cleanups

Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
removed a line - formatting fix :]
693b744 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
name fix in test
413c0fa 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
Minor touches here and there
ccc1b21 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
Unique names generates from names of classes
a191dbe 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
moved graph managed to utils, removed it from init
68bd7e2 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
Moved neural_graph_manager to utils that enable to remove it (along w…
bc216db 
…ith ObjectRegistry) from __init__ files

Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
merged with graph master
50bc6f0 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
added freeze and unfreeze to NeuralGraphs
e170915 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
removed the examples
8b728b7 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
formatting fix
a4ac16b 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia
reverted changes from improper merge
c012c85 
Signed-off-by: Tomasz Kornuta <[email protected]>
okuchaiev
okuchaiev requested changes May 7, 2020
View reviewed changes
nemo/core/neural_modules.py Show resolved Hide resolved
nemo/backends/pytorch/nm.py Show resolved Hide resolved
@tkornuta-nvidia
added nontrainable
72a5972 
Signed-off-by: Tomasz Kornuta <[email protected]>
@tkornuta-nvidia tkornuta-nvidia merged commit 33828a9 into master May 7, 2020
@tkornuta-nvidia tkornuta-nvidia deleted the feat-graph-freezing branch May 7, 2020 22:19
dcurran90 pushed a commit to dcurran90/NeMo that referenced this pull request Oct 15, 2024
@leseb
fix: gentle error handling on server crash during chat response (NVID…
59af182 
…IA#614)

The chat was abruptly failing with the following stack trace if the chat
is answering and the server crashes during that time:

```
lab chat

╭──────────────────────────────────────────────────────────────────── system ────────────────────────────────────────────────────────────────────╮
│ Welcome to Chat CLI w/ MERLINITE-7B-Q4_K_M (type /h for help)                                                                                  │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
>>>                                                                                                                                   [S][default]
>>> hello                                                                                                                             [S][default]
╭───────────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ──────────────────────────────────────────────────────────────╮
│ Hello! I'm here to                                                                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.031 seconds ─╯
Traceback (most recent call last):
  File "/home/leseb/cli/test_run_2Fv9yrYE/venv/lib64/python3.10/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
  File "/home/leseb/cli/test_run_2Fv9yrYE/venv/lib64/python3.10/site-packages/httpx/_transports/default.py", line 113, in __iter__
    for part in self._httpcore_stream:
  File "/home/leseb/cli/test_run_2Fv9yrYE/venv/lib64/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 367, in __iter__
    raise exc from None
  File "/home/leseb/cli/test_run_2Fv9yrYE/venv/lib64/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 363, in __iter__
    for part in self._stream:
  File "/home/leseb/cli/test_run_2Fv9yrYE/venv/lib64/python3.10/site-packages/httpcore/_sync/http11.py", line 349, in __iter__
    raise exc
  File "/home/leseb/cli/test_run_2Fv9yrYE/venv/lib64/python3.10/site-packages/httpcore/_sync/http11.py", line 341, in __iter__
    for chunk in self._connection._receive_response_body(**kwargs):
  File "/home/leseb/cli/test_run_2Fv9yrYE/venv/lib64/python3.10/site-packages/httpcore/_sync/http11.py", line 210, in _receive_response_body
    event = self._receive_event(timeout=timeout)
  File "/home/leseb/cli/test_run_2Fv9yrYE/venv/lib64/python3.10/site-packages/httpcore/_sync/http11.py", line 220, in _receive_event
    with map_exceptions({h11.RemoteProtocolError: RemoteProtocolError}):
  File "/usr/lib64/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/leseb/cli/test_run_2Fv9yrYE/venv/lib64/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)
```

Now we handle this more elagantly and print before exiting:

```
lab chat
╭──────────────────────────────────────────────────────────────────── system ────────────────────────────────────────────────────────────────────╮
│ Welcome to Chat CLI w/ MERLINITE-7B-Q4_K_M (type /h for help)                                                                                  │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
>>> hello                                                                                                                             [S][default]
Connection to the server was closed
Executing chat failed with: API issue found while executing chat: Connection to the server was closed
```

Signed-off-by: Sébastien Han <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@okuchaiev okuchaiev okuchaiev approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tkornuta-nvidia @okuchaiev @blisc