Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ocean Template for C2D with decentralized compute providers #182

Open
wants to merge 106 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
106 commits
Select commit Hold shift + click to select a range
4c2c1d8
Initialized ocean template
smejak May 28, 2022
3f9b2d1
init jinja files
smejak May 28, 2022
0278b14
WIP: added ocean option to run command
smejak May 28, 2022
4e90dae
WIP: removed deploy & aml parts of render
smejak May 28, 2022
c1764dc
WIP: root.jinja
smejak May 29, 2022
f9c8826
WIP: step.jinja
smejak May 29, 2022
4cf6684
WIP: added ocean to backends
smejak May 29, 2022
7ea3839
WIP: removed unused jinja code
smejak May 29, 2022
b732ac9
WIP: root.jinja ocean working template generation
smejak May 29, 2022
fa11eae
WIP: ocean template with ML dependencies
smejak May 29, 2022
5a9ad53
WIP: removed unused template code
smejak May 30, 2022
ba014a4
Added nbconvert python template
smejak May 31, 2022
7b22160
WIP: double tag template
smejak Jun 1, 2022
d4e0e03
Correct code indentation
smejak Jun 1, 2022
726c73e
Added dcgan example
smejak Jun 1, 2022
8f71439
Update README.md
smejak Jun 1, 2022
26836ec
Update README.md
smejak Jun 1, 2022
f348977
Update README.md
smejak Jun 1, 2022
df3ee79
refactor to python_ocean
smejak Jun 2, 2022
02f9071
Merge branch 'main' of https://github.com/AlgoveraAI/same-project int…
smejak Jun 2, 2022
00e1006
WIP: different python_ocean templates
smejak Jun 8, 2022
2f8b253
Merge branch 'SAME-Project:main' into main
smejak Jun 8, 2022
8c3dc75
WIP: cleaning up python_ocean
smejak Jun 8, 2022
c32d174
WIP: starting from clean slate
smejak Jun 8, 2022
6cbacf3
WIP: initialized ocean
smejak Jun 8, 2022
045d0db
Merge branch 'SAME-Project:main' into develop
smejak Jun 8, 2022
1a63967
WIP: added ocean to backends.py
smejak Jun 8, 2022
6f31c14
WIP: init render & deploy methods
smejak Jun 9, 2022
b0696f3
WIP: boilerplate ocean c2d script
smejak Jun 16, 2022
115d4b5
WIP: simplest jinja template
smejak Jun 16, 2022
5b415c9
WIP: render function (without build)
smejak Jun 16, 2022
d3c1f10
Added ocean deploy
smejak Jun 16, 2022
963c9ac
WIP: rendering encoded script from notebook
smejak Jun 16, 2022
48f5a0a
WIP: removed encoding for render
smejak Jun 16, 2022
d2a1e10
WIP: working printing deploy
smejak Jun 17, 2022
943661a
WIP: ocean c2d deploy
smejak Jun 17, 2022
9ad0dbd
WIP: added ocean runtime options
smejak Jun 23, 2022
c6103cf
Added ocean runtime options to init
smejak Jun 23, 2022
e5e4c14
WIP: changed ocean config in deploy.py
smejak Jun 23, 2022
d9d0007
WIP: deploy with config params
smejak Jun 23, 2022
67e0d98
Merge branch 'SAME-Project:main' into develop
smejak Jun 23, 2022
804dd55
WIP: debugging options
smejak Jun 23, 2022
3108875
Merge branch 'develop' of https://github.com/AlgoveraAI/same-project …
smejak Jun 23, 2022
ea17fb4
Config params working in deploy
smejak Jun 23, 2022
d457a7c
WIP: added rawcode to algorithm metadata
smejak Jun 24, 2022
e0a7064
FIX: refactored for ocean v3
smejak Jun 29, 2022
b9dd073
WIP: added ocean to conftest
smejak Jun 29, 2022
9b5bc46
WIP: added algo_url option
smejak Jun 29, 2022
d560224
WIP: added logging statement
smejak Jun 29, 2022
ad8b4d2
WIP: ocean template test
smejak Jun 29, 2022
3457073
FIX: working, modular deploy, passed test
smejak Jun 30, 2022
6b984f0
WIP: added boolean options
smejak Jun 30, 2022
00114d8
Added Ocean-SAME docs
smejak Jun 30, 2022
2a0b970
WIP: added algo-pushed requirement
smejak Jun 30, 2022
eb09d2b
WIP: correct render option
smejak Jun 30, 2022
ef1afc6
WIP: added new runtime options
smejak Jun 30, 2022
b242a8f
WIP: added runtime options
smejak Jun 30, 2022
d41f6a7
WIP: removed publishing from ocean template
Jul 20, 2022
90c288a
WIP: remove ocean_publish
smejak Jul 20, 2022
d67741e
WIP: added algo_did runtime option
smejak Jul 21, 2022
71a2c07
WIP: removed algorithm publishing
smejak Sep 14, 2022
e194eb2
WIP: removed algorithm publishing
smejak Sep 14, 2022
9dd0508
WIP: refactoring ocean deploy
smejak Oct 7, 2022
eabdee6
WIP: fixed wrong nb name
smejak Oct 9, 2022
f9f397b
WIP: refactored render
smejak Oct 9, 2022
42f1f33
WIP: refactored template for c2d
smejak Oct 9, 2022
5aab6ef
WIP: generating the correct python script at the correct location
smejak Oct 9, 2022
754664d
WIP: removed print statements
smejak Oct 9, 2022
56403a2
WIP: removing notebook after creating script
smejak Oct 9, 2022
c1fdab2
WIP: started dockerfile for operator engine
smejak Oct 9, 2022
164af5a
WIP: disabled interactivity in dockerfile
smejak Oct 9, 2022
9fc070d
WIP: removed deploy
smejak Oct 10, 2022
375aacc
WIP: refactoring ocean to aws
smejak Oct 12, 2022
102f6eb
WIP: refactoring to boto3
smejak Oct 12, 2022
b6b91bf
WIP: added create_job from operator engine
smejak Oct 12, 2022
3fa3f81
WIP: removed unused ocean deploy
smejak Oct 31, 2022
1720ebc
WIP: added ocean_c2d for same-ocean integration
smejak Oct 31, 2022
517ea1f
WIP: added python & bash scripts for ocean c2d
smejak Nov 6, 2022
e72cf48
WIP: changed ocean.sh
smejak Nov 6, 2022
7e77a40
WIP: changed render
smejak Nov 6, 2022
4079926
Update ocean.sh
smejak Nov 7, 2022
a7eb252
WIP: changed dockerfile
smejak Nov 7, 2022
590e197
WIP: updated dockerfile & bash script with nbconvert
smejak Nov 8, 2022
adc3359
WIP: updated dockerfile & bash script with nbconvert
smejak Nov 8, 2022
33b85b8
WIP: updated dockerfile & bash script with nbconvert
smejak Nov 8, 2022
31c60da
WIP: removed user input
smejak Nov 11, 2022
64cc284
removed click.prompt
smejak Nov 11, 2022
d153dd4
added click.prompt
smejak Nov 11, 2022
9ed0299
WIP: using same in bash script
smejak Nov 11, 2022
02fab25
WIP: with run
smejak Nov 15, 2022
7d126c9
WIP: back to no deploy
smejak Nov 15, 2022
783e9a9
WIP: updated ocean.sh with correct same run and nbconvert
smejak Nov 16, 2022
9dbe0bb
WIP: correct algorithm name in ocean.sh
smejak Nov 16, 2022
52b1aae
WIP: ocean.sh for 0.2
smejak Nov 17, 2022
89a6eb7
WIP: ocean.sh for 0.3
smejak Nov 17, 2022
dee1ae7
WIP: ocean.sh for 0.4
smejak Nov 17, 2022
7fcaabc
WIP: removed line 51 from root.jinja, added port 8888
smejak Nov 17, 2022
50fc69a
WIP: added empty config.yaml
smejak Nov 17, 2022
4042ab9
WIP: hardcoding host url
smejak Nov 22, 2022
291b161
WIP: hardcoding host url
smejak Nov 22, 2022
73f6fab
WIP: hardcoding host url
smejak Nov 22, 2022
0b78087
WIP: hardcoding host url
smejak Nov 22, 2022
251cfab
WIP: hardcoding host url
smejak Nov 22, 2022
0bae01b
WIP: trying BaseOp
smejak Nov 27, 2022
3c914e0
WIP: trying to mount dataset from same.yaml in init.py
smejak Nov 27, 2022
de500b3
WIP: trying different algo docker image
smejak Nov 28, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update README.md
smejak authored Jun 1, 2022
commit 8f71439d280884d9060e4d44e41c3775e9cc8640
35 changes: 34 additions & 1 deletion sameproject/ops/python_ocean/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,38 @@
## Jinja template for Ocean C2D
# Jinja template for Ocean C2D

> Note: work-in-progress

A template for easily converting jupyter notebooks to Ocean Protocol C2D script. Use tags to specify different parts of your jupyter notebooks.

## 🏗 Initial Setup

### Set up environment
```
#clone repo
git clone https://github.com/AlgoveraAI/same-project.git
cd same-project

#create a virtual environment
python3 -m venv venv

#activate env
source venv/bin/activate

#Install the dependencies
pip install -e .
pip install jupyter
```

### Guide to using the template
Open up a jupyter notebook. This is where you will do all your data analysis and model development. Make sure to write a comment at the start of each cell (this is necessary otherwise the template will generate incorrectly indented code).

When you are done developing locally and want to publish to ocean, you will need to tag specific cells of your notebook to be read by the template. To do this click *View/Cell Toolbar/Tags*.
Tag the cells where you did your data preparation with "input" and the cells with your model and training loop "train".

When you're ready execute the following command in your terminal:
```
jupyter nbconvert path_to_notebook --to python --template=./sameproject/ops/python_ocean
```

Check that the generated script has no syntax errors.