Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

seanytak/ncf test tuning #1092

Merged
merged 21 commits into from
May 1, 2020
Merged

seanytak/ncf test tuning #1092

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5b1e551
update nni to 1.5
seanytak Apr 16, 2020
eb0c50b
Merge branch 'seanytak/ncf_test_tuning' of github.com:seanytak/recomm…
Apr 16, 2020
6e485a1
update nni==1.5
seanytak Apr 17, 2020
67e54de
update nni surprise svd for nni 1.5
seanytak Apr 17, 2020
f20cf2e
add ncf training using nni
seanytak Apr 21, 2020
457d363
add ncf tuning harness notebook
seanytak Apr 21, 2020
7a4f24e
add model comparison for svd and ncf
seanytak Apr 22, 2020
70d5420
clean up ncf training
seanytak Apr 23, 2020
75c2999
clean up ncf markdown
seanytak Apr 23, 2020
d71ab3f
removed unused params in ncf training
seanytak Apr 23, 2020
77214ef
update readme
seanytak Apr 23, 2020
bed9d6a
Merge branch 'staging' into seanytak/ncf_test_tuning
miguelgfierro Apr 23, 2020
5ab3cae
Merge branch 'staging' into seanytak/ncf_test_tuning
miguelgfierro Apr 24, 2020
4cb36ae
remove svd components
seanytak Apr 27, 2020
4f491fb
fix up final test results reporting
seanytak Apr 27, 2020
b5c2c6a
Merge branch 'staging' into seanytak/ncf_test_tuning
miguelgfierro Apr 28, 2020
ead671c
remove output from cell with warnings
seanytak Apr 29, 2020
3f86742
fix output on nni svd surprise
seanytak Apr 29, 2020
1902400
fix notebook outputs for demonstration purposes
seanytak Apr 29, 2020
be04df2
remove old svd references in nni_ncf
seanytak Apr 30, 2020
4f8a3f1
remove old svd model saving
seanytak Apr 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
225 changes: 190 additions & 35 deletions notebooks/02_model/ncf_deep_dive.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,31 @@
"execution_count": 1,
miguelgfierro marked this conversation as resolved.
Show resolved Hide resolved
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
" _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
" _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
" _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
" _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
" _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
" np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"System version: 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) \n",
"System version: 3.6.10 |Anaconda, Inc.| (default, Mar 25 2020, 23:51:54) \n",
"[GCC 7.3.0]\n",
"Pandas version: 0.24.2\n",
"Pandas version: 0.25.3\n",
"Tensorflow version: 1.12.0\n"
]
}
Expand Down Expand Up @@ -206,7 +224,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"4.93MB [00:00, 18.4MB/s] \n"
"100%|██████████| 4.81k/4.81k [00:00<00:00, 10.7kKB/s]\n"
]
},
{
Expand Down Expand Up @@ -380,7 +398,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Took 662.0558905601501 seconds for training.\n"
"Took 806.4961465689998 seconds for training.\n"
]
}
],
Expand Down Expand Up @@ -443,43 +461,43 @@
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>149.0</td>\n",
" <td>0.150561</td>\n",
" <td>0.047002</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1.0</td>\n",
" <td>88.0</td>\n",
" <td>0.307006</td>\n",
" <td>0.566155</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>1.0</td>\n",
" <td>101.0</td>\n",
" <td>0.346554</td>\n",
" <td>0.226456</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.0</td>\n",
" <td>110.0</td>\n",
" <td>0.185647</td>\n",
" <td>0.008283</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>1.0</td>\n",
" <td>103.0</td>\n",
" <td>0.002877</td>\n",
" <td>0.005900</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" userID itemID prediction\n",
"0 1.0 149.0 0.150561\n",
"1 1.0 88.0 0.307006\n",
"2 1.0 101.0 0.346554\n",
"3 1.0 110.0 0.185647\n",
"4 1.0 103.0 0.002877"
"0 1.0 149.0 0.047002\n",
"1 1.0 88.0 0.566155\n",
"2 1.0 101.0 0.226456\n",
"3 1.0 110.0 0.008283\n",
"4 1.0 103.0 0.005900"
]
},
"execution_count": 8,
Expand Down Expand Up @@ -514,7 +532,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Took 2.801203966140747 seconds for prediction.\n"
"Took 2.9599234249999427 seconds for prediction.\n"
]
}
],
Expand Down Expand Up @@ -546,10 +564,10 @@
"name": "stdout",
"output_type": "stream",
"text": [
"MAP:\t0.044980\n",
"NDCG:\t0.189388\n",
"Precision@K:\t0.170626\n",
"Recall@K:\t0.094332\n"
"MAP:\t0.046569\n",
"NDCG:\t0.192938\n",
"Precision@K:\t0.173171\n",
"Recall@K:\t0.096173\n"
]
}
],
Expand Down Expand Up @@ -592,8 +610,8 @@
"name": "stdout",
"output_type": "stream",
"text": [
"HR:\t0.489403\n",
"NDCG:\t0.385408\n"
"HR:\t0.485765\n",
"NDCG:\t0.379814\n"
]
}
],
Expand Down Expand Up @@ -674,7 +692,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Took 514.4518468379974 seconds for training.\n"
"Took 500.5236881850001 seconds for training.\n"
]
}
],
Expand Down Expand Up @@ -717,7 +735,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Took 565.3551139831543 seconds for training.\n"
"Took 552.2027924220001 seconds for training.\n"
]
}
],
Expand Down Expand Up @@ -778,7 +796,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Took 660.8611929416656 seconds for training.\n"
"Took 644.399864857 seconds for training.\n"
]
}
],
Expand Down Expand Up @@ -807,7 +825,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Took 2.8479833602905273 seconds for prediction.\n"
"Took 3.0592451869997603 seconds for prediction.\n"
]
}
],
Expand Down Expand Up @@ -839,10 +857,10 @@
"name": "stdout",
"output_type": "stream",
"text": [
"MAP:\t0.041600\n",
"NDCG:\t0.174057\n",
"Precision@K:\t0.160657\n",
"Recall@K:\t0.091654\n"
"MAP:\t0.046286\n",
"NDCG:\t0.186660\n",
"Precision@K:\t0.169035\n",
"Recall@K:\t0.100020\n"
]
}
],
Expand All @@ -860,9 +878,146 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 20,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: Function record is deprecated and will be removed in verison 1.0.0 (current version 0.19.1). Please see `scrapbook.glue` (nteract-scrapbook) as a replacement for this functionality.\n",
" \n"
]
},
{
"data": {
"application/papermill.record+json": {
"map": 0.046568915592114325
}
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Function record is deprecated and will be removed in verison 1.0.0 (current version 0.19.1). Please see `scrapbook.glue` (nteract-scrapbook) as a replacement for this functionality.\n",
" This is separate from the ipykernel package so we can avoid doing imports until\n"
]
},
{
"data": {
"application/papermill.record+json": {
"ndcg": 0.37981413819338766
}
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/ipykernel_launcher.py:4: DeprecationWarning: Function record is deprecated and will be removed in verison 1.0.0 (current version 0.19.1). Please see `scrapbook.glue` (nteract-scrapbook) as a replacement for this functionality.\n",
" after removing the cwd from sys.path.\n"
]
},
{
"data": {
"application/papermill.record+json": {
"precision": 0.1731707317073171
}
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/ipykernel_launcher.py:5: DeprecationWarning: Function record is deprecated and will be removed in verison 1.0.0 (current version 0.19.1). Please see `scrapbook.glue` (nteract-scrapbook) as a replacement for this functionality.\n",
" \"\"\"\n"
]
},
{
"data": {
"application/papermill.record+json": {
"recall": 0.096173321399973
}
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/ipykernel_launcher.py:6: DeprecationWarning: Function record is deprecated and will be removed in verison 1.0.0 (current version 0.19.1). Please see `scrapbook.glue` (nteract-scrapbook) as a replacement for this functionality.\n",
" \n"
]
},
{
"data": {
"application/papermill.record+json": {
"map2": 0.046286433403948564
}
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/ipykernel_launcher.py:7: DeprecationWarning: Function record is deprecated and will be removed in verison 1.0.0 (current version 0.19.1). Please see `scrapbook.glue` (nteract-scrapbook) as a replacement for this functionality.\n",
" import sys\n"
]
},
{
"data": {
"application/papermill.record+json": {
"ndcg2": 0.1866595787134735
}
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/ipykernel_launcher.py:8: DeprecationWarning: Function record is deprecated and will be removed in verison 1.0.0 (current version 0.19.1). Please see `scrapbook.glue` (nteract-scrapbook) as a replacement for this functionality.\n",
" \n"
]
},
{
"data": {
"application/papermill.record+json": {
"precision2": 0.1690349946977731
}
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/anaconda/envs/recommenders_gpu/lib/python3.6/site-packages/ipykernel_launcher.py:9: DeprecationWarning: Function record is deprecated and will be removed in verison 1.0.0 (current version 0.19.1). Please see `scrapbook.glue` (nteract-scrapbook) as a replacement for this functionality.\n",
" if __name__ == '__main__':\n"
]
},
{
"data": {
"application/papermill.record+json": {
"recall2": 0.10002001904879806
}
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Record results with papermill for tests\n",
"pm.record(\"map\", eval_map)\n",
Expand Down Expand Up @@ -919,9 +1074,9 @@
"metadata": {
"celltoolbar": "Tags",
"kernelspec": {
"display_name": "reco_gpu",
"display_name": "Python (recommenders_gpu)",
"language": "python",
"name": "reco_gpu"
"name": "recommenders_gpu"
},
"language_info": {
"codemirror_mode": {
Expand All @@ -933,9 +1088,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.8"
"version": "3.6.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
}
1 change: 1 addition & 0 deletions notebooks/04_model_select_and_optimize/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ In this directory, notebooks are provided to demonstrate how to tune and optimiz
| [azureml_hyperdrive_wide_and_deep](azureml_hyperdrive_wide_and_deep.ipynb) | Quickstart tutorial on utilizing [Azure Machine Learning service](https://azure.microsoft.com/en-us/services/machine-learning-service/) for hyperparameter tuning of wide-and-deep model.
| [azureml_hyperdrive_surprise_svd](azureml_hyperdrive_surprise_svd.ipynb) | Quickstart tutorial on utilizing [Azure Machine Learning service](https://azure.microsoft.com/en-us/services/machine-learning-service/) for hyperparameter tuning of the matrix factorization method SVD from [Surprise library](https://surprise.readthedocs.io/en/stable/).
| [nni_surprise_svd](nni_surprise_svd.ipynb) | Quickstart tutorial on utilizing the [Neural Network Intelligence toolkit](https://github.com/Microsoft/nni) for hyperparameter tuning of the matrix factorization method SVD from [Surprise library](https://surprise.readthedocs.io/en/stable/).
| [nni_ncf](nni_ncf.ipynb) | Quickstart tutorial on utilizing the [Neural Network Intelligence toolkit](https://github.com/Microsoft/nni) as a tool to tune the [NCF model](../02_model/ncf_deep_dive.ipynb) and [SVD model](../02_model/surprise_svd_deep_dive.ipynb) and compare their performance against one another

### Prerequisites
To run the examples running on the Azure Machine Learning service, the [`azureml-sdk`](https://pypi.org/project/azureml-sdk/) is required. The AzureML Python SDK is already installed after setting up the conda environments from this repository (see [SETUP.md](../../SETUP.md)).
Expand Down
Loading