client.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "168470ff",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<p align=\"center\">\n",
    "<a href=\"https://github.com/jina-ai/dalle\"><img src=\"https://github.com/jina-ai/dalle-flow/blob/main/.github/banner.svg?raw=true\" alt=\"DALL·E Flow: A Human-in-the-Loop workflow for creating HD images from text\" width=\"60%\"></a>\n",
    "<br>\n",
    "</p>\n",
    "\n",
    "\n",
    "<b>A Human-in-the-Loop<sup><a href=\"https://en.wikipedia.org/wiki/Human-in-the-loop\">?</a></sup> workflow for creating HD images from text</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85339e6c",
   "metadata": {},
   "source": [
    "[![GitHub Repo stars](https://img.shields.io/github/stars/jina-ai/dalle-flow?style=social)](https://github.com/jina-ai/dalle-flow) [![Google Colab](https://img.shields.io/badge/Slack-2.8k-blueviolet?logo=slack&amp;logoColor=white&style=flat-square)](https://slack.jina.ai) [![GitHub last commit (branch)](https://img.shields.io/github/last-commit/jina-ai/dalle-flow/main)](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d76e649",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "\n",
    "\n",
    "🚧 If you find your request fails, it is possible the server is occasionally down for maintaince. Please give it a try in 5 minutes.\n",
    "\n",
    "\n",
    "- 🌟 2022/5/7 DALL·E Flow just got updated!\n",
    "    - New DALL·E Mega checkpoint\n",
    "    - Improved GLID3 memory-efficiency and parameters\n",
    "- 🌟 2022/5/6 DALL·E Flow just got updated!\n",
    "    - The first step will generate 16 candidates: **8 from DALL·E Mega, 8 from GLID3-XL**; ranked by CLIP-as-service.\n",
    "    - Optimized the flow efficiency, diffusion and upscaling is much faster now!\n",
    "- ~~⚠️ 2022/5/3 **The number of images is restrict to 9 for DALL·E Mega, and 16 for GLID3-XL**~~\n",
    "- ⚠️ 2022/5/2 **Due to the massive requests now, the server is super busy.** You can deploy your own server by following [the instruction here](https://github.com/jina-ai/dalle-flow#server).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1304c98",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Using client is super easy. The following steps are best run in Jupyter notebook or [Google Colab](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).  \n",
    "\n",
    "The only dependency you will need are [DocArray](https://github.com/jina-ai/docarray) and [Jina](https://github.com/jina-ai/jina):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cb1cd20",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install \"docarray[common]>=0.13.10\" jina"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66a750e1",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "We have provided a demo server for you to play:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee768ad6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "server_url = 'grpc://dalle-flow.jina.ai:51005'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaaac9dc",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Step 1: Generate via DALL·E Mega\n",
    "\n",
    "Now let's define the prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f60975f",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "prompt = 'an oil painting of a humanoid robot playing chess in the style of Matisse'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8168d6c6",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Let's submit it to the server and visualize the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b280af05",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "from docarray import Document\n",
    "\n",
    "da = Document(text=prompt).post(server_url, parameters={'num_images': 8}).matches\n",
    "\n",
    "da.plot_image_sprites(fig_size=(10,10), show_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a86369e0",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Here we generate 16 candidates, 8 from DALLE-mega and 8 from GLID3 XL, this is as defined in `num_images`, which takes about ~2 minutes. You can use a smaller value if it is too long for you. The results are sorted by [CLIP-as-service](https://github.com/jina-ai/clip-as-service), with index-`0` as the best candidate judged by CLIP. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fd9bc33",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Step 2: Select and refinement via GLID3 XL\n",
    "\n",
    "Of course, you may think differently. Notice the number in the top-left corner? Select the one you like the most and get a better view:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b57eef8",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fav_id = 9\n",
    "\n",
    "fav = da[fav_id]\n",
    "\n",
    "fav.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "385c28b2",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now let's submit the selected candidates to the server for diffusion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4629f9c6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "diffused = fav.post(f'{server_url}', parameters={'skip_rate': 0.5, 'num_images': 9}, target_executor='diffusion').matches\n",
    "\n",
    "diffused.plot_image_sprites(fig_size=(10,10), show_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c7ccbb7",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "This will give 9 images based on the given image. You may allow the model to improvise more by giving `skip_rate` a near-zero value, or a near-one value to force its closeness to the given image. The whole procedure takes about ~1 minutes.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e421f371",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Step 3: Select and upscale via SwinIR\n",
    "\n",
    "Select the image you like the most, and give it a closer look:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f83734fa",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dfav_id = 2\n",
    "\n",
    "fav = diffused[dfav_id]\n",
    "\n",
    "fav.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c807dcb",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Finally, submit to the server for the last step: upscaling to 1024 x 1024px."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "102fbcda",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "fav = fav.post(f'{server_url}/upscale', target_executor='upscaler')\n",
    "fav.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed93a102",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "> 💁‍♂️ On Google colab this image may render exactly the same size as before. But it is in 1024x1024 already. Right click on the image and copy/save it. You will see.\n",
    "\n",
    "That's it! It is _the one_. If not satisfied, please repeat the procedure. Btw, DocArray is a powerful and easy-to-use data structure for unstructured data. It is super productive for data scientists who work in cross-/multi-modal domain. To learn more about DocArray, [please check out the docs](https://docarray.jina.ai)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcd6d387",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}