gemini/getting-started/intro_gemini_curl.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ijGzTHJJUCPY"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZPC2X_a9ErW7"
      },
      "source": [
        "# Getting Started with the Gemini API in Vertex AI with cURL\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fgetting-started%2Fintro_gemini_curl.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Run in Colab Enterprise\n",
        "    </a>\n",
        "  </td>       \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/getting-started/intro_gemini_curl.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/53/X_logo_2023_original.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>            \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f0cc0f48513b"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "|Author(s) | [Eric Dong](https://github.com/gericdong), [Polong Lin](https://github.com/polong-lin) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "axauUzNXEl_R"
      },
      "source": [
        "## Overview\n",
        "\n",
        "### Gemini\n",
        "Gemini is a family of generative AI models developed by Google DeepMind. Gemini models support prompts that include text, image, and video as input and support text responses as output.\n",
        "\n",
        "### Gemini API in Vertex AI\n",
        "\n",
        "The Gemini API in Vertex AI provides a unified interface for interacting with Gemini models. You can interact with the Gemini API by using the following methods:\n",
        "\n",
        "* Use [Vertex AI Studio](https://cloud.google.com/generative-ai-studio) for quick testing and command generation.\n",
        "* Use cURL commands in Cloud Shell.\n",
        "* Use the Vertex AI SDK for Python in a Jupyter notebook\n",
        "\n",
        "This notebook focuses on using the **cURL commands** to call the Gemini API in Vertex AI.\n",
        "\n",
        "For more information, see the [Generative AI on Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview) documentation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yyaOZAg_El_R"
      },
      "source": [
        "### Objectives\n",
        "\n",
        "In this tutorial, you learn how to use the Gemini API in Vertex AI with cURL commands to interact with the Gemini 1.5 Pro (`gemini-1.5-pro`) model.\n",
        "\n",
        "You will complete the following tasks:\n",
        "\n",
        "- Install the Python SDK.\n",
        "- Use the Gemini API in Vertex AI to interact with each model.\n",
        "  - Gemini 1.5 Pro (`gemini-1.5-pro`) model:\n",
        "    - Generate text from text prompts.\n",
        "    - Explore various features and configuration options.\n",
        "    - Generate text from image(s) and text prompts.\n",
        "    - Generate text from video.\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wJf9sLIIEl_S"
      },
      "source": [
        "### Costs\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "- Vertex AI\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D50ekWXjEl_S"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, run the following cell to authenticate your environment.\n",
        "\n",
        "This step is not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# Additional authentication is required for Google Colab\n",
        "if \"google.colab\" in sys.modules:\n",
        "    # Authenticate user to Google Cloud\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O6ZGaZlxP9L0"
      },
      "source": [
        "### Set Google Cloud project\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u8IivOG5SqY6"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project_id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8s84m7h6HSTR"
      },
      "source": [
        "### Defining environment variables for cURL commands\n",
        "\n",
        "These environment variables are used to construct the cURL commands."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "krJ8UOHKoPn3"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "os.environ[\"PROJECT_ID\"] = PROJECT_ID\n",
        "os.environ[\"LOCATION\"] = LOCATION\n",
        "os.environ[\"API_ENDPOINT\"] = f\"{LOCATION}-aiplatform.googleapis.com\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "854fbf388e2b"
      },
      "source": [
        "## Use the Gemini 1.5 Pro model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7eeb063ac6d4"
      },
      "outputs": [],
      "source": [
        "os.environ[\"MODEL_ID\"] = \"gemini-1.5-pro\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ZZUVBSzc0cR"
      },
      "source": [
        "### Generate content\n",
        "\n",
        "The generateContent method can handle a wide variety of use cases, including multi-turn chat and multimodal input, depending on what the underlying model supports. In this example, you send a text prompt using the `generateContent` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1979afec8834"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:generateContent \\\n",
        "  -d '{\n",
        "    \"contents\": {\n",
        "      \"role\": \"USER\",\n",
        "      \"parts\": { \"text\": \"Why is the sky blue?\" }\n",
        "    }\n",
        "  }'\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "27701e417da6"
      },
      "source": [
        "### Streaming\n",
        "\n",
        "The Gemini API provides a streaming response mechanism. With this approach, you don't need to wait for the complete response; you can start processing fragments as soon as they're accessible."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rzkCij_iS0we"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:streamGenerateContent \\\n",
        "  -d '{\n",
        "    \"contents\": {\n",
        "      \"role\": \"USER\",\n",
        "      \"parts\": { \"text\": \"Why is the sky blue?\" }\n",
        "    }\n",
        "  }'\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3e56BV7PH9t8"
      },
      "source": [
        "### Model parameters\n",
        "\n",
        "Every prompt you send to the model includes parameter values that control how the model generates a response. The model can generate different results for different parameter values. You can experiment with different model parameters to see how the results change. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Px8hSHhiH9t8"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:generateContent \\\n",
        "  -d '{\n",
        "    \"contents\": {\n",
        "      \"role\": \"USER\",\n",
        "      \"parts\": [\n",
        "        {\"text\": \"Describe this image\"},\n",
        "        {\"file_data\": {\n",
        "          \"mime_type\": \"image/png\",\n",
        "          \"file_uri\": \"gs://cloud-samples-data/generative-ai/image/320px-Felis_catus-cat_on_snow.jpg\"\n",
        "        }}\n",
        "      ]\n",
        "    },\n",
        "    \"generation_config\": {\n",
        "      \"temperature\": 0.2,\n",
        "      \"top_p\": 0.1,\n",
        "      \"top_k\": 16,\n",
        "      \"max_output_tokens\": 2048,\n",
        "      \"candidate_count\": 1,\n",
        "      \"stop_sequences\": []\n",
        "    },\n",
        "    \"safety_settings\": {\n",
        "      \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
        "      \"threshold\": \"BLOCK_LOW_AND_ABOVE\"\n",
        "    }\n",
        "  }'\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I4-XhmPn_Pb-"
      },
      "source": [
        "### Chat\n",
        "\n",
        "The Gemini API supports natural multi-turn conversations and is ideal for text tasks that require back-and-forth interactions.\n",
        "\n",
        "Specify the `role` field only if the content represents a turn in a conversation. You can set `role` to one of the following values: `user`, `model`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YqSQSK-K-KVU"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:generateContent \\\n",
        "  -d '{\n",
        "    \"contents\": [\n",
        "      {\n",
        "        \"role\": \"user\",\n",
        "        \"parts\": [\n",
        "          { \"text\": \"Hello\" }\n",
        "        ]\n",
        "      },\n",
        "      {\n",
        "        \"role\": \"model\",\n",
        "        \"parts\": [\n",
        "          { \"text\": \"Hello! I am glad you could both make it.\" }\n",
        "        ]\n",
        "      },\n",
        "      {\n",
        "        \"role\": \"user\",\n",
        "        \"parts\": [\n",
        "          { \"text\": \"So what is the first order of business?\" }\n",
        "        ]\n",
        "      }\n",
        "    ]\n",
        "  }'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5f0f5fe3b331"
      },
      "source": [
        "### Function calling\n",
        "\n",
        "Function calling lets you create a description of a function in their code, then pass that description to a language model in a request. This sample is an example of passing in a description of a function that returns information about where a movie is playing. Several function declarations are included in the request, such as `find_movies` and `find_theaters`.\n",
        "\n",
        "Learn more about [function calling](https://cloud.google.com/vertex-ai/docs/generative-ai/multimodal/function-calling)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "680b11b0ba4c"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  https://${API_ENDPOINT}/v1beta1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:generateContent \\\n",
        "  -d '{\n",
        "  \"contents\": {\n",
        "    \"role\": \"user\",\n",
        "    \"parts\": {\n",
        "      \"text\": \"Which theaters in Mountain View show Barbie movie?\"\n",
        "    }\n",
        "  },\n",
        "  \"tools\": [\n",
        "    {\n",
        "      \"function_declarations\": [\n",
        "        {\n",
        "          \"name\": \"find_movies\",\n",
        "          \"description\": \"find movie titles currently playing in theaters based on any description, genre, title words, etc.\",\n",
        "          \"parameters\": {\n",
        "            \"type\": \"object\",\n",
        "            \"properties\": {\n",
        "              \"location\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"The city and state, e.g. San Francisco, CA or a zip code e.g. 95616\"\n",
        "              },\n",
        "              \"description\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"Any kind of description including category or genre, title words, attributes, etc.\"\n",
        "              }\n",
        "            },\n",
        "            \"required\": [\n",
        "              \"description\"\n",
        "            ]\n",
        "          }\n",
        "        },\n",
        "        {\n",
        "          \"name\": \"find_theaters\",\n",
        "          \"description\": \"find theaters based on location and optionally movie title which are is currently playing in theaters\",\n",
        "          \"parameters\": {\n",
        "            \"type\": \"object\",\n",
        "            \"properties\": {\n",
        "              \"location\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"The city and state, e.g. San Francisco, CA or a zip code e.g. 95616\"\n",
        "              },\n",
        "              \"movie\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"Any movie title\"\n",
        "              }\n",
        "            },\n",
        "            \"required\": [\n",
        "              \"location\"\n",
        "            ]\n",
        "          }\n",
        "        },\n",
        "        {\n",
        "          \"name\": \"get_showtimes\",\n",
        "          \"description\": \"Find the start times for movies playing in a specific theater\",\n",
        "          \"parameters\": {\n",
        "            \"type\": \"object\",\n",
        "            \"properties\": {\n",
        "              \"location\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"The city and state, e.g. San Francisco, CA or a zip code e.g. 95616\"\n",
        "              },\n",
        "              \"movie\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"Any movie title\"\n",
        "              },\n",
        "              \"theater\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"Name of theater\"\n",
        "              },\n",
        "              \"date\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"Date for requested showtime\"\n",
        "              }\n",
        "            },\n",
        "            \"required\": [\n",
        "              \"location\",\n",
        "              \"movie\",\n",
        "              \"theater\",\n",
        "              \"date\"\n",
        "            ]\n",
        "          }\n",
        "        }\n",
        "      ]\n",
        "    }\n",
        "  ]\n",
        "}'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R3g5n23lDtsN"
      },
      "source": [
        "## Multimodal input\n",
        "\n",
        "The Gemini 1.5 Pro  (`gemini-1.5-pro`) is a multimodal model that supports adding image and video in text or chat prompts for a text response.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uTfL2DDch4Lp"
      },
      "source": [
        "### Download an image from Google Cloud Storage"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KmtWSNLtJ7oD"
      },
      "outputs": [],
      "source": [
        "! gsutil cp \"gs://cloud-samples-data/generative-ai/image/320px-Felis_catus-cat_on_snow.jpg\" ./image.jpg"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BlyyaPgmhpyv"
      },
      "source": [
        "### Generate text from a local image\n",
        "\n",
        "Specify the [base64](https://en.wikipedia.org/wiki/Base64) encoding of the image or video to include inline in the prompt and the `mime_type` field. The supported [MIME types](https://en.wikipedia.org/wiki/Media_type) for images include `image/png` and `image/jpeg`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-uqZ-RWdtdit"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "# Encode image data in base64\n",
        "# NOTE: This command only works on Linux.\n",
        "data=$(base64 -w 0 image.jpg)\n",
        "\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:generateContent \\\n",
        "  -d \"{\n",
        "      'contents': {\n",
        "        'role': 'USER',\n",
        "        'parts': [\n",
        "          {\n",
        "            'text': 'Is it a cat?'\n",
        "          },\n",
        "          {\n",
        "            'inline_data': {\n",
        "              'data': '${data}',\n",
        "              'mime_type':'image/jpeg'\n",
        "            }\n",
        "          }\n",
        "        ]\n",
        "       }\n",
        "     }\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fKr-BklmhjgP"
      },
      "source": [
        "### Generate text from an image on Google Cloud Storage\n",
        "\n",
        "Specify the Cloud Storage URI of the image to include in the prompt. The bucket that stores the file must be in the same Google Cloud project that's sending the request. You must also specify the `mime_type` field. The supported image MIME types include `image/png` and `image/jpeg`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "43pQE3_z3OjG"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "MODEL_ID=\"gemini-1.5-pro\"\n",
        "\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:generateContent \\\n",
        "  -d '{\n",
        "    \"contents\": {\n",
        "      \"role\": \"USER\",\n",
        "      \"parts\": [\n",
        "        {\n",
        "          \"text\": \"Describe this image\"\n",
        "        },\n",
        "        {\n",
        "          \"file_data\": {\n",
        "            \"mime_type\": \"image/png\",\n",
        "            \"file_uri\": \"gs://cloud-samples-data/generative-ai/image/320px-Felis_catus-cat_on_snow.jpg\"\n",
        "          }\n",
        "        }\n",
        "      ]\n",
        "    },\n",
        "    \"generation_config\": {\n",
        "      \"temperature\": 0.2,\n",
        "      \"top_p\": 0.1,\n",
        "      \"top_k\": 16,\n",
        "      \"max_output_tokens\": 2048,\n",
        "      \"candidate_count\": 1,\n",
        "      \"stop_sequences\": []\n",
        "    },\n",
        "    \"safety_settings\": {\n",
        "      \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
        "      \"threshold\": \"BLOCK_LOW_AND_ABOVE\"\n",
        "    }\n",
        "  }'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TVF4vHuBOD8N"
      },
      "source": [
        "### Generate text from a video file\n",
        "\n",
        "Specify the Cloud Storage URI of the video to include in the prompt. The bucket that stores the file must be in the same Google Cloud project that's sending the request. You must also specify the `mime_type` field. The supported MIME types for video include `video/mp4`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F8kS5p0l_uHE"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:generateContent \\\n",
        "  -d \\\n",
        "'{\n",
        "    \"contents\": {\n",
        "      \"role\": \"USER\",\n",
        "      \"parts\": [\n",
        "        {\n",
        "          \"text\": \"Answer the following questions using the video only. What is the profession of the main person? What are the main features of the phone highlighted?Which city was this recorded in?Provide the answer JSON.\"\n",
        "        },\n",
        "        {\n",
        "          \"file_data\": {\n",
        "            \"mime_type\": \"video/mp4\",\n",
        "            \"file_uri\": \"gs://github-repo/img/gemini/multimodality_usecases_overview/pixel8.mp4\"\n",
        "          }\n",
        "        }\n",
        "      ]\n",
        "    }\n",
        "  }'"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "intro_gemini_curl.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}