docs: add samples from python-docs-samples/tasks (#36)

* Add samples for Cloud Tasks [(#1068)](GoogleCloudPlatform/python-docs-samples#1068) * Add samples for Cloud Tasks * Respond to tasks sample review * Update app engine queues samples * Address review feedback * Address review issues and convert pull queue sample to not use API key auth * Reform pull queues to match appengine queues changes to auth, command line input, readme * flake8 and fix comment * Fix Tasks sample test issues. * Remove queue.yaml, now unused * Add required flag on certain arguments to command-line tools * Auto-update dependencies. [(#1116)](GoogleCloudPlatform/python-docs-samples#1116) * Auto-update dependencies. [(#1133)](GoogleCloudPlatform/python-docs-samples#1133) * Auto-update dependencies. * Fix missing http library Change-Id: I99faa600f2f3f1f50f57694fc9835d7f35bda250 * Update Task Queue samples for beta * Use full import to get client * Auto-update dependencies. [(#1186)](GoogleCloudPlatform/python-docs-samples#1186) * Auto-update dependencies. [(#1217)](GoogleCloudPlatform/python-docs-samples#1217) * Added "Open in Cloud Shell" buttons to README files [(#1254)](GoogleCloudPlatform/python-docs-samples#1254) * Add comments and region tags to Cloud Tasks samples [(#1271)](GoogleCloudPlatform/python-docs-samples#1271) * Standardize on CamelCase, reword confusing endpoint name [(#1288)](GoogleCloudPlatform/python-docs-samples#1288) * Auto-update dependencies. [(#1309)](GoogleCloudPlatform/python-docs-samples#1309) * Rename pull to lease and fix name/parent confusion [(#1311)](GoogleCloudPlatform/python-docs-samples#1311) * Updated region tags for pull queue samples [(#1329)](GoogleCloudPlatform/python-docs-samples#1329) * Auto-update dependencies. [(#1320)](GoogleCloudPlatform/python-docs-samples#1320) * Auto-update dependencies. [(#1355)](GoogleCloudPlatform/python-docs-samples#1355) * Auto-update dependencies. [(#1359)](GoogleCloudPlatform/python-docs-samples#1359) * Auto-update dependencies. * Update Cloud Tasks Samples [(#1529)](GoogleCloudPlatform/python-docs-samples#1529) * passing create task * Passing tests * updates to region tags * update region tags [(#1532)](GoogleCloudPlatform/python-docs-samples#1532) * update Tasks Sample for App Engine [(#1541)](GoogleCloudPlatform/python-docs-samples#1541) * update gcloud command for creating queues * deploys and runs * update license * passing tests * Fix run command [(#1563)](GoogleCloudPlatform/python-docs-samples#1563) * Updated library for TTS GA [(#1552)](GoogleCloudPlatform/python-docs-samples#1552) * update gcloud command for Cloud Tasks [(#1566)](GoogleCloudPlatform/python-docs-samples#1566) * update gcloud command * update pull queue command * update pull queue command * Update Cloud Tasks Push Queue Sample [(#1698)](GoogleCloudPlatform/python-docs-samples#1698) * deleted pull queues * updated samples * fix dependency versions * [Cloud Tasks] Move samples to new folder [(#2114)](GoogleCloudPlatform/python-docs-samples#2114) * Move samples to keep consistent with other langauges * Ad system tests as well * [Cloud Tasks] Add task with authentication sample [(#2113)](GoogleCloudPlatform/python-docs-samples#2113) * Add task with authentication sample * Fix linting * Fix linting * Fix spacing * Update tests with service account * Move samples and update READMEs * Update version and linting * Update task sample comments [(#2156)](GoogleCloudPlatform/python-docs-samples#2156) * Update task comments * Update readme * Update queue name * update gcloud [(#2208)](GoogleCloudPlatform/python-docs-samples#2208) * Add protobuf dep and install instructions [(#2250)](GoogleCloudPlatform/python-docs-samples#2250) * Add Migration Guide Snippets for Cloud Tasks [(#2316)](GoogleCloudPlatform/python-docs-samples#2316) * Migration guide * remove app and update migraitonn * snippets for migration guide - tests added * lint * remove print statements * Styling changes * Travis trigger * Update create_http_task.py [(#2187)](GoogleCloudPlatform/python-docs-samples#2187) Updates `create_http_task.py` to have missing `in_seconds` variable * Adds updates for samples profiler ... vision [(#2439)](GoogleCloudPlatform/python-docs-samples#2439) * Update Cloud Tasks library version [(#2516)](GoogleCloudPlatform/python-docs-samples#2516) * Update to new library * update library version * Adds Task name while creating Task [(#2543)](GoogleCloudPlatform/python-docs-samples#2543) * Task Name in Creating HTTP Task with Token [(#2700)](GoogleCloudPlatform/python-docs-samples#2700) * chore(deps): update dependency google-cloud-tasks to v1.5.0 [(#3168)](GoogleCloudPlatform/python-docs-samples#3168) * chore(deps): update dependency googleapis-common-protos to v1.51.0 [(#3171)](GoogleCloudPlatform/python-docs-samples#3171) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [googleapis-common-protos](https://togithub.com/googleapis/googleapis) | minor | `==1.6.0` -> `==1.51.0` | --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * Simplify noxfile setup. [(#2806)](GoogleCloudPlatform/python-docs-samples#2806) * chore(deps): update dependency requests to v2.23.0 * Simplify noxfile and add version control. * Configure appengine/standard to only test Python 2.7. * Update Kokokro configs to match noxfile. * Add requirements-test to each folder. * Remove Py2 versions from everything execept appengine/standard. * Remove conftest.py. * Remove appengine/standard/conftest.py * Remove 'no-sucess-flaky-report' from pytest.ini. * Add GAE SDK back to appengine/standard tests. * Fix typo. * Roll pytest to python 2 version. * Add a bunch of testing requirements. * Remove typo. * Add appengine lib directory back in. * Add some additional requirements. * Fix issue with flake8 args. * Even more requirements. * Readd appengine conftest.py. * Add a few more requirements. * Even more Appengine requirements. * Add webtest for appengine/standard/mailgun. * Add some additional requirements. * Add workaround for issue with mailjet-rest. * Add responses for appengine/standard/mailjet. Co-authored-by: Renovate Bot <[email protected]> * Remove dependency googleapis-common-protos [(#3955)](GoogleCloudPlatform/python-docs-samples#3955) * Update dependency googleapis-common-protos to v1.52.0 * Update requirements.txt * Update requirements.txt * Update requirements.txt Co-authored-by: Bu Sun Kim <[email protected]> * Replace GCLOUD_PROJECT with GOOGLE_CLOUD_PROJECT. [(#4022)](GoogleCloudPlatform/python-docs-samples#4022) * [tasks] testing: use fixtures for the queue [(#4049)](GoogleCloudPlatform/python-docs-samples#4049) fixes #4045 fixes #4044 I don't know why these tests started to fail, but anyways we'd better use fixtures and temporary queues. * add python snippets and tests for creating, listing, and deleting queues [(#4012)](GoogleCloudPlatform/python-docs-samples#4012) * add python snippets and tests for creating, listing, and deleting queues * fix grammar * update licenses * apply suggested fixes and format with black * refine delete_queue_test with fixture for setup * utilize fixtures and match format of create_http_task_test * utilize fixtures in list_queues_test and create_queue_test * make create_queue_test call the right function * still attempt to delete queue after test runs in case of failure * attempt to delete queue in case of failure, using try/except approach * add print when NotFound is caught * fix import Co-authored-by: Averi Kitsch <[email protected]> Co-authored-by: Takashi Matsuo <[email protected]> * docs(tasks): service_account_email parameter example [(#4183)](GoogleCloudPlatform/python-docs-samples#4183) * docs(tasks): serviceAccountEmail parameter example * docs(tasks): parameter name camel => snake case * docs(tasks): comment order = function param order * chore(deps): update dependency pytest to v5.4.3 [(#4279)](GoogleCloudPlatform/python-docs-samples#4279) * chore(deps): update dependency pytest to v5.4.3 * specify pytest for python 2 in appengine Co-authored-by: Leah Cole <[email protected]> * Update dependency pytest to v6 [(#4390)](GoogleCloudPlatform/python-docs-samples#4390) * tasks: added json content-type request [(#4473)](GoogleCloudPlatform/python-docs-samples#4473) - added json payload compatibility - fix imports and code block used on https://cloud.google.com/tasks/docs/creating-http-target-tasks#python ## Description Fixes #<ISSUE-NUMBER> Note: It's a good idea to open an issue first for discussion. ## Checklist - [ ] I have followed [Sample Guidelines from AUTHORING_GUIDE.MD](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md) - [ ] README is updated to include [all relevant information](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#readme-file) - [ ] **Tests** pass: `nox -s py-3.6` (see [Test Environment Setup](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#test-environment-setup)) - [ ] **Lint** pass: `nox -s lint` (see [Test Environment Setup](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#test-environment-setup)) - [ ] These samples need a new **API enabled** in testing projects to pass (let us know which ones) - [ ] These samples need a new/updated **env vars** in testing projects set to pass (let us know which ones) - [ ] Please **merge** this PR for me once it is approved. - [ ] This sample adds a new sample directory, and I updated the [CODEOWNERS file](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/.github/CODEOWNERS) with the codeowners for this sample * docs: add samples from python-docs-samples/tasks * chore: update templates Co-authored-by: Andrew Gorcester <[email protected]> Co-authored-by: Andrew Gorcester <[email protected]> Co-authored-by: DPE bot <[email protected]> Co-authored-by: michaelawyu <[email protected]> Co-authored-by: ellenevans <[email protected]> Co-authored-by: Averi Kitsch <[email protected]> Co-authored-by: michaelawyu <[email protected]> Co-authored-by: Noah Negrey <[email protected]> Co-authored-by: Alex Voorhees <[email protected]> Co-authored-by: Gus Class <[email protected]> Co-authored-by: Sarath Kaul <[email protected]> Co-authored-by: Sarath Kaul <[email protected]> Co-authored-by: WhiteSource Renovate <[email protected]> Co-authored-by: Kurtis Van Gent <[email protected]> Co-authored-by: Bu Sun Kim <[email protected]> Co-authored-by: Takashi Matsuo <[email protected]> Co-authored-by: Aaron Johnson <[email protected]> Co-authored-by: Adam Ross <[email protected]> Co-authored-by: Leah Cole <[email protected]> Co-authored-by: Joab Leite S. Neto <[email protected]>
googleapis · Aug 31, 2020 · 9d022f7 · 9d022f7
1 parent a88e42b
commit 9d022f7
Show file tree

Hide file tree

Showing 30 changed files with 1,709 additions and 27 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -0,0 +1,8 @@
+# Code owners file.
+# This file controls who is tagged for review for any given pull request.
+#
+# For syntax help see:
+# https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners#codeowners-syntax
+
+
+/samples/**/*.py                           @averikitsch @googleapis/python-samples-owners
diff --git a/.gitignore b/.gitignore
@@ -46,6 +46,7 @@ pip-log.txt
 # Built documentation
 docs/_build
 bigquery/docs/generated
+docs.metadata
 
 # Virtual environment
 env/
@@ -57,4 +58,4 @@ system_tests/local_test_setup
 
 # Make sure a generated file isn't accidentally committed.
 pylintrc
-pylintrc.test
+pylintrc.test
diff --git a/.kokoro/build.sh b/.kokoro/build.sh
@@ -36,4 +36,10 @@ python3.6 -m pip uninstall --yes --quiet nox-automation
 python3.6 -m pip install --upgrade --quiet nox
 python3.6 -m nox --version
 
-python3.6 -m nox
+# If NOX_SESSION is set, it only runs the specified session,
+# otherwise run all the sessions.
+if [[ -n "${NOX_SESSION:-}" ]]; then
+    python3.6 -m nox -s "${NOX_SESSION:-}"
+else
+    python3.6 -m nox
+fi
diff --git a/.kokoro/docker/docs/Dockerfile b/.kokoro/docker/docs/Dockerfile
@@ -0,0 +1,98 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ubuntu:20.04
+
+ENV DEBIAN_FRONTEND noninteractive
+
+# Ensure local Python is preferred over distribution Python.
+ENV PATH /usr/local/bin:$PATH
+
+# Install dependencies.
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends \
+    apt-transport-https \
+    build-essential \
+    ca-certificates \
+    curl \
+    dirmngr \
+    git \
+    gpg-agent \
+    graphviz \
+    libbz2-dev \
+    libdb5.3-dev \
+    libexpat1-dev \
+    libffi-dev \
+    liblzma-dev \
+    libreadline-dev \
+    libsnappy-dev \
+    libssl-dev \
+    libsqlite3-dev \
+    portaudio19-dev \
+    redis-server \
+    software-properties-common \
+    ssh \
+    sudo \
+    tcl \
+    tcl-dev \
+    tk \
+    tk-dev \
+    uuid-dev \
+    wget \
+    zlib1g-dev \
+  && add-apt-repository universe \
+  && apt-get update \
+  && apt-get -y install jq \
+  && apt-get clean autoclean \
+  && apt-get autoremove -y \
+  && rm -rf /var/lib/apt/lists/* \
+  && rm -f /var/cache/apt/archives/*.deb
+
+
+COPY fetch_gpg_keys.sh /tmp
+# Install the desired versions of Python.
+RUN set -ex \
+    && export GNUPGHOME="$(mktemp -d)" \
+    && echo "disable-ipv6" >> "${GNUPGHOME}/dirmngr.conf" \
+    && /tmp/fetch_gpg_keys.sh \
+    && for PYTHON_VERSION in 3.7.8 3.8.5; do \
+        wget --no-check-certificate -O python-${PYTHON_VERSION}.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz" \
+        && wget --no-check-certificate -O python-${PYTHON_VERSION}.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc" \
+        && gpg --batch --verify python-${PYTHON_VERSION}.tar.xz.asc python-${PYTHON_VERSION}.tar.xz \
+        && rm -r python-${PYTHON_VERSION}.tar.xz.asc \
+        && mkdir -p /usr/src/python-${PYTHON_VERSION} \
+        && tar -xJC /usr/src/python-${PYTHON_VERSION} --strip-components=1 -f python-${PYTHON_VERSION}.tar.xz \
+        && rm python-${PYTHON_VERSION}.tar.xz \
+        && cd /usr/src/python-${PYTHON_VERSION} \
+        && ./configure \
+            --enable-shared \
+            # This works only on Python 2.7 and throws a warning on every other
+            # version, but seems otherwise harmless.
+            --enable-unicode=ucs4 \
+            --with-system-ffi \
+            --without-ensurepip \
+        && make -j$(nproc) \
+        && make install \
+        && ldconfig \
+  ; done \
+  && rm -rf "${GNUPGHOME}" \
+  && rm -rf /usr/src/python* \
+  && rm -rf ~/.cache/
+
+RUN wget -O /tmp/get-pip.py 'https://bootstrap.pypa.io/get-pip.py' \
+  && python3.7 /tmp/get-pip.py \
+  && python3.8 /tmp/get-pip.py \
+  && rm /tmp/get-pip.py
+
+CMD ["python3.7"]
diff --git a/.kokoro/docker/docs/fetch_gpg_keys.sh b/.kokoro/docker/docs/fetch_gpg_keys.sh
@@ -0,0 +1,45 @@
+#!/bin/bash
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# A script to fetch gpg keys with retry.
+# Avoid jinja parsing the file.
+# 
+
+function retry {
+    if [[ "${#}" -le 1 ]]; then
+	echo "Usage: ${0} retry_count commands.."
+	exit 1
+    fi
+    local retries=${1}
+    local command="${@:2}"
+    until [[ "${retries}" -le 0 ]]; do
+	$command && return 0
+	if [[ $? -ne 0 ]]; then
+	    echo "command failed, retrying"
+	    ((retries--))
+	fi
+    done
+    return 1
+}
+
+# 3.6.9, 3.7.5 (Ned Deily)
+retry 3 gpg --keyserver ha.pool.sks-keyservers.net --recv-keys \
+      0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D
+
+# 3.8.0 (Łukasz Langa)
+retry 3 gpg --keyserver ha.pool.sks-keyservers.net --recv-keys \
+      E3FF2839C048B25C084DEBE9B26995E310250568
+
+# 
diff --git a/.kokoro/docs/common.cfg b/.kokoro/docs/common.cfg
@@ -11,12 +11,12 @@ action {
 gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
 
 # Use the trampoline script to run in docker.
-build_file: "python-tasks/.kokoro/trampoline.sh"
+build_file: "python-tasks/.kokoro/trampoline_v2.sh"
 
 # Configure the docker image for kokoro-trampoline.
 env_vars: {
     key: "TRAMPOLINE_IMAGE"
-    value: "gcr.io/cloud-devrel-kokoro-resources/python-multi"
+    value: "gcr.io/cloud-devrel-kokoro-resources/python-lib-docs"
 }
 env_vars: {
     key: "TRAMPOLINE_BUILD_FILE"
@@ -28,6 +28,23 @@ env_vars: {
     value: "docs-staging"
 }
 
+env_vars: {
+    key: "V2_STAGING_BUCKET"
+    value: "docs-staging-v2-staging"
+}
+
+# It will upload the docker image after successful builds.
+env_vars: {
+    key: "TRAMPOLINE_IMAGE_UPLOAD"
+    value: "true"
+}
+
+# It will always build the docker image.
+env_vars: {
+    key: "TRAMPOLINE_DOCKERFILE"
+    value: ".kokoro/docker/docs/Dockerfile"
+}
+
 # Fetch the token needed for reporting release status to GitHub
 before_action {
   fetch_keystore {

diff --git a/.kokoro/docs/docs-presubmit.cfg b/.kokoro/docs/docs-presubmit.cfg
@@ -0,0 +1,17 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "STAGING_BUCKET"
+    value: "gcloud-python-test"
+}
+
+env_vars: {
+    key: "V2_STAGING_BUCKET"
+    value: "gcloud-python-test"
+}
+
+# We only upload the image in the main `docs` build.
+env_vars: {
+    key: "TRAMPOLINE_IMAGE_UPLOAD"
+    value: "false"
+}
diff --git a/.kokoro/publish-docs.sh b/.kokoro/publish-docs.sh
@@ -18,26 +18,16 @@ set -eo pipefail
 # Disable buffering, so that the logs stream through.
 export PYTHONUNBUFFERED=1
 
-cd github/python-tasks
-
-# Remove old nox
-python3.6 -m pip uninstall --yes --quiet nox-automation
+export PATH="${HOME}/.local/bin:${PATH}"
 
 # Install nox
-python3.6 -m pip install --upgrade --quiet nox
-python3.6 -m nox --version
+python3 -m pip install --user --upgrade --quiet nox
+python3 -m nox --version
 
 # build docs
 nox -s docs
 
-python3 -m pip install gcp-docuploader
-
-# install a json parser
-sudo apt-get update
-sudo apt-get -y install software-properties-common
-sudo add-apt-repository universe
-sudo apt-get update
-sudo apt-get -y install jq
+python3 -m pip install --user gcp-docuploader
 
 # create metadata
 python3 -m docuploader create-metadata \
@@ -52,4 +42,23 @@ python3 -m docuploader create-metadata \
 cat docs.metadata
 
 # upload docs
-python3 -m docuploader upload docs/_build/html --metadata-file docs.metadata --staging-bucket docs-staging
+python3 -m docuploader upload docs/_build/html --metadata-file docs.metadata --staging-bucket "${STAGING_BUCKET}"
+
+
+# docfx yaml files
+nox -s docfx
+
+# create metadata.
+python3 -m docuploader create-metadata \
+  --name=$(jq --raw-output '.name // empty' .repo-metadata.json) \
+  --version=$(python3 setup.py --version) \
+  --language=$(jq --raw-output '.language // empty' .repo-metadata.json) \
+  --distribution-name=$(python3 setup.py --name) \
+  --product-page=$(jq --raw-output '.product_documentation // empty' .repo-metadata.json) \
+  --github-repository=$(jq --raw-output '.repo // empty' .repo-metadata.json) \
+  --issue-tracker=$(jq --raw-output '.issue_tracker // empty' .repo-metadata.json)
+
+cat docs.metadata
+
+# upload docs
+python3 -m docuploader upload docs/_build/html/docfx_yaml --metadata-file docs.metadata --destination-prefix docfx --staging-bucket "${V2_STAGING_BUCKET}"