Optimizing publish from the core-tools

[asavaritayal]

A consistent theme across private preview has been the extended time taken to publish an app from the core-tools.

Current Workflow

Behind the scenes the publish command will: InternalPreparePythonDeployment Method

1. Pull and run microsoft/azure-functions-python3.6:v2.0.11651-alpha
2. Copy your python functions and requirements.txt to the running container
3. In the container, create a venv for your project named worker_venv under /home/site/wwwroot
4. Run pip install -r requirements.txt in the container
5. Package up your functions and venv
6. Copy package outside of container

Proposed Workflow

1. Create a venv for the project named worker_env. Activate it.

2. For packages in requirements.txt, determine if they are pure .py packages or c extensions with universal wheels available on PyPI. Install these packages to worker_env using pip.

3. If requirements.txt contains c extension packages that do not have universal wheels on PyPI,

   • Pull and run microsoft/azure-functions-python3.6:v2.0.11651-alpha
   • Copy worker_env and requirements.txt to the running container under /home/site/wwwroot
   • Pip install the remaining packages
   • Copy worker_env outside the container

4. Package the python functions and worker_env together as a zip file for deployment to Azure

Notes

Currently, there are no wheels available for azure.functions and azure.worker. This is related to #118

A few related issues:

• Missing tag error while deploying functions with core tools azure-functions-core-tools#484
• Recloning from git every time during consequent deployments azure-functions-core-tools#483
• Initializing python project references a worker brancher/commit that doesn't exist azure-functions-core-tools#476
• grpcio use binary wheels in docker install azure-functions-core-tools#480
• Optimizing the publish gesture for Python azure-functions-core-tools#466

[asavaritayal]

/cc @pragnagopa , @priyaananthasankar

@elprans Here's a brain dump. I'll let you take it from here.

[priyaananthasankar]

Adding another suggestion/question to the proposed workflow: When you pip install pure .py packages subsequently after a first installation - docker has a way to figure if it is a re-install or a fresh install, through a build context. Or maybe provide a flag if docker needs to forget and consider a re-install. Noticed that irrespective of first time or subsequent times, all the python packages are again installed. Would be great if this can be fixed.

I think when publishing anything for the first time, we are prepared for a decent latency. But subsequent publish - like changing a line of code and redeploying etc should be much faster than a first publish.

[aluong]

Why is the docker container even necessary if we are doing a pip install outside the container already? Could you just zip up the current folder and zipdeploy it to the function app?

[asavaritayal]

@aluong you're right, the docker step is not necessary. In the optimized workflow, step #3 would only be triggered if you're using a c extension package that does not have pre-built wheels available on PyPI. I don't have the exact data available but expect it to be an uncommon scenario.

The whole idea behind using docker is to build the right binaries for our production environment. The alternate to docker would be to use a Linux based CI server to build these binaries. The later workflow is on our backlog but further down the pipeline.

[ahmelsayed]

@elprans So I'm stuck because of this issue pypa/pip#4390

I was testing with pip install -r req.txt -t .packages

I was thinking I'd install the packages in a folder .packages and then run python with PYTHONPATH=$PYTHONPATH:/home/site/wwwroot/.packages python worker.py

However it looks like -e is not compatible with -t.

[elprans]

@ahmelsayed You don't need -e. In fact, installing the worker with -e creates conflicts with azure-nspkg due to how the latter is packaged.

[ahmelsayed]

bear with me since I'm not very familiar with the python eco system :)

without -e, when I do pip freeze I get this

azure-functions-python==1.0.0a2.dev0
grpcio==1.9.1
grpcio-tools==1.9.1
protobuf==3.5.2.post1
six==1.11.0

as oppose to

-e git+https://github.com/Azure/azure-functions-python-worker.git@042f934e7af3af44794b36f34d15bc764d96a32a#egg=azure_functions_python
...

and when you do pip install -r requirements.txt on the former you get this error

 Could not find a version that satisfies the requirement azure-functions-python==1.0.0a2.dev0 (from -r requirements.txt (line 1)) (from versions: )
No matching distribution found for azure-functions-python==1.0.0a2.dev0 (from -r requirements.txt (line 1))

[elprans]

Until we publish the worker on PyPi you need to install the worker from the git checkout, just without -e:

$ git clone https://github.com/Azure/azure-functions-python-worker.git
$ cd azure-functions-python-worker
$ pip install .

[elprans]

@ahmelsayed @asavaritayal I decided to build a standalone Python module to create Python app packages for Azure. It would take the requirements.txt and generate the correct app.zip:

azure-functions-python-packer -r requirements.txt --platform linux --python-version 36 --output app.zip

core-tools would then simply need to add the above call.

[asavaritayal]

@ahmelsayed @asavaritayal I decided to build a standalone Python module to create Python app packages for Azure. It would take the requirements.txt and generate the correct app.zip:

How long does the azure-functions-python-packer module take to install the base packages, such as in the requirements.txt below? Will it only download the wheels?

azure-functions-python==1.0.0a2.dev0
grpcio==1.9.1
grpcio-tools==1.9.1
protobuf==3.5.2.post1
six==1.11.0

[elprans]

The packer will download wheels if available, and if not, it will try to build them locally (using Docker for binary wheels).

Regarding the base requirements: why do we have to install them every time? Can these be baked into the base image? In fact, the docker-less deployment relies on the virtual Python environment already being present in the image.

[asavaritayal]

The azure.functions and azure.functions.worker packages are now published to PyPI -

https://pypi.org/project/azure-functions/

pip install azure-functions

https://pypi.org/project/azure-functions-worker/

pip install azure-functions-worker

/cc @elprans

[ahmelsayed]

I'm having issues trying to get packapp.py script to work.

First I made a small change to copy the output to a folder under the current project .python_packages. My idea is that I can package that folder with the app, then add that to PYTHONPATH before running the worker in the docker image

It works fine with the cli on my machine, but it keeps erroring out with

Worker encountered an error.
System.TimeoutException: The operation has timed out.

[elprans]

Do you have steps to reproduce?

[ahmelsayed]

# Create a new empty venv
$ python3 -m venv test_venv
$ source test_venv/bin/activate

# download my modified packapp.py
$ wget https://raw.githubusercontent.com/Azure/azure-functions-python-worker/1fb57ee25bcfac3026ae3e42f6f2a13bc3e5cced/azure/worker/tools/packapp.py

# install latest dislib
$ pip install -U 'git+https://github.com/vsajip/distlib.git@15dba58a827f56195b0fa0afe80a8925a92e8bf5'

$ git clone https://github.com/ahmelsayed/python-functions
$ cd python-functions

# this should create a .python_packages in the current directory
$ python ../packapp.py  --platform linux --python-version 36 .

# The Dockerfile just copies everything under /home/site/wwwroot
$ sudo docker build -t docker-python .

$ sudo docker run -it -p 8080:80 docker-python
$ curl http://localhost:8080/api/HttpTriggerPython

[ahmelsayed]

But make sure to wait until this image is done building https://hub.docker.com/r/microsoft/azure-functions-python3.6/builds/

[elprans]

Works on my machine with the above steps.

[ahmelsayed]

hmmm, it works on my home machine as well. Let me try again in Azure. I'm not sure what might be wrong with my machine at work

[ahmelsayed]

running packapp.py --platform linux --python-version 36 on this req.txt on Windows (works fine on Linux)

azure-functions==1.0.0a1
azure-functions-worker==1.0.0a3
grpcio==1.12.1
grpcio-tools==1.12.1
protobuf==3.6.0
six==1.11.0
requests
tensorflow==1.5.0
pillow==5.0.0
scipy==1.0.0
numpy==1.14.1
flask==0.12.2

results in this error

There was an error restoring dependencies.  Could not find a version that satisfies the requirement tensorflow==1.5.0 (from -r C:\temp\customvision\requirements.txt (line 8)) (from versions: )

No matching distribution found for tensorflow==1.5.0 (from -r C:\temp\customvision\requirements.txt (line 8))
You are using pip version 9.0.3, however version 10.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "C:\users\ahmels\Downloads\cli\tools\python\packapp.py", line 210, in <module>
    main()
  File "C:\users\ahmels\Downloads\cli\tools\python\packapp.py", line 38, in main
    find_and_build_deps(args)
  File "C:\users\ahmels\Downloads\cli\tools\python\packapp.py", line 57, in find_and_build_deps
    ], check=True)
  File "C:\Users\ahmels\AppData\Local\Programs\Python\Python36-32\lib\subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pip', 'download', '-r', 'C:\\temp\\customvision\\requirements.txt', '--dest', 'C:\\Users\\ahmels\\AppData\\Local\\Temp\\azureworker3us7_xs6']' returned non-zero exit status 1.

[elprans]

@ahmelsayed Should we move this script to a different repo to make improvements and fixes more straightforward?

[ahmelsayed]

yes, I actually added it here https://github.com/Azure/azure-functions-core-tools/blob/master/tools/python/packapp.py to be included in the cli build

with the same change I mentioned earlier (not zipping) and adding a --packages-dir-name

[asavaritayal]

Thanks everyone for the feedback. The new version of the core tools - 2.0.1-beta.33 contains improvements to the publishing workflow (thanks @ahmelsayed @elprans ). Please update to the latest version and give it a try again.

Note that the quick start instructions have changed in the last couple of weeks - https://github.com/Azure/azure-functions-python-worker/wiki/Create-Function-(CLI)

If you're not starting over from scratch, confirm that the requirements.txt file contains the following list of dependencies:

azure-functions==1.0.0a1
azure-functions-worker==1.0.0a3
grpcio==1.12.1
grpcio-tools==1.12.1
protobuf==3.6.0
six==1.11.0