Add pythonSrcRoot option to python servers (aiohttp/flask/blueplanet)…

… to support src/ layout projects [and reenable/fix all python server tests] (OpenAPITools#5423)

* python server: Add pythonSrcRoot option

This will allow the python project to be in a subdirectory (as specified
in pythonSrcRoot). That could mean following the src layout with sources
under src/ or lib/. Multi-language projects might use a sub directory
like python/, or whatever makes sense for the project.

By default, the pythonSrcRoot is "", meaning the existing behavior is
the default.

* python server: update template files to support pythonSrcRoot

* python server: update docs to add pythonSrcRoot option

* python server: add pythonSrcRoot sample script

* python server: build sample srclayout project

* [Python] copy test files preserving history

* [Python] Make a conflict to preserve file copy history

* [Python] customize pom.xml for src layout tests

* [Python] add python-aiohttp-srclayout tests

* [Python] Fix server tests by updating requirements

Reverts the PR that disabled python2 server tests:
OpenAPITools#4949

Reverts commits that disabled python3 server tests:
9adfedb
17ee990

Issue about the python 3 tests:
OpenAPITools#5235

I couldn't find an issue about the python2 tests being disabled.
I'm guessing build errors like the following were the trigger:
https://travis-ci.org/github/OpenAPITools/openapi-generator/builds/634238181

Let's see what breaks!

* [Python] Copy setup.py to python-aiohttp

* [Python] Save history while copying setup.py to python-aiohttp

* [Python] Add aiohttp server setup.py

* [Python] Fix python server tests with src layout

* [Python] bump Flask-Testing version

* [Python] Pin pyyaml for py2.7 flask server

* [Python] simplify flask server requirements

* consolidate server tests

* [Python] rebuild python server samples

* [Python] Fix python server requirements for older pythons

Documented minimum python version for our aiohttp server is 3.5.
Documented minimum python version for our flask server is 3.4.
Connexion 2.3 is the last version to support python 3.4 and 3.5, so fix
the version pinning to correctly select <=2.3 for these EOL python
versions. Newer pythons should get the latest if possible as there many
relevant bug fixes.

Werkzeug also needs to be pinned for these old versions in the aiohttp
server just like for the flask server.

3.4 and 3.5 are EOL. We really should increase the minimum supported
version, but that is for another PR to do.

bin/python-server-aiohttp-srclayout-petstore.sh

@@ -0,0 +1,44 @@
+#!/bin/sh
+
+SCRIPT="$0"
+echo "# START SCRIPT: $SCRIPT"
+
+while [ -h "$SCRIPT" ] ; do
+  ls=`ls -ld "$SCRIPT"`
+  link=`expr "$ls" : '.*-> \(.*\)$'`
+  if expr "$link" : '/.*' > /dev/null; then
+    SCRIPT="$link"
+  else
+    SCRIPT=`dirname "$SCRIPT"`/"$link"
+  fi
+done
+
+if [ ! -d "${APP_DIR}" ]; then
+  APP_DIR=`dirname "$SCRIPT"`/..
+  APP_DIR=`cd "${APP_DIR}"; pwd`
+fi
+
+executable="./modules/openapi-generator-cli/target/openapi-generator-cli.jar"
+
+if [ ! -f "$executable" ]
+then
+  mvn -B clean package
+fi
+
+generator=python-aiohttp
+input=modules/openapi-generator/src/test/resources/2_0/petstore.yaml
+out_folder=samples/server/petstore/${generator}-srclayout
+resources=modules/openapi-generator/src/main/resources/${generator}
+
+# if you've executed sbt assembly previously it will use that instead.
+export JAVA_OPTS="${JAVA_OPTS} -Xmx1024M -DloggerPath=conf/log4j.properties -Dservice"
+ags="generate -t $resources -i $input -g $generator -o $out_folder --additional-properties pythonSrcRoot=src $@"
+
+rm -rf $out_folder/.openapi*
+rm -rf $out_folder/openapi_server
+rm -rf $out_folder/tests*
+rm $out_folder/README.md
+rm $out_folder/requirements.txt
+rm $out_folder/test-requirements.txt
+
+java $JAVA_OPTS -jar $executable $ags

bin/python-server-all.sh

@@ -1,6 +1,7 @@
 #!/bin/sh
 
 ./bin/python-server-aiohttp-petstore.sh
+./bin/python-server-aiohttp-srclayout-petstore.sh
 ./bin/python-server-flask-petstore.sh
 ./bin/python-server-flask-petstore-python2.sh
 ./bin/python-server-blueplanet-petstore.sh

docs/generators/python-aiohttp.md

@@ -12,6 +12,7 @@ sidebar_label: python-aiohttp
 |packageName|python package name (convention: snake_case).| |openapi_server|
 |packageVersion|python package version.| |1.0.0|
 |prependFormOrBodyParameters|Add form or body parameters to the beginning of the parameter list.| |false|
+|pythonSrcRoot|put python sources in this subdirectory of output folder (defaults to "" for). Use this for src/ layout.| ||
 |serverPort|TCP port to listen to in app.run| |8080|
 |sortModelPropertiesByRequiredFlag|Sort model properties to place required parameters before optional parameters.| |true|
 |sortParamsByRequiredFlag|Sort method arguments to place required parameters before optional parameters.| |true|

docs/generators/python-blueplanet.md

@@ -12,6 +12,7 @@ sidebar_label: python-blueplanet
 |packageName|python package name (convention: snake_case).| |openapi_server|
 |packageVersion|python package version.| |1.0.0|
 |prependFormOrBodyParameters|Add form or body parameters to the beginning of the parameter list.| |false|
+|pythonSrcRoot|put python sources in this subdirectory of output folder (defaults to "" for). Use this for src/ layout.| ||
 |serverPort|TCP port to listen to in app.run| |8080|
 |sortModelPropertiesByRequiredFlag|Sort model properties to place required parameters before optional parameters.| |true|
 |sortParamsByRequiredFlag|Sort method arguments to place required parameters before optional parameters.| |true|

docs/generators/python-flask.md

@@ -12,6 +12,7 @@ sidebar_label: python-flask
 |packageName|python package name (convention: snake_case).| |openapi_server|
 |packageVersion|python package version.| |1.0.0|
 |prependFormOrBodyParameters|Add form or body parameters to the beginning of the parameter list.| |false|
+|pythonSrcRoot|put python sources in this subdirectory of output folder (defaults to "" for). Use this for src/ layout.| ||
 |serverPort|TCP port to listen to in app.run| |8080|
 |sortModelPropertiesByRequiredFlag|Sort model properties to place required parameters before optional parameters.| |true|
 |sortParamsByRequiredFlag|Sort method arguments to place required parameters before optional parameters.| |true|

modules/openapi-generator/src/main/java/org/openapitools/codegen/languages/PythonAbstractConnexionServerCodegen.java

@@ -51,6 +51,7 @@ public class PythonAbstractConnexionServerCodegen extends DefaultCodegen impleme
     public static final String SUPPORT_PYTHON2 = "supportPython2";
     // nose is a python testing framework, we use pytest if USE_NOSE is unset
     public static final String USE_NOSE = "useNose";
+    public static final String PYTHON_SRC_ROOT = "pythonSrcRoot";
     static final String MEDIA_TYPE = "mediaType";
 
     protected int serverPort = 8080;
@@ -61,6 +62,7 @@ public class PythonAbstractConnexionServerCodegen extends DefaultCodegen impleme
     protected Map<Character, String> regexModifiers;
     protected boolean fixBodyName;
     protected boolean useNose = Boolean.FALSE;
+    protected String pythonSrcRoot;
 
     public PythonAbstractConnexionServerCodegen(String templateDirectory, boolean fixBodyNameValue) {
         super();
@@ -165,6 +167,8 @@ public PythonAbstractConnexionServerCodegen(String templateDirectory, boolean fi
                 defaultValue("8080"));
         cliOptions.add(CliOption.newBoolean(USE_NOSE, "use the nose test framework").
                 defaultValue(Boolean.FALSE.toString()));
+        cliOptions.add(new CliOption(PYTHON_SRC_ROOT, "put python sources in this subdirectory of output folder (defaults to \"\" for). Use this for src/ layout.").
+                defaultValue(""));
     }
 
     protected void addSupportingFiles() {
@@ -212,6 +216,12 @@ public void processOpts() {
         if (additionalProperties.containsKey(USE_NOSE)) {
             setUseNose((String) additionalProperties.get(USE_NOSE));
         }
+        if (additionalProperties.containsKey(PYTHON_SRC_ROOT)) {
+            setPythonSrcRoot((String) additionalProperties.get(PYTHON_SRC_ROOT));
+            additionalProperties.put(PYTHON_SRC_ROOT, pythonSrcRoot);
+        } else {
+            setPythonSrcRoot("");
+        }
         supportingFiles.add(new SupportingFile("__main__.mustache", packagePath(), "__main__.py"));
         supportingFiles.add(new SupportingFile("util.mustache", packagePath(), "util.py"));
         supportingFiles.add(new SupportingFile("typing_utils.mustache", packagePath(), "typing_utils.py"));
@@ -230,6 +240,25 @@ public void setUseNose(String val) {
         this.useNose = Boolean.valueOf(val);
     }
 
+    public void setPythonSrcRoot(String val) {
+        String pySrcRoot;
+        if (val == null) {
+            pySrcRoot = "";
+        } else {
+            pySrcRoot = val.replaceAll("[/\\\\]+$", "");
+        }
+
+        if (pySrcRoot.isEmpty() || pySrcRoot == ".") {
+            this.pythonSrcRoot = "";
+        } else {
+            this.pythonSrcRoot = pySrcRoot + File.separator;
+        }
+    }
+
+    public String pythonSrcOutputFolder() {
+        return outputFolder + File.separator + pythonSrcRoot;
+    }
+
     private static String packageToPath(String pkg) {
         return pkg.replace(".", File.separator);
     }
@@ -304,7 +333,14 @@ public String escapeReservedWord(String name) {
      */
     @Override
     public String apiFileFolder() {
-        return outputFolder + File.separator + apiPackage().replace('.', File.separatorChar);
+        String pkgPath = apiPackage().replace('.', File.separatorChar);
+        return pythonSrcOutputFolder() + pkgPath;
+    }
+
+    @Override
+    public String modelFileFolder() {
+        String pkgPath = modelPackage().replace('.', File.separatorChar);
+        return pythonSrcOutputFolder() + pkgPath;
     }
 
     @Override
@@ -799,7 +835,8 @@ public void setPackageVersion(String packageVersion) {
     }
 
     public String packagePath() {
-        return packageName.replace('.', File.separatorChar);
+        String pkgPath = packageName.replace('.', File.separatorChar);
+        return pythonSrcRoot + pkgPath;
     }
 
     @Override

modules/openapi-generator/src/main/java/org/openapitools/codegen/languages/PythonAiohttpConnexionServerCodegen.java

@@ -71,6 +71,7 @@ protected void addSupportingFiles() {
         supportingFiles.add(new SupportingFile("conftest.mustache", testPackage, "conftest.py"));
         supportingFiles.add(new SupportingFile("__init__test.mustache", testPackage, "__init__.py"));
         supportingFiles.add(new SupportingFile("__init__main.mustache", packagePath(), "__init__.py"));
+        supportingFiles.add(new SupportingFile("setup.mustache", "", "setup.py"));
         supportingFiles.add(new SupportingFile("tox.mustache", "", "tox.ini"));
         supportingFiles.add(new SupportingFile("gitignore.mustache", "", ".gitignore"));
     }

modules/openapi-generator/src/main/resources/python-aiohttp/conftest.mustache

@@ -11,7 +11,8 @@ def client(loop, aiohttp_client):
     options = {
         "swagger_ui": True
         }
-    specification_dir = os.path.join(os.path.dirname(__file__), '..',
+    specification_dir = os.path.join(os.path.dirname(__file__), '..',{{#pythonSrcRoot}}
+                                     "{{{.}}}",{{/pythonSrcRoot}}
                                      '{{packageName}}',
                                      'openapi')
     app = connexion.AioHttpApp(__name__, specification_dir=specification_dir,

modules/openapi-generator/src/main/resources/python-aiohttp/requirements.mustache

@@ -1,3 +1,9 @@
-connexion[aiohttp,swagger-ui] == 2.0.2
-swagger-ui-bundle == 0.0.2
-aiohttp_jinja2 == 1.1.0
+connexion[aiohttp,swagger-ui] >= 2.6.0; python_version>="3.6"
+# 2.3 is the last version that supports python 3.5
+connexion[aiohttp,swagger-ui] <= 2.3.0; python_version=="3.5" or python_version=="3.4"
+# connexion requires werkzeug but connexion < 2.4.0 does not install werkzeug
+# we must peg werkzeug versions below to fix connexion
+# https://github.com/zalando/connexion/pull/1044
+werkzeug == 0.16.1; python_version=="3.5" or python_version=="3.4"
+swagger-ui-bundle == 0.0.6
+aiohttp_jinja2 == 1.2.0

modules/openapi-generator/src/main/resources/python-aiohttp/setup.mustache

@@ -0,0 +1,40 @@
+# coding: utf-8
+
+import sys
+from setuptools import setup, find_packages
+
+NAME = "{{packageName}}"
+VERSION = "{{packageVersion}}"
+{{#apiInfo}}{{#apis}}{{^hasMore}}
+# To install the library, run the following
+#
+# python setup.py install
+#
+# prerequisite: setuptools
+# http://pypi.python.org/pypi/setuptools
+
+REQUIRES = [
+    "connexion==2.6.0",
+    "swagger-ui-bundle==0.0.6",
+    "aiohttp_jinja2==1.2.0",
+]
+
+setup(
+    name=NAME,
+    version=VERSION,
+    description="{{appName}}",
+    author_email="{{infoEmail}}",
+    url="{{packageUrl}}",
+    keywords=["OpenAPI", "{{appName}}"],
+    install_requires=REQUIRES,
+    packages=find_packages({{#pythonSrcRoot}}"{{{.}}}"{{/pythonSrcRoot}}),{{#pythonSrcRoot}}
+    package_dir={"": "{{{.}}}"},{{/pythonSrcRoot}}
+    package_data={'': ['{{#pythonSrcRoot}}{{{.}}}/{{/pythonSrcRoot}}openapi/openapi.yaml']},
+    include_package_data=True,
+    entry_points={
+        'console_scripts': ['{{packageName}}={{packageName}}.__main__:main']},
+    long_description="""\
+    {{appDescription}}
+    """
+)
+{{/hasMore}}{{/apis}}{{/apiInfo}}

modules/openapi-generator/src/main/resources/python-aiohttp/tox.mustache

@@ -5,6 +5,7 @@ skipsdist=True
 [testenv]
 deps=-r{toxinidir}/requirements.txt
      -r{toxinidir}/test-requirements.txt
+	 {toxinidir}
 
 commands=
-   {{^useNose}}pytest --cov={{{packageName}}}{{/useNose}}{{#useNose}}nosetests{{/useNose}}
+   {{^useNose}}pytest --cov={{{pythonSrcRoot}}}{{{packageName}}}{{/useNose}}{{#useNose}}nosetests{{/useNose}}

modules/openapi-generator/src/main/resources/python-blueplanet/app/setup.mustache

@@ -23,7 +23,8 @@ setup(
     url="{{packageUrl}}",
     keywords=["Swagger", "{{appName}}"],
     install_requires=REQUIRES,
-    packages=find_packages(),
+    packages=find_packages({{#pythonSrcRoot}}"{{{.}}}"{{/pythonSrcRoot}}),{{#pythonSrcRoot}}
+    package_dir={"": "{{{.}}}"},{{/pythonSrcRoot}}
     package_data={'': ['swagger/swagger.yaml']},
     include_package_data=True,
     entry_points={

modules/openapi-generator/src/main/resources/python-blueplanet/app/tox.mustache

@@ -6,4 +6,4 @@ deps=-r{toxinidir}/requirements.txt
      -r{toxinidir}/test-requirements.txt
 
 commands=
-   {{^useNose}}pytest --cov={{{packageName}}}{{/useNose}}{{#useNose}}nosetests{{/useNose}}
+   {{^useNose}}pytest --cov={{{pythonSrcRoot}}}{{{packageName}}}{{/useNose}}{{#useNose}}nosetests{{/useNose}}

modules/openapi-generator/src/main/resources/python-flask/requirements.mustache

@@ -1,10 +1,20 @@
-connexion >= 2.6.0; python_version>="3.6"
-connexion >= 2.3.0; python_version=="3.5"
-connexion >= 2.3.0; python_version=="3.4"
-connexion == 2.4.0; python_version<="2.7"
+connexion[swagger-ui] >= 2.6.0; python_version>="3.6"
+# 2.3 is the last version that supports python 3.4-3.5
+connexion[swagger-ui] <= 2.3.0; python_version=="3.5" or python_version=="3.4"
+{{#supportPython2}}
+connexion[swagger-ui] == 2.4.0; python_version<="2.7"
+{{/supportPython2}}
+# connexion requires werkzeug but connexion < 2.4.0 does not install werkzeug
+# we must peg werkzeug versions below to fix connexion
+# https://github.com/zalando/connexion/pull/1044
+werkzeug == 0.16.1; python_version=="3.5" or python_version=="3.4"
 swagger-ui-bundle >= 0.0.2
 python_dateutil >= 2.6.0
 {{#supportPython2}}
 typing >= 3.5.2.2
+# For specs with timestamps, pyyaml 5.3 broke connexion's spec parsing in python 2.
+# Connexion uses copy.deepcopy() on the spec, thus hitting this bug:
+# https://github.com/yaml/pyyaml/issues/387
+pyyaml < 5.3; python_version<="2.7"
 {{/supportPython2}}
 setuptools >= 21.0.0

modules/openapi-generator/src/main/resources/python-flask/setup.mustache

@@ -28,8 +28,9 @@ setup(
     url="{{packageUrl}}",
     keywords=["OpenAPI", "{{appName}}"],
     install_requires=REQUIRES,
-    packages=find_packages(),
-    package_data={'': ['openapi/openapi.yaml']},
+    packages=find_packages({{#pythonSrcRoot}}"{{{.}}}"{{/pythonSrcRoot}}),{{#pythonSrcRoot}}
+    package_dir={"": "{{{.}}}"},{{/pythonSrcRoot}}
+    package_data={'': ['{{#pythonSrcRoot}}{{{.}}}/{{/pythonSrcRoot}}openapi/openapi.yaml']},
     include_package_data=True,
     entry_points={
         'console_scripts': ['{{packageName}}={{packageName}}.__main__:main']},

modules/openapi-generator/src/main/resources/python-flask/test-requirements.mustache

@@ -10,4 +10,4 @@ pytest~=4.6.7 # needed for python 2.7+3.4
 pytest-cov>=2.8.1
 pytest-randomly==1.2.3 # needed for python 2.7+3.4
 {{/useNose}}
-flask_testing==0.6.1
+Flask-Testing==0.8.0

modules/openapi-generator/src/main/resources/python-flask/tox.mustache

@@ -1,9 +1,11 @@
 [tox]
 envlist = {{#supportPython2}}py27, {{/supportPython2}}py3
+skipsdist=True
 
 [testenv]
 deps=-r{toxinidir}/requirements.txt
      -r{toxinidir}/test-requirements.txt
+	 {toxinidir}
 
 commands=
-   {{^useNose}}pytest --cov={{{packageName}}}{{/useNose}}{{#useNose}}nosetests{{/useNose}}
+   {{^useNose}}pytest --cov={{{pythonSrcRoot}}}{{{packageName}}}{{/useNose}}{{#useNose}}nosetests{{/useNose}}

pom.xml

@@ -1230,8 +1230,14 @@
                 </property>
             </activation>
             <modules>
-                <!-- comment out below due to https://github.com/OpenAPITools/openapi-generator/issues/5235
-                <module>samples/server/petstore/python-aiohttp</module> -->
+                <!-- servers -->
+                <module>samples/server/petstore/python-aiohttp</module>
+                <module>samples/server/petstore/python-aiohttp-srclayout</module>
+                <module>samples/server/petstore/python-flask</module>
+                <module>samples/server/petstore/python-flask-python2</module>
+                <module>samples/server/petstore/php-slim</module>
+                <module>samples/server/petstore/php-slim4</module>
+                <module>samples/server/petstore/rust-server</module>
                 <!-- clients -->
                 <module>samples/client/petstore/bash</module>
                 <module>samples/client/petstore/c</module>
@@ -1241,8 +1247,6 @@
                 <!--<module>samples/client/petstore/perl</module>-->
                 <module>samples/client/petstore/php/OpenAPIClient-php</module>
                 <module>samples/openapi3/client/petstore/php/OpenAPIClient-php</module>
-                <module>samples/server/petstore/php-slim</module>
-                <module>samples/server/petstore/php-slim4</module>
                 <module>samples/client/petstore/javascript</module>
                 <!--<module>samples/client/petstore/javascript-apollo</module>-->
                 <module>samples/client/petstore/javascript-es6</module>
@@ -1273,10 +1277,6 @@
                 <module>samples/client/petstore/typescript-angular-v4.3/npm</module>-->
                 <module>samples/client/petstore/typescript-angular-v6-provided-in-root</module>
                 <module>samples/client/petstore/typescript-angular-v7-provided-in-root</module>
-                <module>samples/server/petstore/rust-server</module>
-                <!-- comment out below due to https://github.com/OpenAPITools/openapi-generator/issues/5235
-                <module>samples/server/petstore/python-flask</module> -->
-                <!--<module>samples/server/petstore/python-flask-python2</module>-->
             </modules>
         </profile>
         <!-- test with JDK8 in CircleCI -->