remove tensorflow dependency #221

fcakyon · 2021-03-22T21:56:06Z

Due to the tensorflow dependency, this package is not usable with tensorflow-cpu package.

Can you please make tensorflow dependency optional so that I can use it with tensorflow-cpu?

adriangb · 2021-03-22T22:04:18Z

Hmm, there's basically no description on the PyPi package (other than some generic what is TensorFlow stuff). What is the tensorflow-cpu package, and how does it differ from the plain tensorflow package? Or can you link me to any relevant docs? Thanks!

fcakyon · 2021-03-22T22:08:22Z

@adriangb It is also maintained by Google and has the same api with tensorflow package. It doesnt include the compiled cuda binaries so that it takes less space and easier to deploy on machines without gpu.

fcakyon · 2021-03-22T22:09:17Z

It is especially useful when deploying a ML app to Heroku: https://stackoverflow.com/a/62356779

adriangb · 2021-03-22T22:17:21Z

Interesting, I sure wish the TensorFlow team had better descriptions. Also it would have been nice if they packaged it as an extra like pip install tensorflow[cpu-only], instead of making it a completely separate package 😞

The main issue with just removing the dependency is that we actually do have minimum TensorFlow versions, and will probably bump them soon. Of course we can replace the requirement with a runtime check, but that has its own disadvantages.

For the time being, it should be possible to install scikeras with pip install --no-deps scikeras

stsievert · 2021-03-22T23:16:23Z

For the time being, it should be possible to install scikeras with pip install --no-deps scikeras

That might be worth adding to the docs.

adriangb · 2021-03-22T23:19:36Z

Agreed, I'll add it in the install section.

adriangb · 2021-03-22T23:33:59Z

@fcakyon I opened #222 to add some guidance in the docs. Do you feel that this would be an acceptable solution?

fcakyon · 2021-03-23T06:26:34Z

@adriangb Unfortunately its not an acceptable solution in my case.

Im using this package in this web app project. Heroku creates the environment using this requirements.txt. And during Heroku deployment I cant specify to install scikeras with --no-deps argument, it directly installs the dependencies from requirements.txt file.

However since scikeras has tensorflow dependency, it downloads both tensorflow-cpu and tensorflow into the environment and fails the deployment due to limited storage.

Better solution would be to make tensorflow dependency optional: if tensorflow-cpu>2.4.0 is present then dont install tensorflow, else require tensorflow>2.4.0

adriangb · 2021-03-23T16:14:19Z

I see.

It seems that then we are left with two options:

Do what you suggested in remove tensorflow dependency #221 (comment) and make tensorflow optional so that it would need to be installed via pip install scikeras[tensorflow] or pip install scikeras[tensorflow-cpu]. However, I think this would be a breaking change, so it would have to wait for a major version.
Make a scikeras-core package that has no dependencies (only runtime checks). You could then use this scikeras-core package and other users could continue to use scikeras.

fcakyon · 2021-03-23T16:23:04Z

Would the second approach bring lots of maintenance cost for you? If not, it would be very useful for the ones that are in same situation with me.

adriangb · 2021-03-23T16:25:10Z

It would carry some maintenance burden, I'm not sure how much, but I'll look into it and at least scope it out.

stsievert · 2021-03-23T18:49:37Z

Would per-requirement overrides work?

fcakyon · 2021-03-23T20:23:45Z

@stsievert I tried adding --install-option="--no-deps" into requirements.txt but pip installation fails since pre-requirement commands disable wheel installation. Here is the full heroku deploy log:

-----> Building on the Heroku-20 stack
-----> Using buildpacks:
       1. heroku/python
       2. https://github.com/heroku/heroku-buildpack-apt
-----> Python app detected
-----> Stack has changed from heroku-18 to heroku-20, clearing cache
-----> Installing python-3.8.8
-----> Installing pip 20.1.1, setuptools 47.1.1 and wheel 0.34.2
-----> Installing SQLite3
-----> Installing requirements with pip
       /app/.heroku/python/lib/python3.8/site-packages/pip/_internal/req/req_file.py:200: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
         cmdoptions.check_install_build_global(options, line.opts)
       Collecting flask==1.1.1
         Downloading Flask-1.1.1.tar.gz (625 kB)
       Collecting gunicorn==19.9.0
         Downloading gunicorn-19.9.0.tar.gz (415 kB)
       Collecting opencv-contrib-python>=4.0
         Downloading opencv-contrib-python-4.5.1.48.tar.gz (148.8 MB)
         Installing build dependencies: started
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: finished with status 'done'
         Getting requirements to build wheel: started
         Getting requirements to build wheel: finished with status 'done'
           Preparing wheel metadata: started
           Preparing wheel metadata: finished with status 'done'
       Collecting matplotlib==3.1.3
         Downloading matplotlib-3.1.3.tar.gz (40.9 MB)
       Collecting imutils==0.5.3
         Downloading imutils-0.5.3.tar.gz (17 kB)
       Collecting requests==2.22.0
         Downloading requests-2.22.0.tar.gz (113 kB)
       Collecting scipy==1.5.4
         Downloading scipy-1.5.4.tar.gz (25.2 MB)
         Installing build dependencies: started
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: still running...
         Installing build dependencies: finished with status 'done'
         Getting requirements to build wheel: started
         Getting requirements to build wheel: finished with status 'done'
           Preparing wheel metadata: started
           Preparing wheel metadata: finished with status 'error'
           ERROR: Command errored out with exit status 1:
            command: /app/.heroku/python/bin/python /app/.heroku/python/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpp9jql2k3
                cwd: /tmp/pip-install-iqk9zng4/scipy
           Complete output (171 lines):
           setup.py:460: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /tmp/pip-modern-metadata-8vvfmt5g'), proceeding with generating Cython sources and expanding templates
             warnings.warn("Unrecognized setuptools command ('{}'), proceeding with "
           Running from SciPy source directory.
           lapack_opt_info:
           lapack_mkl_info:
           customize UnixCCompiler
             libraries mkl_rt not found in ['/app/.heroku/python/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
             NOT AVAILABLE
           
           openblas_lapack_info:
           customize UnixCCompiler
           customize UnixCCompiler
             libraries openblas not found in ['/app/.heroku/python/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
             NOT AVAILABLE
           
           openblas_clapack_info:
           customize UnixCCompiler
           customize UnixCCompiler
             libraries openblas,lapack not found in ['/app/.heroku/python/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
             NOT AVAILABLE
           
           flame_info:
           customize UnixCCompiler
             libraries flame not found in ['/app/.heroku/python/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
             NOT AVAILABLE
           
           atlas_3_10_threads_info:
           Setting PTATLAS=ATLAS
           customize UnixCCompiler
             libraries lapack_atlas not found in /app/.heroku/python/lib
           customize UnixCCompiler
             libraries tatlas,tatlas not found in /app/.heroku/python/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/local/lib
           customize UnixCCompiler
             libraries tatlas,tatlas not found in /usr/local/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib64
           customize UnixCCompiler
             libraries tatlas,tatlas not found in /usr/lib64
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib
           customize UnixCCompiler
             libraries tatlas,tatlas not found in /usr/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
           customize UnixCCompiler
             libraries tatlas,tatlas not found in /usr/lib/x86_64-linux-gnu
           <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
             NOT AVAILABLE
           
           atlas_3_10_info:
           customize UnixCCompiler
             libraries lapack_atlas not found in /app/.heroku/python/lib
           customize UnixCCompiler
             libraries satlas,satlas not found in /app/.heroku/python/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/local/lib
           customize UnixCCompiler
             libraries satlas,satlas not found in /usr/local/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib64
           customize UnixCCompiler
             libraries satlas,satlas not found in /usr/lib64
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib
           customize UnixCCompiler
             libraries satlas,satlas not found in /usr/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
           customize UnixCCompiler
             libraries satlas,satlas not found in /usr/lib/x86_64-linux-gnu
           <class 'numpy.distutils.system_info.atlas_3_10_info'>
             NOT AVAILABLE
           
           atlas_threads_info:
           Setting PTATLAS=ATLAS
           customize UnixCCompiler
             libraries lapack_atlas not found in /app/.heroku/python/lib
           customize UnixCCompiler
             libraries ptf77blas,ptcblas,atlas not found in /app/.heroku/python/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/local/lib
           customize UnixCCompiler
             libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib64
           customize UnixCCompiler
             libraries ptf77blas,ptcblas,atlas not found in /usr/lib64
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib
           customize UnixCCompiler
             libraries ptf77blas,ptcblas,atlas not found in /usr/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
           customize UnixCCompiler
             libraries ptf77blas,ptcblas,atlas not found in /usr/lib/x86_64-linux-gnu
           <class 'numpy.distutils.system_info.atlas_threads_info'>
             NOT AVAILABLE
           
           atlas_info:
           customize UnixCCompiler
             libraries lapack_atlas not found in /app/.heroku/python/lib
           customize UnixCCompiler
             libraries f77blas,cblas,atlas not found in /app/.heroku/python/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/local/lib
           customize UnixCCompiler
             libraries f77blas,cblas,atlas not found in /usr/local/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib64
           customize UnixCCompiler
             libraries f77blas,cblas,atlas not found in /usr/lib64
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib
           customize UnixCCompiler
             libraries f77blas,cblas,atlas not found in /usr/lib
           customize UnixCCompiler
             libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
           customize UnixCCompiler
             libraries f77blas,cblas,atlas not found in /usr/lib/x86_64-linux-gnu
           <class 'numpy.distutils.system_info.atlas_info'>
             NOT AVAILABLE
           
           accelerate_info:
             NOT AVAILABLE
           
           lapack_info:
           customize UnixCCompiler
             libraries lapack not found in ['/app/.heroku/python/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
             NOT AVAILABLE
           
           /tmp/pip-build-env-jn9d_eos/overlay/lib/python3.8/site-packages/numpy/distutils/system_info.py:1712: UserWarning:
               Lapack (http://www.netlib.org/lapack/) libraries not found.
               Directories to search for the libraries can be specified in the
               numpy/distutils/site.cfg file (section [lapack]) or by setting
               the LAPACK environment variable.
             if getattr(self, '_calc_info_{}'.format(lapack))():
           lapack_src_info:
             NOT AVAILABLE
           
           /tmp/pip-build-env-jn9d_eos/overlay/lib/python3.8/site-packages/numpy/distutils/system_info.py:1712: UserWarning:
               Lapack (http://www.netlib.org/lapack/) sources not found.
               Directories to search for the sources can be specified in the
               numpy/distutils/site.cfg file (section [lapack_src]) or by setting
               the LAPACK_SRC environment variable.
             if getattr(self, '_calc_info_{}'.format(lapack))():
             NOT AVAILABLE
           
           Traceback (most recent call last):
             File "/app/.heroku/python/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
               main()
             File "/app/.heroku/python/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
               json_out['return_val'] = hook(**hook_input['kwargs'])
             File "/app/.heroku/python/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
               return hook(metadata_directory, config_settings)
             File "/tmp/pip-build-env-jn9d_eos/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 166, in prepare_metadata_for_build_wheel
               self.run_setup()
             File "/tmp/pip-build-env-jn9d_eos/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 258, in run_setup
               super(_BuildMetaLegacyBackend,
             File "/tmp/pip-build-env-jn9d_eos/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
               exec(compile(code, __file__, 'exec'), locals())
             File "setup.py", line 583, in <module>
               setup_package()
             File "setup.py", line 579, in setup_package
               setup(**metadata)
             File "/tmp/pip-build-env-jn9d_eos/overlay/lib/python3.8/site-packages/numpy/distutils/core.py", line 137, in setup
               config = configuration()
             File "setup.py", line 477, in configuration
               raise NotFoundError(msg)
           numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.
           ----------------------------------------
       ERROR: Command errored out with exit status 1: /app/.heroku/python/bin/python /app/.heroku/python/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpp9jql2k3 Check the logs for full command output.
 !     Push rejected, failed to compile Python app.
 !     Push failed

adriangb · 2021-03-24T16:04:35Z

Just as a sanity check, I tried installing tensorflow followed by tensorflow-cpu to see if having tensorflow installed would satisfy the tensorflow-cpu dependency or if they would both get installed. They both get installed, which is unfortunate because if tensorflow was a superset of tensorflow-cpu we could just list tensorflow-cpu since SciKeras doesn't do anything specific related to GPU.

fcakyon · 2021-03-24T16:12:40Z

They both get installed, which is unfortunate

Yes, indeed. That results in oversized heroku deployment and failure. Trying to find a way with per-requirement overrides but not successful at the moment.

adriangb · 2021-03-24T16:20:30Z

I think maybe a good next step is to open an issue in tensorflow to look for some guidance. Would you like to do that @fcakyon or would you prefer I do it?

fcakyon · 2021-03-24T17:29:48Z

I think you could express the situation better then me as a package maintainer. Would you mind opening the issue?

adriangb · 2021-03-24T17:33:05Z

Sure, no problem, I'll tag you.

adriangb · 2021-03-29T19:20:57Z

@fcakyon I openend tensorflow/tensorflow#48165

adriangb · 2021-03-29T23:29:15Z

It looks like there was already an issue in tensorflow for the same thing: tensorflow/tensorflow#7166

It seems like indeed the move is to make tensorflow an optional dependency. I think we're just going to have to bite the bullet on this one next major version bump @stsievert

stsievert · 2021-03-30T00:04:15Z

I think we're just going to have to bite the bullet on this one next major version bump

What exactly do you mean by that?

What exactly is required for installation now?

adriangb · 2021-03-30T00:05:19Z

Sorry, I should have been clearer.

Accept that we will need to make a backwards incompatible change that has its own set of downsides.

stsievert · 2021-03-30T01:37:07Z

Semantic versioning lays out the following:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes,

MINOR version when you add functionality in a backwards compatible manner, and

PATCH version when you make backwards compatible bug fixes.

So sounds like the minor version should be bumped from 0.2.1 to 0.3.0.

adriangb · 2021-03-30T01:40:30Z

Ugh sorry yes, you are right, spaghetti in my brain. I meant minor. So what you are saying is spot on.

fcakyon · 2021-08-02T17:02:45Z

@adriangb thats great news!

adriangb · 2021-08-02T18:21:43Z

Yep! Just decided to bite the bullet on it... If you can test off of master and confirm this fixes things for you, that'd be much appreciated!

adriangb mentioned this issue Mar 22, 2021

DOC: Add note on installing without dependencies #222

Merged

adriangb linked a pull request Mar 22, 2021 that will close this issue

DOC: Add note on installing without dependencies #222

Merged

adriangb closed this as completed in #222 Jun 7, 2021

adriangb reopened this Jun 7, 2021

fcakyon mentioned this issue Jul 6, 2021

scikeras and tensorflow compatibility Problem fcakyon/face-recognition-app-tutorial#11

Closed

adriangb mentioned this issue Aug 1, 2021

Remove hard dependency on TensorFlow #237

Merged

adriangb closed this as completed in #237 Aug 2, 2021

adriangb mentioned this issue Jan 5, 2022

tensorflow should be a main dependency (instead of a dev dependency) #261

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remove tensorflow dependency #221

remove tensorflow dependency #221

fcakyon commented Mar 22, 2021

adriangb commented Mar 22, 2021

fcakyon commented Mar 22, 2021

fcakyon commented Mar 22, 2021 •

edited

Loading

adriangb commented Mar 22, 2021 •

edited

Loading

stsievert commented Mar 22, 2021

adriangb commented Mar 22, 2021

adriangb commented Mar 22, 2021

fcakyon commented Mar 23, 2021 •

edited

Loading

adriangb commented Mar 23, 2021

fcakyon commented Mar 23, 2021

adriangb commented Mar 23, 2021

stsievert commented Mar 23, 2021

fcakyon commented Mar 23, 2021

adriangb commented Mar 24, 2021

fcakyon commented Mar 24, 2021

adriangb commented Mar 24, 2021

fcakyon commented Mar 24, 2021

adriangb commented Mar 24, 2021

adriangb commented Mar 29, 2021

adriangb commented Mar 29, 2021

stsievert commented Mar 30, 2021 •

edited

Loading

adriangb commented Mar 30, 2021

stsievert commented Mar 30, 2021 •

edited

Loading

adriangb commented Mar 30, 2021

fcakyon commented Aug 2, 2021

adriangb commented Aug 2, 2021

remove tensorflow dependency #221

remove tensorflow dependency #221

Comments

fcakyon commented Mar 22, 2021

adriangb commented Mar 22, 2021

fcakyon commented Mar 22, 2021

fcakyon commented Mar 22, 2021 • edited Loading

adriangb commented Mar 22, 2021 • edited Loading

stsievert commented Mar 22, 2021

adriangb commented Mar 22, 2021

adriangb commented Mar 22, 2021

fcakyon commented Mar 23, 2021 • edited Loading

adriangb commented Mar 23, 2021

fcakyon commented Mar 23, 2021

adriangb commented Mar 23, 2021

stsievert commented Mar 23, 2021

fcakyon commented Mar 23, 2021

adriangb commented Mar 24, 2021

fcakyon commented Mar 24, 2021

adriangb commented Mar 24, 2021

fcakyon commented Mar 24, 2021

adriangb commented Mar 24, 2021

adriangb commented Mar 29, 2021

adriangb commented Mar 29, 2021

stsievert commented Mar 30, 2021 • edited Loading

adriangb commented Mar 30, 2021

stsievert commented Mar 30, 2021 • edited Loading

adriangb commented Mar 30, 2021

fcakyon commented Aug 2, 2021

adriangb commented Aug 2, 2021

fcakyon commented Mar 22, 2021 •

edited

Loading

adriangb commented Mar 22, 2021 •

edited

Loading

fcakyon commented Mar 23, 2021 •

edited

Loading

stsievert commented Mar 30, 2021 •

edited

Loading

stsievert commented Mar 30, 2021 •

edited

Loading