Dependency on aiobotocore renders packaging s3fs no longer possible

[mgorny]

I don't suppose there's anything that can be done about this. Nevertheless, I'd like to point out that due to dependency on aiobotocore we are no longer able to package new versions s3fs on Gentoo. FWICS the few other distributions that used to package s3fs are having the same problem.

The problem is that aiobotocore is a huge hack on internals of botocore, and therefore it requires very specific version of botocore. With the daily release cycle of botocore and rare releases of aiobotocore, we can't package it unless we repeatedly force old versions of botocore on our users.

[martindurant]

I am surprised! Indeed, I was aware that aiobotocore has very narrow dependency requirements on botocore, but for users who need it, isn't that a necessary and acceptable situation? I wouldn't have said that aiobotocore releases were rare, exactly ( https://github.com/aio-libs/aiobotocore/releases ).

[mgorny]

Indeed, I was aware that aiobotocore has very narrow dependency requirements on botocore, but for users who need it, isn't that a necessary and acceptable situation?

Users who need aiobotocore are relatively small group compared to users who need botocore in general, and it would be unfair to block the latter to satisfy the former. Especially that it's not botocore's fault that someone chose to implement a huge hack against it.

I wouldn't have said that aiobotocore releases were rare, exactly ( https://github.com/aio-libs/aiobotocore/releases ).

Depends on your definition of 'rare'. Compared to daily releases of botocore, they might be rare. But not even that is the problem. The release from 4 days ago depends on botocore 1.17.44, i.e. the release from 18 days ago. That's 12 releases behind already, and it's going to grow.

[mrocklin]

Thank you for raising this issue @mgorny . This is, as @martindurant says, both surprising and unfortunate. I at least wasn't aware of the relationship between aiobotocore and botocore.

I'm curious though, isn't this the kind of thing that a dependency solver should be able to handle? I would expect aiobotocore to be pretty explicit about its requirements. If someone wanted to install s3fs wouldn't the solver find a recent version of aiobotocore, then find the matching version of botocore and install those? Or is the challenge that other packages also depend on very specific versions of botocore, and so it is hard to find a good global match?

[mgorny]

This depends on the system.

In case of Gentoo, we package every Nth version of botocore, and we just happened not to package the one aiobotocore needs. While I could package this one more version, these kind of dependencies are a maintenance nightmare. Since they're older than the newest one, they're given very little testing. I need to remember not to remove them when cleaning up old stuff because it'd break aiobotocore. Finally, I can't simultaneously have s3fs installed and test new versions of botocore.

PkgSrc, for example, does not support multiple versions at all.

[martindurant]

Obviously, I have no idea how to go about linux package management... but I thought I should point out that, according to https://pypistats.org/ , last month had 72M botocore downloads and 5M s3fs, so you may well find that aiobotocore will not be so niche any more ( see the graph here ).

Of course, users have access to pip and conda too - system packages have different requirements.

[dwink]

This affects more than just OS packages -- the transitive dependency chain for basically anything that depends on boto3 is broken unless you can fully convert all packages to aiobotocore, or if the various pinned versions end up being compatible all the time.

For instance, I have a project that had s3fs 0.4.2 and boto3, and it wasn't pinned to 0.4.2 of s3fs; on Friday a minor code change went through CI and the entire project broke, because pip won't downgrade a package in order to satisfy a transitive dependency. So keeping up with s3fs means tracking aiobotocore releases and pinning/managing pinning of all boto3-based software that might be included in a project. Yikes.

[martindurant]

Sure - as I said, I don't know much about the constraints that you must be working under.
I would be surprised, though if this sort of situation were unusual. In practice, a large number of our users have conda and/or venv environments, so should be insluated.

[dwink]

I have a venv environment. This change broke it. That's what I'm trying to communicate here.

Here's a minimal example. I use the moto library for mocking boto calls. If I pip install s3fs, boto3, and mock, I get a broken setup:

pip install s3fs boto3 moto
...
ERROR: aiobotocore 1.1.1 has requirement botocore<1.17.45,>=1.17.44, but you'll have botocore 1.17.56 which is incompatible.
Installing collected packages: fsspec, typing-extensions, aioitertools, chardet, multidict, idna, yarl, async-timeout, attrs, ai
ohttp, wrapt, six, python-dateutil, jmespath, docutils, urllib3, botocore, aiobotocore, s3fs, s3transfer, boto3, MarkupSafe, Jin
ja2, certifi, requests, responses, more-itertools, pytz, werkzeug, boto, zipp, mock, xmltodict, moto
    Running setup.py install for wrapt ... done
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 aiobotocore-1.1.1 aiohttp-3.6.2 aioitertools-0.7.0 async-timeout-3.0.1 att
rs-20.2.0 boto-2.49.0 boto3-1.14.56 botocore-1.17.56 certifi-2020.6.20 chardet-3.0.4 docutils-0.15.2 fsspec-0.8.0 idna-2.10 jmes
path-0.10.0 mock-4.0.2 more-itertools-8.5.0 moto-1.3.15 multidict-4.7.6 python-dateutil-2.8.1 pytz-2020.1 requests-2.24.0 respon
ses-0.12.0 s3fs-0.5.0 s3transfer-0.3.3 six-1.15.0 typing-extensions-3.7.4.3 urllib3-1.25.10 werkzeug-1.0.1 wrapt-1.12.1 xmltodic
t-0.12.0 yarl-1.5.1 zipp-3.1.0
...

[martindurant]

So it's pip's fault... I'm not sure what to do about that. Presumably if you installed from scratch, you would get aiobotocore and botocore which match, and further installs would not update them. Conda's solver will downgrade packages to get a consistent environment.

[dwink]

So it's pip's fault... I'm not sure what to do about that. Presumably if you installed from scratch, you would get aiobotocore and botocore which match, and further installs would not update them. Conda's solver will downgrade packages to get a consistent environment.

The example above was installing from scratch. I created a new venv, installed just s3fs and boto3 and moto, and ended up with a broken setup.

I'll see what I can do to work around this, but I suspect this won't be nearly as isolated an issue as you seem to think, and at minimum I'd recommend explicitly documenting that in order to use s3fs going forward, users will need to curate all their dependencies on aiobotocore's release cadence.

[TomAugspurger]

Pip's new dependency resolver is for case like this. ```console $ pip install s3fs boto3 mock --use-feature=2020-resolver ... Successfully installed aiobotocore-1.1.1 aiohttp-3.6.2 aioitertools-0.7.0 async-timeout-3.0.1 attrs-20.2.0 boto3-1.14.44 botocore-1.17.44 chardet-3.0.4 docutils-0.15.2 fsspec-0.8.0 idna-2.10 jmespath-0.10.0 mock-4.0.2 multidict-4.7.6 python-dateutil-2.8.1 s3fs-0.5.0 s3transfer-0.3.3 six-1.15.0 typing-extensions-3.7.4.3 urllib3-1.25.10 wrapt-1.12.1 yarl-1.5.1 ```

…

On Tue, Sep 8, 2020 at 10:17 AM David Harks ***@***.***> wrote: So it's pip's fault... I'm not sure what to do about that. Presumably if you installed from scratch, you would get aiobotocore and botocore which match, and further installs would not update them. Conda's solver *will* downgrade packages to get a consistent environment. The example above *was* installing from scratch. I created a new venv, installed just s3fs and boto3 and moto, and ended up with a broken setup. I'll see what I can do to work around this, but I suspect this won't be nearly as isolated an issue as you seem to think, and at minimum I'd recommend explicitly documenting that in order to use s3fs going forward, users will need to curate all their dependencies on aiobotocore's release cadence. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#357 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOITEUXTME5UX4AQJAE3SEZDKFANCNFSM4QZGVGEQ> .

[graingert]

there's an extra; aiobotocore[boto3] which is supposed to fix this

[dwink]

there's an extra; aiobotocore[boto3] which is supposed to fix this

Which is great -- except I don't depend on aiobotocore except transitively through THIS package. So I have to know about this dependency and manage it (and hopefully this package doesn't depend on anything that isn't in the botocore version aiobotocore[boto3] provides). I know this is a weakness of the Python ecosystem but it's a really frustrating one.

[graingert]

I could try adding an s3fs[boto3] extra that selects aiobotocore[boto3]

[dwink]

I could try adding an s3fs[boto3] extra that selects aiobotocore[boto3]

That sounds like it might be perfect.

[martindurant]

(note that s3fs doesn't actually depend on boto3, but these all seem to be intermingled)

[graingert]

Yeah the same with aiobotocore, it doesn't depend on boto3, it's just enough to let pip 20.2 old mode do the right thing

[mrocklin]

@jakirkham you may also find this issue interesting with your packaging hat on

…

On Wed, Sep 9, 2020 at 8:59 AM Martin Durant ***@***.***> wrote: (note that s3fs doesn't actually depend on boto3, but these all seem to be intermingled) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#357 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTG2C7EUEL4Z3TTCNLLSE6Q57ANCNFSM4QZGVGEQ> .

[jakirkham]

I don't spend that much time with pip. So doubt I'm much help here.

[graingert]

I think you have to specify the boto3 and awscli directly as they are in aiobotocore and bump them each time there's a release

[caseyhartnett]

Blocking our build when transitioning from 3.7 python to 3.8 pythons. Include s3fs and it fails otherwise build completes.

[graingert]

@caseyhartnett pinning both boto3 and botocore seems the best option here. Otherwise use the --use-feature=2020-resolver flag

In general I can't think of any other fix than vendoring aiobotocore or using --use-feature=2020-resolver

[shughes-uk]

I have a long back and forth with one of the aiobotocore devs over this aio-libs/aiobotocore#829 (comment) , might be helpful to have more people weigh in. I think the only solution they have on their end is to vendorize botocore, which I understand them being reluctant to do as it's a pain in the butt.

[CloudySnake]

Sorry if this has been asked already, but why does s3fs have this dependency on aiobotocore? I have the same issue with versioning (and also Moto now doesn't play nicely with s3fs due to the async additions) - as a workaround I have pinned s3fs to v0.4.* but that's not a permanent solution.

[shlomiken]

@graingert - thanks for the reply , unfortunately i see many of these in my pip install command.

WARNING: aiobotocore 0.3.0 does not provide the extra 'awscli'

[CalebEverett]

What is the recommended way to install these packages together?

s3fs = "*"
moto = "*"
boto3 = "*"

edit:
s3fs<0.5.0 works

[graingert]

pip install s3fs[boto3] moto

[CalebEverett]

I still get incompatible versions using trying that with pipenv.

  botocore<1.20.50,>=1.20.49 (from aiobotocore==1.3.0->s3fs[boto3]==2021.4.0->-r /tmp/pipenvnk225cejrequirements/pipenv-upj0zg8x-constraints.txt (line 3))
  botocore<1.21.0,>=1.20.57 (from boto3==1.17.57->moto==2.0.5->-r /tmp/pipenvnk225cejrequirements/pipenv-upj0zg8x-constraints.txt (line 6))
  botocore>=1.12.201 (from moto==2.0.5->-r /tmp/pipenvnk225cejrequirements/pipenv-upj0zg8x-constraints.txt (line 6))

[damonmaria]

@CalebEverett Have you tried using Pipenv with SYSTEM_VERSION_COMPAT=1. That will use the newer Pip resolver. I haven't tried it specifically with s3fs.

[CalebEverett]

@damonmaria thanks, but no such luck - it still complains about incompatible versions without specifying s3fs<0.5.0

[theholy7]

@CalebEverett
what we did was:

aiobotocore[boto3]
s3fs~=0.4.0

Does that work for you?

[CalebEverett]

@theholy7 thanks - yes, not sure if I ended up with 0.4.0 exactly, but found that <0.5.0 works.

[tinkerware]

For anyone who ended up in this issue while trying to get moto, s3fs and aiobotocore together, this is what I currently use for the most recent releases of these dependencies in Pipenv:

[dev-packages]
moto = "~= 2.0"

[packages]
# The two dependencies below should be kept together, since they solve a thorny
# transitive dependency problem documented at https://github.com/dask/s3fs/issues/357.
s3fs = {version = "~=2021.4", extras = ["boto3"]}
aiobotocore = {version = "*", extras = ["boto3"]}

[dazza-codes]

This kind of issue was anticipated in aio-libs/aiobotocore#862

s3fs < 0.5 does not depend on aiobotocore at all, so there are no conflicts in those versions.

It would be ideal if this were resolved by relaxing boto* version dependencies in aiobotocore, but a PR to do so was not merged. I was compelled to publish a fork in

• https://pypi.org/project/aio-botocore/ (a thin fork on aiobotocore 1.3.x)
• this is not an ideal approach, for several reasons that I wont go into in detail here

But, that doesn't solve the s3fs dependency problem for anyone using s3fs > 0.4.x, unless s3fs were to switch to the forked version of aio-botocore or the upstream aiobotocore were to merge aio-libs/aiobotocore#834 (or an update to that effect)

I don't see how using any extras solves this problem because the boto3 extras are also very tightly pinned in aiobotocore:

• https://github.com/aio-libs/aiobotocore/blob/master/setup.py#L21-L24

This often results in pip (and poetry) trying to resolve dep-trees with 100s or 1000s of variants of botocore and boto3 libs, leading to very long install times to explore all the version permutations and often results in incompatible failures.

[tuky]

this pipfile works for us now together with SYSTEM_VERSION_COMPAT=1:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
aiobotocore = {version = "*", extras = ["boto3"]}
s3fs = "*"
django-storages = "*"

we also had to make sure, that we have neither boto3 nor botocore listed explicitly in our pipfile, although we are using it directly ourselves.