[Bug] dbt-core is not compatible with latest protobuf version

[JasperHG90]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Protobuf 5.26.0 was released today.

When installing dbt and running dbt deps, I see:

dbt deps
18:45:37  Running with dbt=1.7.9
18:45:38  Installing dbt-labs/dbt_utils
18:45:38  Installed from version 1.1.1
18:45:38  Up to date!
18:45:38  Installing calogica/dbt_expectations
18:45:39  Installed from version 0.10.0
18:45:39  Updated version available: 0.10.3
18:45:39  Installing calogica/dbt_date
18:45:39  Installed from version 0.10.0
18:45:39  Up to date!
18:45:39  
18:45:39  Updates available for packages: ['calogica/dbt_expectations']                 
Update your versions in packages.yml, then run dbt deps
Traceback (most recent call last):
  File "/home/vscode/.pyenv/versions/3.9.18/bin/dbt", line 8, in <module>
    sys.exit(cli())
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/dbt/cli/main.py", line 155, in wrapper
    return func(*args, **kwargs)
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/dbt/cli/requires.py", line 109, in wrapper
    fire_event(
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/dbt/events/functions.py", line 270, in fire_event
    EVENT_MANAGER.fire_event(e, level=level)
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/dbt/events/eventmgr.py", line 17, in fire_event
    msg = msg_from_base_event(e, level=level)
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/dbt/events/base_types.py", line 143, in msg_from_base_event
    "msg": event.message(),
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/dbt/events/types.py", line 2414, in message
    return f"Resource report: {self.to_json()}"
  File "/home/vscode/.pyenv/versions/3.9.18/lib/python3.9/site-packages/dbt/events/base_types.py", line 104, in to_json
    return MessageToJson(
TypeError: MessageToJson() got an unexpected keyword argument 'including_default_value_fields'

Fixing protobuf dependency on protobuf 4.25.3 fixes the error.

Expected Behavior

With protobuf < 5.26.0, I don't see an error.

Steps To Reproduce

1. Install dbt
2. Run dbt deps

Relevant log output

See above

Environment

- OS: ubuntu 22.04 (devcontainer)
- Python: 3.9.18
- dbt: 1.7.9

Which database adapter are you using with dbt?

spark

Additional Context

No response

[JasperHG90]

Should be fixed by #9724

[akan72]

@JasperHG90 Thanks for opening up this issue! Pinning to protobuf==4.25.3 solved this for us. Saved me from a lot of headache!

[katepaydhealth]

@akan72, how could you pin the protobuf to solve the issue?

[dbeatty10]

@katepaydhealth I didn't confirm myself, but probably by executing something like this in your dbt environment:

python -m pip install protobuf==4.25.3

[akan72]

@katepaydhealth I didn't confirm myself, but probably by executing something like this in your dbt environment:

python -m pip install protobuf==4.25.3

Yup this is exactly what I did!

[gshank]

A new release 1.7.10 is being built now with protobuf pinned.

[gshank]

dbt-core 1.7.10 has been released.

[dbeatty10]

Thank you @gshank ❤️

I can confirm that the the following raised an error before, but is not raising an error now that dbt-core 1.7.10 has been released:

deactivate
python -m venv env_1.7
source env_1.7/bin/activate
python -m pip install --upgrade pip
python -m pip install pipdeptree
python -m pip install dbt-core~=1.7.0 dbt-postgres~=1.7.0
source env_1.7/bin/activate
dbt --version
dbt debug

I'm going to close this as resolved, but anyone runs into issues, please reach out and we'll revisit this.

[VShkaberda]

@gshank Maybe an upper version limit for other modules might be set too?

[peter-at-work]

Maybe an upper version limit for other modules might be set too?

Tedious, but true. Many module developers use semantic versioning to convey if there are breaking changes in a release.

Unfortunately, pip doesn't have the equivalent of the caret ^ semantics as used in the NPM package ecosystem, as it would have been easy to write the dependency as protobuf ^4.25.3 and it would not cross over to installing the 5.x release. With pip, we would have to write this as protobuf >=4.25.3, <5.0.0 (as what has been done for this issue) or maybe more intuitively protobuf >=4.25.3, ==4.*

[VShkaberda]

pip install protobuf~=4.25 is an equivalent of protobuf==4.* and protobuf>=4.25. It would not cross over to installing the 5.x in this case. If requirement includes a patch, then protobuf >=4.25.3, ==4.* seems to be good enough.

[QMalcolm]

Maybe an upper version limit for other modules might be set too?

Hi @VShkaberda that is actually exactly what we're doing now (with only a few exceptions). We switched not too long ago to specifying upper bounds for all most all of our dependencies (see setup.py) 🙂

[peter-at-work]

Hi @VShkaberda that is actually exactly what we're doing (with only a few exceptions). We switched not too long ago to specifying upper bounds for all most all of our dependencies (see setup.py) 🙂

        ...
        # Expect compatibility with all new versions of these packages, so lower bounds only.
        "packaging>20.9",
        "protobuf>=4.0.0,<5",
        "pytz>=2015.7",
        ...

Might help future readers of the code to move the protobuf entry out of the section mentioned by the inline comment 🙂

[QMalcolm]

pip install protobuf~=4.25 is an equivalent of protobuf==4.* and protobuf>=4.25. It would not cross over to installing the 5.x in this case. If requirement includes a patch, then protobuf >=4.25.3, ==4.* seems to be good enough.

@VShkaberda We'd love to move to using ~= notation, and originally we were in dbt-semantic-interfaces. However, ~= has some interesting bugs with compatible version specification resolution. Some of these have been resolved (pypa/pip#12229, pypa/packaging#421) and some have not (pypa/pip#9613). Once they're all taken care of and released, we hope to move to using ~= again. However, even once those fixes are released we'll have to wait a good bit of time because people will first need to upgrade their pip version... which people do infrequently at best.

[QMalcolm]

@peter-at-work, you are absolutely correct 😅 That's actually directly on me, I did the capping... Correction in #9786

[mcDandy]

Thx for solution. I would be however happier if I did not have to downgrade protobuf.

[dbeatty10]

@mcDandy if you are interested in support for protobuf 5, would you mind opening a feature request?

#9566 has a little bit of additional context of why we put an upper bound on the protobuf dependency. It sounds like we could theoretically support protobuf 5, but it would take additional work for us to assess, and the implementation would likely make our code base more complex as well.

[dbeatty10]

Feature request to support protobuf 5 for dbt-core is here: #9724

[acauquil]

Thxs for the mitigation but I can't downgrade protobuf since my projects already requires >5
Would LOVE a fix on that 🙂