Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add get_bq_config_path() to _cloud_sdk.py #1358

Merged
merged 6 commits into from
Aug 8, 2023

Conversation

wj-chen
Copy link
Contributor

@wj-chen wj-chen commented Jul 24, 2023

Will be used by the Cloud CLI's config to write a credential file for the bq tool to use. Part of the work to support BQ CLI's auth library upgrade from oauth2client to google-auth.

@wj-chen wj-chen requested review from a team as code owners July 24, 2023 21:19
@wj-chen
Copy link
Contributor Author

wj-chen commented Jul 24, 2023

I believe the kokoro failures are unrelated to my change, but this is my first time making a change to this repo so please advise.

@@ -42,8 +44,11 @@
)


def get_config_path():
"""Returns the absolute path the the Cloud SDK's configuration directory.
def get_config_path(config_directory=_CONFIG_DIRECTORY):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should wrap this function with a gcloud_get_config_path so the intent is clearly documented.

Would it be a large change to modify existing uses of get_config_path?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, not too much overhead to modify existing uses of get_config_path

@clundin25
Copy link
Contributor

@wj-chen I refreshed the test credentials. The next test run should have all the noise removed.

@clundin25 clundin25 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 25, 2023
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 25, 2023
@wj-chen
Copy link
Contributor Author

wj-chen commented Jul 25, 2023

@wj-chen I refreshed the test credentials. The next test run should have all the noise removed.

Thank you! There appears to be lint errors. Running nox -s lint has always given me the following output, do you have suggestions?

nox > Running session lint
nox > Missing interpreters will error by default on CI systems.
nox > Session lint skipped: Python interpreter 3.8 not found.

@clundin25
Copy link
Contributor

The interpreter for the lint step uses Python 3.8.

I suggest installing it via pyenv.

Here is a sample that will give you a shell that has 3.8.

$ pyenv install 3.8.15
$ pyenv shell 3.8.15
$ python --version # Confirm the interpreter is 3.8.

Now the active shell should have access to 3.8.

@wj-chen
Copy link
Contributor Author

wj-chen commented Jul 26, 2023

The interpreter for the lint step uses Python 3.8.

I suggest installing it via pyenv.

Here is a sample that will give you a shell that has 3.8.

$ pyenv install 3.8.15
$ pyenv shell 3.8.15
$ python --version # Confirm the interpreter is 3.8.

Now the active shell should have access to 3.8.

Thanks! Linter would run now but would result in an AttributionError:

      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 17, in finalize_options
        self.set_undefined_options('install',('install_layout','install_layout'))
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 296, in set_undefined_options
        setattr(self, dst_option, getattr(src_cmd_obj, src_option))
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
        raise AttributeError(attr)
    AttributeError: install_layout
    [end of output]

pyenv --version: 2.3.23
Also tried reinstalling setuptools (currently version 68.0) and setting SETUPTOOLS_USE_DISTUTILS=stdlib etc. as suggested online to no avail.
Looks like a version mismatch. Is there another python 3.8 minor version to try?

@clundin25
Copy link
Contributor

Do you mind sharing the full trace with me either here or offline? I don't think I have enough info to help debug.

@wj-chen
Copy link
Contributor Author

wj-chen commented Jul 26, 2023

Do you mind sharing the full trace with me either here or offline? I don't think I have enough info to help debug.

Apologies that this has turned into a troubleshooting session but much appreciate your assistance! Sharing the stack trace here for visibility:

wjchen@wjchen:/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python$ python --version
Python 3.8.15
wjchen@wjchen:/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python$ nox -s lint
nox > Running session lint
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/lint
nox > python -m pip install flake8 flake8-import-order docutils click==8.0.4 black==19.3b0
nox > python -m pip install -e .
nox > Command python -m pip install -e . failed with exit code 1:
Obtaining file:///google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting cachetools<6.0,>=2.0.0 (from google-auth==2.22.0)
  Using cached cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Collecting pyasn1-modules>=0.2.1 (from google-auth==2.22.0)
  Using cached pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
Collecting rsa<5,>=3.1.4 (from google-auth==2.22.0)
  Using cached rsa-4.9-py3-none-any.whl (34 kB)
Collecting urllib3<2.0 (from google-auth==2.22.0)
  Using cached urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
Collecting pyasn1<0.6.0,>=0.4.6 (from pyasn1-modules>=0.2.1->google-auth==2.22.0)
  Using cached pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
Installing collected packages: urllib3, pyasn1, cachetools, rsa, pyasn1-modules, google-auth
  Running setup.py develop for google-auth
    error: subprocess-exited-with-error
    
    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [39 lines of output]
        running develop
        /google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/command/easy_install.py:146: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        /google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        Traceback (most recent call last):
          File "<string>", line 2, in <module>
          File "<pip-setuptools-caller>", line 34, in <module>
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/setup.py", line 52, in <module>
            setup(
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/__init__.py", line 108, in setup
            return distutils.core.setup(**attrs)
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
            return run_commands(dist)
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
            dist.run_commands()
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
            self.run_command(cmd)
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/dist.py", line 1213, in run_command
            super().run_command(command)
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
            cmd_obj.ensure_finalized()
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
            self.finalize_options()
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/command/develop.py", line 52, in finalize_options
            easy_install.finalize_options(self)
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 292, in finalize_options
            self.set_undefined_options(
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 293, in set_undefined_options
            src_cmd_obj.ensure_finalized()
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
            self.finalize_options()
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 17, in finalize_options
            self.set_undefined_options('install',('install_layout','install_layout'))
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 296, in set_undefined_options
            setattr(self, dst_option, getattr(src_cmd_obj, src_option))
          File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
            raise AttributeError(attr)
        AttributeError: install_layout
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [39 lines of output]
    running develop
    /google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/command/easy_install.py:146: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    /google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/setup.py", line 52, in <module>
        setup(
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/__init__.py", line 108, in setup
        return distutils.core.setup(**attrs)
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
        return run_commands(dist)
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
        dist.run_commands()
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
        self.run_command(cmd)
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/dist.py", line 1213, in run_command
        super().run_command(command)
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
        cmd_obj.ensure_finalized()
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
        self.finalize_options()
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/command/develop.py", line 52, in finalize_options
        easy_install.finalize_options(self)
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 292, in finalize_options
        self.set_undefined_options(
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 293, in set_undefined_options
        src_cmd_obj.ensure_finalized()
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
        self.finalize_options()
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 17, in finalize_options
        self.set_undefined_options('install',('install_layout','install_layout'))
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 296, in set_undefined_options
        setattr(self, dst_option, getattr(src_cmd_obj, src_option))
      File "/google/src/cloud/wjchen/osconfig-agent-cider/google3/experimental/users/wjchen/google-auth-library-python/.nox/lint/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
        raise AttributeError(attr)
    AttributeError: install_layout
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
nox > Session lint failed.

@clundin25
Copy link
Contributor

Ah interesting. I do not have this issue.

Can you try working from your cloudtop's file system, instead of Piper?

@wj-chen
Copy link
Contributor Author

wj-chen commented Jul 28, 2023

Ah interesting. I do not have this issue.

Can you try working from your cloudtop's file system, instead of Piper?

Thank you! I managed to resolve it. Turned out to be a faulty nox installation and it worked after python3 -m pip install --user nox.

Linter failure has been fixed. default_explicit_authorized_user-3.7 and default_explicit_authorized_user_explicit_project-3.7 tests failed again though...

@wj-chen
Copy link
Contributor Author

wj-chen commented Jul 31, 2023

Thanks for the credential refresh! The checks have passed.

@clundin25 clundin25 added the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 31, 2023
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 31, 2023
@@ -42,32 +44,53 @@
)


def get_config_path():
"""Returns the absolute path the the Cloud SDK's configuration directory.
def get_config_path(config_directory=_CONFIG_DIRECTORY_GCLOUD):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A cursory look suggests this is only used in gcloud's code base.

Can you own making sure that there is no regression when this library is updated in Piper?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes of course. I was actually making a change in the gcloud code base first and that's when I realized the path definitions happen here. How often do you merge into google3 and when will this change be expected to land?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The merge is manual. I can provide instructions for how you can do it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Could you help me reach @sai-sunder-s for a review so I can proceed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @clundin25, I've received an approval from @sai-sunder-s and am ready for merge instructions now.

@wj-chen
Copy link
Contributor Author

wj-chen commented Aug 3, 2023

Hi @sai-sunder-s , let me know if you have any questions about this change

@clundin25 clundin25 merged commit 9f52f66 into googleapis:main Aug 8, 2023
clundin25 added a commit to clundin25/google-auth-library-python that referenced this pull request Aug 9, 2023
clundin25 added a commit that referenced this pull request Aug 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants