Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore optional cache target #118

Merged
merged 8 commits into from
Nov 22, 2023
Merged

Restore optional cache target #118

merged 8 commits into from
Nov 22, 2023

Conversation

sbquinlan
Copy link
Member

Currently leaving the CacheTarget as it's default empty string throws:

(base) [ec2-user@ip-172-31-19-37 pangeo-forge-recipes]$ pangeo-forge-runner bake --repo=$(pwd) --feedstock-subdir=./recipes/mursst/
Target Storage is FSSpecTarget(LocalFileSystem(, root_path="/home/ec2-user/pangeo-forge-recipes/data/{job_name}")

Input Cache Storage is CacheFSSpecTarget(AbstractFileSystem(, root_path="")

Metadata Cache Storage is MetadataTarget(LocalFileSystem(, root_path="/home/ec2-user/pangeo-forge-recipes/data/{job_name}/cache/metadata")

Traceback (most recent call last):
  File "/home/ec2-user/micromamba/bin/pangeo-forge-runner", line 8, in <module>
    sys.exit(main())
  File "/home/ec2-user/micromamba/lib/python3.9/site-packages/pangeo_forge_runner/cli.py", line 28, in main
    app.start()
  File "/home/ec2-user/micromamba/lib/python3.9/site-packages/pangeo_forge_runner/cli.py", line 23, in start
    super().start()
  File "/home/ec2-user/micromamba/lib/python3.9/site-packages/traitlets/config/application.py", line 474, in start
    return self.subapp.start()
  File "/home/ec2-user/micromamba/lib/python3.9/site-packages/pangeo_forge_runner/commands/bake.py", line 184, in start
    cache_target = input_cache_storage.get_forge_target(job_name=self.job_name)
  File "/home/ec2-user/micromamba/lib/python3.9/site-packages/pangeo_forge_runner/storage.py", line 59, in get_forge_target
    return cls(
  File "<string>", line 5, in __init__
  File "/home/ec2-user/micromamba/lib/python3.9/site-packages/pangeo_forge_recipes/storage.py", line 148, in __post_init__
    if not self.fs.isdir(self.root_path):
  File "/home/ec2-user/micromamba/lib/python3.9/site-packages/fsspec/spec.py", line 721, in isdir
    return self.info(path)["type"] == "directory"
  File "/home/ec2-user/micromamba/lib/python3.9/site-packages/fsspec/spec.py", line 681, in info
    out = self.ls(self._parent(path), detail=True, **kwargs)
  File "/home/ec2-user/micromamba/lib/python3.9/site-packages/fsspec/spec.py", line 350, in ls
    raise NotImplementedError
NotImplementedError

This change fixes that by restoring the former behavior of ignoring the default config (of empty string).

@codecov
Copy link

codecov bot commented Oct 27, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (d31f4e0) 96.36% compared to head (79fb5df) 96.36%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #118   +/-   ##
=======================================
  Coverage   96.36%   96.36%           
=======================================
  Files          15       15           
  Lines         495      495           
=======================================
  Hits          477      477           
  Misses         18       18           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@cisaacstern cisaacstern requested a review from yuvipanda October 27, 2023 04:51
@cisaacstern
Copy link
Member

LGTM @sbquinlan!

A basic test would be great.

I don't think we have a test_storage.py yet, so testing might require adding that module.

@cisaacstern
Copy link
Member

Just a note to say IMO we can merge this as soon as a test is added.

@sbquinlan
Copy link
Member Author

@cisaacstern yeah I'm just caught up in other things. I'll try come back to this pr this week.

@cisaacstern
Copy link
Member

No rush from my end! Thanks!

@cisaacstern cisaacstern added test-flink Add this label to PRs to trigger Dataflow integration test. test-dataflow Add this label to PRs to trigger Dataflow integration test. and removed test-flink Add this label to PRs to trigger Dataflow integration test. labels Nov 21, 2023
@cisaacstern
Copy link
Member

Dataflow tests are failing because the head branch is from a fork, so credentials are not available in Github Workflows.

But that's okay, because the unit tests are convincing in themselves!

Thanks @sbquinlan!

@cisaacstern cisaacstern merged commit 95ce6f4 into pangeo-forge:main Nov 22, 2023
10 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test-dataflow Add this label to PRs to trigger Dataflow integration test.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants