-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[yaml] add FileIO docs #33185
base: master
Are you sure you want to change the base?
[yaml] add FileIO docs #33185
Conversation
R: @robertwb |
R: @damccorm |
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks
Oh, please fix the precommits |
7ae8638
to
5c3904d
Compare
Looks like precommits are still failing |
629390a
to
50b4cef
Compare
Signed-off-by: Jeffrey Kinard <[email protected]>
Signed-off-by: Jeffrey Kinard <[email protected]>
Signed-off-by: Jeffrey Kinard <[email protected]>
50b4cef
to
2eac016
Compare
Signed-off-by: Jeffrey Kinard <[email protected]>
2eac016
to
c90624e
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #33185 +/- ##
=========================================
Coverage 57.41% 57.42%
Complexity 1474 1474
=========================================
Files 970 970
Lines 154498 154526 +28
Branches 1076 1076
=========================================
+ Hits 88708 88732 +24
- Misses 63586 63590 +4
Partials 2204 2204
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure we want to go this direction; re-using the docs from the original transforms rather than duplicating them is a feature, not a bug. Certainly they could be improved (which would improve both).
@@ -86,10 +86,16 @@ def _fake_value(name, beam_type): | |||
raise ValueError(f"Unrecognized type_info: {type_info!r}") | |||
|
|||
|
|||
EXCLUDE_ARGS = ['args', 'kwargs'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can't always filter these out, e.g. https://beam.apache.org/releases/yamldoc/current/#pytransform
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah.. right.. let me rethink
<https://pandas.pydata.org/docs/reference/api/pandas.read_json.html> | ||
|
||
Args: | ||
path (str): The file path to read from as a local file path or a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that we support more than local files and gcs; the set of filesystems is not closed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, I also found that limiting, but I copied that description from another docstring, e.g.
beam/sdks/python/apache_beam/io/textio.py
Lines 757 to 759 in 142e392
file_pattern (str): The file path to read from as a local file path or a | |
GCS ``gs://`` path. The path can contain glob characters | |
(``*``, ``?``, and ``[...]`` sets). |
Signed-off-by: Jeffrey Kinard <[email protected]>
Add docs for all the FileIO YAML Providers. This includes separating out the providers into standalone function calls rather than using a remaning provider referencing th source transform. This allows for YAML-specific docs and possibe future YAML-specific pre/post processing
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123
), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>
instead.CHANGES.md
with noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.