Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

download: CSV/TSV aware download command #654

Open
jqnatividad opened this issue Dec 13, 2022 · 4 comments
Open

download: CSV/TSV aware download command #654

jqnatividad opened this issue Dec 13, 2022 · 4 comments
Assignees
@jqnatividad
Labels
CKAN interoperability with CKAN Data Management System enhancement New feature or request. Once marked with this label, its in the backlog.

Comments

@jqnatividad
Copy link
Collaborator

jqnatividad commented Dec 13, 2022 •
edited
Loading

The download command will be CSV/TSV aware in that:

  • you can download the first n --sample records
  • and for servers that support Range Requests, be able to specify an offset (in MB) before downloading the following n --sample records, sniffing the head of the CSV first to determine the CSV schema
  • take advantage of the same Range Request server support typically used for resumable downloads, to get new records that have been appended to a remote CSV file, downloading only the new records and appending it to the local copy.
  • be able to download multiple files (using http2 and brotli/gzip/deflate compression if the server supports it) using a glob pattern if a web server supports directory listing
  • support http, https, ftp, sftp & s3 url schemes
@jqnatividad jqnatividad added the enhancement New feature or request. Once marked with this label, its in the backlog. label Dec 13, 2022
@jqnatividad
Copy link
Collaborator Author

  • also add --random sampling option if the server supports Range Requests

This was referenced Jan 4, 2023
Closed
Closed
@github-actions
Copy link

Stale issue message

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 5, 2023
@jqnatividad jqnatividad reopened this Mar 5, 2023
@jqnatividad jqnatividad self-assigned this Mar 25, 2023
@jqnatividad
Copy link
Collaborator Author

also add support for "ckan://" scheme

@jqnatividad jqnatividad added the CKAN interoperability with CKAN Data Management System label Apr 7, 2023
@jqnatividad jqnatividad added the WIP work in progress label May 8, 2023
@github-actions
Copy link

github-actions bot commented Jul 7, 2023

Stale issue message

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 15, 2023
@jqnatividad jqnatividad mentioned this issue Aug 6, 2024
Open
@jqnatividad jqnatividad reopened this Aug 6, 2024
@jqnatividad jqnatividad removed no-issue-activity WIP work in progress labels Aug 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jqnatividad jqnatividad

Labels
CKAN interoperability with CKAN Data Management System enhancement New feature or request. Once marked with this label, its in the backlog.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jqnatividad