Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import commits from apache/arrow #1

Closed
kou opened this issue Aug 29, 2024 · 12 comments
Closed

Import commits from apache/arrow #1

kou opened this issue Aug 29, 2024 · 12 comments
Milestone

Comments

@kou
Copy link
Member

kou commented Aug 29, 2024

We need only commits for https://github.com/apache/arrow/tree/main/go . Other commits such as commits for .github/ aren't needed. We need to adjust other files for this repository.

We can do this by the following command lines:

git clone [email protected]:apache/arrow-go.git
cd arrow-go
git remote add arrow https://github.com/apache/arrow.git
git fetch arrow
git switch -c arrow-main-go arrow/main
git filter-branch --subdirectory-filter go
git switch main
git rebase arrow-main-go
git push origin --force main
@raulcd
Copy link
Member

raulcd commented Aug 29, 2024

I've tested the commands locally (without the push obviously) and the repository looks good on a first check. Should we run some testing to validate functionality before pushing?

@kou
Copy link
Member Author

kou commented Aug 29, 2024

Good point. We can use go test ./... but we need to setup ARROW_TEST_DATA and PARQUET_TEST_DATA:

ok  	github.com/apache/arrow/go/v18/arrow	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/array	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/arrio	(cached)
?   	github.com/apache/arrow/go/v18/arrow/avro/avro2parquet	[no test files]
ok  	github.com/apache/arrow/go/v18/arrow/avro	(cached)
?   	github.com/apache/arrow/go/v18/arrow/cdata	[no test files]
ok  	github.com/apache/arrow/go/v18/arrow/bitutil	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/compute	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/compute/exec	(cached)
?   	github.com/apache/arrow/go/v18/arrow/compute/internal/kernels	[no test files]
ok  	github.com/apache/arrow/go/v18/arrow/compute/exprs	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/csv	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/decimal128	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/decimal256	(cached)
?   	github.com/apache/arrow/go/v18/arrow/endian	[no test files]
ok  	github.com/apache/arrow/go/v18/arrow/encoded	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/extensions	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/flight	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/flight/flightsql	(cached)
?   	github.com/apache/arrow/go/v18/arrow/flight/flightsql/example	[no test files]
ok  	github.com/apache/arrow/go/v18/arrow/flight/flightsql/driver	(cached)
?   	github.com/apache/arrow/go/v18/arrow/flight/flightsql/example/cmd/sqlite_flightsql_server	[no test files]
?   	github.com/apache/arrow/go/v18/arrow/flight/flightsql/schema_ref	[no test files]
?   	github.com/apache/arrow/go/v18/arrow/flight/gen/flight	[no test files]
ok  	github.com/apache/arrow/go/v18/arrow/flight/session	(cached) [no tests to run]
ok  	github.com/apache/arrow/go/v18/arrow/float16	(cached)
?   	github.com/apache/arrow/go/v18/arrow/internal	[no test files]
?   	github.com/apache/arrow/go/v18/arrow/internal/arrdata	[no test files]
ok  	github.com/apache/arrow/go/v18/arrow/internal/arrjson	(cached)
?   	github.com/apache/arrow/go/v18/arrow/internal/debug	[no test files]
ok  	github.com/apache/arrow/go/v18/arrow/internal/dictutils	(cached)
?   	github.com/apache/arrow/go/v18/arrow/internal/flatbuf	[no test files]
?   	github.com/apache/arrow/go/v18/arrow/internal/flight_integration	[no test files]
?   	github.com/apache/arrow/go/v18/arrow/internal/flight_integration/cmd/arrow-flight-integration-client	[no test files]
?   	github.com/apache/arrow/go/v18/arrow/internal/flight_integration/cmd/arrow-flight-integration-server	[no test files]
?   	github.com/apache/arrow/go/v18/arrow/internal/testing/gen	[no test files]
ok  	github.com/apache/arrow/go/v18/arrow/internal/testing/tools	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/ipc	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/ipc/cmd/arrow-cat	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/ipc/cmd/arrow-file-to-stream	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/ipc/cmd/arrow-json-integration-test	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/ipc/cmd/arrow-ls	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/ipc/cmd/arrow-stream-to-file	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/math	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/memory	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/memory/mallocator	(cached)
ok  	github.com/apache/arrow/go/v18/arrow/scalar	(cached)
?   	github.com/apache/arrow/go/v18/arrow/util/util_message	[no test files]
ok  	github.com/apache/arrow/go/v18/arrow/tensor	(cached)
?   	github.com/apache/arrow/go/v18/internal/json	[no test files]
?   	github.com/apache/arrow/go/v18/internal/types	[no test files]
ok  	github.com/apache/arrow/go/v18/arrow/util	(cached)
ok  	github.com/apache/arrow/go/v18/internal/bitutils	(cached)
ok  	github.com/apache/arrow/go/v18/internal/hashing	(cached)
ok  	github.com/apache/arrow/go/v18/internal/utils	(cached)
?   	github.com/apache/arrow/go/v18/parquet/cmd/parquet_reader	[no test files]
?   	github.com/apache/arrow/go/v18/parquet/cmd/parquet_schema	[no test files]
?   	github.com/apache/arrow/go/v18/parquet/internal/debug	[no test files]
?   	github.com/apache/arrow/go/v18/parquet/internal/encryption	[no test files]
?   	github.com/apache/arrow/go/v18/parquet/internal/gen-go/parquet	[no test files]
?   	github.com/apache/arrow/go/v18/parquet/internal/testutils	[no test files]
?   	github.com/apache/arrow/go/v18/parquet/internal/thrift	[no test files]
ok  	github.com/apache/arrow/go/v18/parquet	0.255s
ok  	github.com/apache/arrow/go/v18/parquet/compress	(cached)
ok  	github.com/apache/arrow/go/v18/parquet/file	17.140s
ok  	github.com/apache/arrow/go/v18/parquet/internal/bmi	(cached)
ok  	github.com/apache/arrow/go/v18/parquet/internal/encoding	4.240s
ok  	github.com/apache/arrow/go/v18/parquet/internal/utils	(cached)
ok  	github.com/apache/arrow/go/v18/parquet/metadata	(cached)
ok  	github.com/apache/arrow/go/v18/parquet/pqarrow	3.482s
ok  	github.com/apache/arrow/go/v18/parquet/schema	(cached)

Hmm. It seems that we misses some files. (There are some "[no test files]".)

I haven't looked into them yet but I think that we can work on them as follow-up tasks.

@joellubi
Copy link
Member

Hmm. It seems that we misses some files. (There are some "[no test files]".)

This doesn't necessarily mean we missed files, not all directories contain test files. I've confirmed I get the same test output running on apache/arrow:main.

@kou
Copy link
Member Author

kou commented Aug 29, 2024

Thanks. I didn't know about it.

@raulcd
Copy link
Member

raulcd commented Aug 29, 2024

Thanks @kou the output LGTM. @joellubi @zeroshade unless you want to do some extra testing I would go with the current.
BTW we should track if there is any new commit being pushed to the main arrow repo on the go side to manually cherry-pick it in the interim.

@zeroshade
Copy link
Member

This LGTM, we can do a follow-up PR that will update all of the import paths across the code-base to the new URL

@kou
Copy link
Member Author

kou commented Aug 29, 2024

Thanks for confirming this approach. I'll do it soon.

FYI: Here are related issues:

BTW we should track if there is any new commit being pushed to the main arrow repo on the go side to manually cherry-pick it in the interim.

apache/arrow#43881 [Go] Notify apache/arrow-go migration to all open PRs

we can do a follow-up PR that will update all of the import paths across the code-base to the new URL

#4 Update package name

@kou
Copy link
Member Author

kou commented Aug 30, 2024

Sorry. I haven't done this yet.
I'll do this now.

@kou
Copy link
Member Author

kou commented Aug 30, 2024

We need to remove the protected branch configuration in .asf.yaml:

$ git push origin --force-with-lease main
Enumerating objects: 13921, done.
Counting objects: 100% (13921/13921), done.
Delta compression using up to 24 threads
Compressing objects: 100% (3993/3993), done.
Writing objects: 100% (13921/13921), 13.79 MiB | 2.75 MiB/s, done.
Total 13921 (delta 9809), reused 13212 (delta 9721), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (9809/9809), done.
remote: error: GH006: Protected branch update failed for refs/heads/main.
remote: error: Cannot force-push to this branch
To github.com:apache/arrow-go.git
 ! [remote rejected]     main -> main (protected branch hook declined)
error: failed to push some refs to 'github.com:apache/arrow-go.git'

kou added a commit to kou/arrow-go that referenced this issue Aug 30, 2024
https://cwiki.apache.org/confluence/pages/viewpage.action?spaceKey=INFRA&title=Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Branchprotection

> To completely remove all branch protection rules, set the
> protected_branches section to null, as such:
>
> Prevent force pushes
>
>     github:
>       protected_branches: ~
@kou
Copy link
Member Author

kou commented Aug 30, 2024

I tried git cherry-pick arrow-main-go instead of git merge to avoid force push. But it causes conflicts...

Can we enable force push temporary? #80

@kou
Copy link
Member Author

kou commented Aug 30, 2024

Oh, (git log --format=%H --reverse arrow-main-go | while read commit; do git cherry-pick $commit; done) works without conflict. I'll push it without #80.

kou pushed a commit that referenced this issue Aug 30, 2024
…comments.

The reset method allow the data structures to be re-used so they don't have to be allocated over and over again.

Closes #6430 from richardartoul/ra/merge-upstream and squashes the following commits:

5a0828187 <Richard Artoul> Add license to test file
d76be05ca <Richard Artoul> Add test for data reset
d102b1fff <Richard Artoul> Add tests
d3e6e6785 <Richard Artoul> cleanup comments
c8525aece <Richard Artoul> Add Reset method to int array (#5)
489ca2593 <Richard Artoul> Fix array.setData() to retain before release (#4)
88cd05ff8 <Richard Artoul> Add reset method to Data (#3)
6d1b2775c <Richard Artoul> Add Reset() method to String array (#2)
dca230383 <Richard Artoul> Add Reset method to buffer and cleanup comments (#1)

Lead-authored-by: Richard Artoul <[email protected]>
Co-authored-by: Richard Artoul <[email protected]>
Signed-off-by: Sebastien Binet <[email protected]>
@kou
Copy link
Member Author

kou commented Aug 30, 2024

Pushed!

@kou kou closed this as completed Aug 30, 2024
@zeroshade zeroshade added this to the 18.0.0 milestone Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants