Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add pkg extractor #5

Merged
merged 41 commits into from
Dec 29, 2023
Merged

feat: add pkg extractor #5

merged 41 commits into from
Dec 29, 2023

Conversation

leohhhn
Copy link
Contributor

@leohhhn leohhhn commented Oct 4, 2023

Description

This PR adds a Go program that extracts Gno packages from log files, later to be uploaded to a separate archive repo.

@leohhhn
Copy link
Contributor Author

leohhhn commented Oct 4, 2023 •
edited
Loading

Current state of the extractor is the following:

  • Discovers .log files with root at LOGS_DIR, set in extractor/.env
  • Unmarshals log entries and extracts packages into EXTRACTION_DIR, keeping the path structure same as GnoLand
  • Adds metadata file to each package folder, which includes creator of package, the deposit upon package creation, and is a place for useful info in the future (i.e. timestamp, block height, chain-id, etc.)

Blockers:

  • How and where do we upload the archived packages? Should we make a gno-bot GH account for this?

TODO:

  • Write a script to upload the extracted packages to an archive repo

Down the line:

  • Modify extractor script to match Milos' txsync & indexer efforts

@leohhhn leohhhn requested a review from zivkovicmilos October 4, 2023 16:56
thehowl
thehowl reviewed Oct 5, 2023
View reviewed changes
extractor/extractor.go Outdated Show resolved Hide resolved
@leohhhn leohhhn self-assigned this Oct 5, 2023
@leohhhn leohhhn mentioned this pull request Oct 6, 2023
Open
zivkovicmilos
zivkovicmilos reviewed Oct 9, 2023
View reviewed changes
extractor/helpers.go Outdated Show resolved Hide resolved
zivkovicmilos
zivkovicmilos reviewed Oct 9, 2023
View reviewed changes
extractor/.gitignore Outdated Show resolved Hide resolved
zivkovicmilos
zivkovicmilos reviewed Oct 9, 2023
View reviewed changes
extractor/README.md Outdated Show resolved Hide resolved
@leohhhn leohhhn marked this pull request as ready for review October 20, 2023 15:06
@leohhhn leohhhn mentioned this pull request Oct 20, 2023
Closed
thehowl
thehowl reviewed Oct 26, 2023
View reviewed changes
extractor/main.go Outdated Show resolved Hide resolved
thehowl
thehowl reviewed Oct 26, 2023
View reviewed changes
Copy link
Member

@thehowl thehowl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah mostly fine. just trying to run it myself but I think it's good to merge aside from some minor changes.

extractor/main.go Outdated Show resolved Hide resolved
zivkovicmilos
zivkovicmilos requested changes Oct 27, 2023
View reviewed changes
Copy link
Member

@zivkovicmilos zivkovicmilos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but I've left some comments that need to be addressed 💯

extractor/main_test.go Outdated Show resolved Hide resolved
extractor/main_test.go Show resolved Hide resolved
extractor/main_test.go Outdated Show resolved Hide resolved
extractor/main_test.go Show resolved Hide resolved
extractor/main_test.go Outdated Show resolved Hide resolved
thehowl
thehowl reviewed Oct 27, 2023
View reviewed changes
extractor/main_test.go Outdated Show resolved Hide resolved
@leohhhn leohhhn requested a review from zivkovicmilos October 27, 2023 21:39
zivkovicmilos
zivkovicmilos requested changes Nov 30, 2023
View reviewed changes
Copy link
Member

@zivkovicmilos zivkovicmilos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've left a second round of reviews, please see the comments and we can go from there 🙏

extractor/main.go Outdated Show resolved Hide resolved
extractor/main.go Outdated Show resolved Hide resolved
extractor/main.go Outdated Show resolved Hide resolved
extractor/main.go Outdated Show resolved Hide resolved
extractor/main.go Outdated Show resolved Hide resolved
extractor/main_test.go Outdated Show resolved Hide resolved
extractor/main_test.go Outdated Show resolved Hide resolved
extractor/main_test.go Show resolved Hide resolved
extractor/main_test.go Outdated Show resolved Hide resolved
extractor/main_test.go Outdated Show resolved Hide resolved
ajnavarro
ajnavarro requested changes Dec 18, 2023
View reviewed changes
Copy link
Contributor

@ajnavarro ajnavarro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a bunch of nits, but I found a swallowed error that we have to fix!

extractor/main.go Outdated Show resolved Hide resolved
extractor/main.go Outdated Show resolved Hide resolved
extractor/main.go Outdated Show resolved Hide resolved
extractor/main.go Outdated Show resolved Hide resolved
extractor/main.go Outdated Show resolved Hide resolved
extractor/main.go Outdated Show resolved Hide resolved
extractor/main.go Show resolved Hide resolved
extractor/main.go Outdated Show resolved Hide resolved
extractor/main.go Outdated Show resolved Hide resolved
@zivkovicmilos zivkovicmilos mentioned this pull request Dec 20, 2023
Open
leohhhn and others added 9 commits December 21, 2023 08:51
@leohhhn @ajnavarro
rename error
36253a3 
Co-authored-by: Antonio Navarro Perez <[email protected]>
@leohhhn @ajnavarro
remove unused anon vars
b56850a 
Co-authored-by: Antonio Navarro Perez <[email protected]>
@leohhhn @ajnavarro
edit comment
0932074 
Co-authored-by: Antonio Navarro Perez <[email protected]>
@leohhhn @ajnavarro
rename error
55c4fa0 
Co-authored-by: Antonio Navarro Perez <[email protected]>
@leohhhn @ajnavarro
update map storage for paths
afc7487 
Co-authored-by: Antonio Navarro Perez <[email protected]>
@leohhhn @ajnavarro
update map storage for paths gnolang#2
3a86407 
Co-authored-by: Antonio Navarro Perez <[email protected]>
@leohhhn @ajnavarro
add conversion check
01ca7fb 
Co-authored-by: Antonio Navarro Perez <[email protected]>
@leohhhn @ajnavarro
add error check
6b6349c 
Co-authored-by: Antonio Navarro Perez <[email protected]>
@leohhhn
format & handle errors
928801c
@leohhhn leohhhn requested a review from ajnavarro December 21, 2023 08:04
ajnavarro
ajnavarro requested changes Dec 21, 2023
View reviewed changes
extractor/main.go Outdated Show resolved Hide resolved
@ajnavarro ajnavarro self-requested a review December 21, 2023 09:14
zivkovicmilos
zivkovicmilos approved these changes Dec 29, 2023
View reviewed changes
Copy link
Member

@zivkovicmilos zivkovicmilos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great now 💯

Thank you for looking into the conversations

@leohhhn leohhhn merged commit ca7cf22 into gnolang:main Dec 29, 2023
@leohhhn leohhhn deleted the feat/extractor branch December 29, 2023 15:29
@leohhhn leohhhn mentioned this pull request Jan 9, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thehowl thehowl thehowl approved these changes

@zivkovicmilos zivkovicmilos zivkovicmilos approved these changes

@ajnavarro ajnavarro ajnavarro approved these changes

@moul moul moul approved these changes

Assignees

@leohhhn leohhhn

Labels
None yet
Projects
👥 Team: Devrels
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@leohhhn @moul @ajnavarro @thehowl @zivkovicmilos