Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement git-lfs support #344

Merged
merged 20 commits into from
Aug 4, 2023
Merged

feat: implement git-lfs support #344

merged 20 commits into from
Aug 4, 2023

Conversation

aymanbagabas
Copy link
Member

@aymanbagabas aymanbagabas commented Jul 20, 2023

  • Support Git LFS SSH and HTTP basic transfers (server-side)
  • Implement Git LFS Basic transfer (client)
  • Import missing LFS objects when importing a repository
  • Support user access tokens and JWT to authenticate

TODO:

Fixes: #218
Fixes: #26

@codecov
Copy link

codecov bot commented Jul 20, 2023

Codecov Report

Merging #344 (5e0c14f) into main (050a0d1) will decrease coverage by 0.18%.
Report is 5 commits behind head on main.
The diff coverage is 45.26%.

@@            Coverage Diff             @@
##             main     #344      +/-   ##
==========================================
- Coverage   50.52%   50.35%   -0.18%     
==========================================
  Files         107      133      +26     
  Lines        8209    11272    +3063     
==========================================
+ Hits         4148     5676    +1528     
- Misses       3767     5157    +1390     
- Partials      294      439     +145     
Files Changed Coverage Δ
git/repo.go 50.69% <ø> (+6.71%) ⬆️
server/access/access.go 64.70% <0.00%> (-26.97%) ⬇️
server/backend/hooks.go 0.00% <0.00%> (ø)
server/config/context.go 85.71% <0.00%> (ø)
server/config/file.go 100.00% <ø> (ø)
server/daemon/daemon.go 48.34% <0.00%> (+1.61%) ⬆️
server/db/errors.go 31.57% <ø> (ø)
server/db/logger.go 37.31% <0.00%> (+12.31%) ⬆️
server/db/migrate/migrate.go 32.53% <0.00%> (ø)
server/git/git.go 26.66% <ø> (ø)
... and 67 more

... and 4 files with indirect coverage changes

@aymanbagabas aymanbagabas marked this pull request as ready for review July 26, 2023 21:30
- Support Git LFS SSH transfer (server-side)
- Implement Git LFS Basic transfer (client)
- Import missing LFS objects when importing a repository

wip

fix: user can be nil for anonymous connections

fix: wrap db errors

fix: lint errors
Generate jwt tokens
Implement git-lfs-authenticate to generate tokens through ssh
Authenticate user using HTTP

fix: git lfs endpoint auth

feat: git lfs locks

Implement git lfs locks endpoints

fix: tests

fix: access tokens migration

add expires_at

fix: lint errors

fix: cleanup

Revert "fix: cleanup"

This reverts commit 728173f.

fix(db): don't drop tables
Users now can generate access tokens and use them to authenticate with
Soft Serve HTTP Git server. It supports basic username & password,
generated access tokens, and JWT tokens.

As of now there is no way the user can set a password. This will be
implemented in a separate PR.

Access tokens hashes are stored in the database along with an optional
expiry date.

Access tokens can be used as the Git user or password in a HTTP clone
URL e.g. `https://<token>@git.example.com/repo.git`

fix: lint errors

fix: ensure default branch on http push

fix: address carlos comments
Enable/disable LFS endpoints
Enable/disable SSH LFS transfer

fix: lfs request validation

chore: add jwt, lfs, and http tests

fix: tests race

feat: more tests

fix: return 403 on bad creds
@aymanbagabas aymanbagabas changed the title feat: implement git-lfs preliminary support feat: implement git-lfs support Jul 28, 2023
@aymanbagabas aymanbagabas force-pushed the lfs-ssh branch 7 times, most recently from 567bcbe to e861c74 Compare July 31, 2023 22:19
- Fix git commands errors on invalid args and permissions
- Use Cobra to handle git commands
- Add Git SSH tests
- Better ssh and git pktline error handling
Ensure to pull the latest lfs objects when mirroring a repository with
lfs enabled.
Remove repo files when the repo doesn't exist in the database but exists
in the filesystem
Make it up to the context to continue the process
Prints the repo ssh clone url on create

Fixes: #297
Implement a task manager that can run different tasks given a unique ID.

This is needed to accommodate expensive tasks like importing a large
repository. The current behavior uses the connection's context (the SSH
connection) to import the repository. However, if the server has defined
an SSH `idle_timeout`, `max_timeout`, and/or the connection drops,
Soft Serve cancels the git clone process and aborts importing the
repository.

Instead, we add the import task to the "task manager" and wait on the
connection context. If a task already exists for the same repository,
return `Error: import already in progress`.

Fixes: #348
Now you can add a collaborator with a specific access level

Fixes: #281
@aymanbagabas aymanbagabas merged commit 882e701 into main Aug 4, 2023
@aymanbagabas aymanbagabas deleted the lfs-ssh branch August 4, 2023 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

git lfs support Add support for Git LFS
1 participant