Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LFS Migration and Mirror #14726

Merged
merged 85 commits into from
Apr 8, 2021
Merged
Show file tree
Hide file tree
Changes from 79 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
5e4642d
Implemented LFS client.
KN4CK3R Feb 18, 2021
6a236f1
Implemented scanning for pointer files.
KN4CK3R Feb 18, 2021
ac8127b
Implemented downloading of lfs files.
KN4CK3R Feb 18, 2021
0189942
Moved model-dependent code into services.
KN4CK3R Feb 18, 2021
9aa898f
Removed models dependency. Added TryReadPointerFromBuffer.
KN4CK3R Feb 18, 2021
4fd390f
Migrated code from service to module.
KN4CK3R Feb 18, 2021
f94c599
Centralised storage creation.
KN4CK3R Feb 18, 2021
85621c9
Removed dependency from models.
KN4CK3R Feb 18, 2021
ae60e34
Moved ContentStore into modules.
KN4CK3R Feb 18, 2021
37fa1c2
Share structs between server and client.
KN4CK3R Feb 18, 2021
c6ce58c
Moved method to services.
KN4CK3R Feb 18, 2021
7a86be8
Implemented lfs download on clone.
KN4CK3R Feb 18, 2021
68e34cd
Implemented LFS sync on clone and mirror update.
KN4CK3R Feb 18, 2021
4775b80
Added form fields.
KN4CK3R Feb 18, 2021
e458d64
Updated templates.
KN4CK3R Feb 18, 2021
4d2abc3
Fixed condition.
KN4CK3R Feb 18, 2021
a5e64dd
Use alternate endpoint.
KN4CK3R Feb 18, 2021
5615d1e
Added missing methods.
KN4CK3R Feb 19, 2021
a6b0d26
Fixed typo and make linter happy.
KN4CK3R Feb 19, 2021
dd889c6
Detached pointer parser from gogit dependency.
KN4CK3R Feb 19, 2021
86acab3
Fixed TestGetLFSRange test.
KN4CK3R Feb 19, 2021
534886c
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Feb 20, 2021
a951992
Added context to support cancellation.
KN4CK3R Feb 20, 2021
3b69a7d
Use ReadFull to probably read more data.
KN4CK3R Feb 21, 2021
ceefad0
Removed duplicated code from models.
KN4CK3R Feb 21, 2021
ef86fa8
Moved scan implementation into pointer_scanner_nogogit.
KN4CK3R Feb 21, 2021
bffe86c
Changed method name.
KN4CK3R Feb 21, 2021
50d4814
Added comments.
KN4CK3R Feb 21, 2021
e9024a5
Added more/specific log/error messages.
KN4CK3R Feb 21, 2021
59a55b3
Embedded lfs.Pointer into models.LFSMetaObject.
KN4CK3R Feb 22, 2021
649d236
Moved code from models to module.
KN4CK3R Feb 22, 2021
73ce39d
Moved code from models to module.
KN4CK3R Feb 22, 2021
80fe20b
Moved code from models to module.
KN4CK3R Feb 22, 2021
37a1d02
Reduced pointer usage.
KN4CK3R Feb 22, 2021
b0e0804
Embedded type.
KN4CK3R Feb 22, 2021
3779dea
Use promoted fields.
KN4CK3R Feb 22, 2021
932a370
Fixed unexpected eof.
KN4CK3R Feb 23, 2021
6a47890
Added unit tests.
KN4CK3R Feb 23, 2021
55618f0
Merge branch 'master' into feature-lfs-clone
KN4CK3R Mar 5, 2021
bbac160
Merge branch 'master' into feature-lfs-clone
KN4CK3R Mar 7, 2021
64dc148
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Mar 8, 2021
3e30421
Implemented migration of local file paths.
KN4CK3R Mar 8, 2021
0ef54f7
Show an error on invalid LFS endpoints.
KN4CK3R Mar 8, 2021
a44781f
Hide settings if not used.
KN4CK3R Mar 8, 2021
856416a
Added LFS info to mirror struct.
KN4CK3R Mar 10, 2021
5e9c606
Fixed comment.
KN4CK3R Mar 10, 2021
c0b5e50
Check LFS endpoint.
KN4CK3R Mar 10, 2021
4c127de
Manage LFS settings from mirror page.
KN4CK3R Mar 10, 2021
9cf903c
Fixed selector.
KN4CK3R Mar 10, 2021
745812b
Adjusted selector.
KN4CK3R Mar 10, 2021
8808d77
Added more tests.
KN4CK3R Mar 11, 2021
4fcc2f6
Added local filesystem migration test.
KN4CK3R Mar 11, 2021
3283e58
Fixed typo.
KN4CK3R Mar 11, 2021
d190403
Reset settings.
KN4CK3R Mar 11, 2021
4e864f3
Added special windows path handling.
KN4CK3R Mar 11, 2021
750ad79
Added unit test for HTTPClient.
KN4CK3R Mar 12, 2021
8fd1f7d
Added unit test for BasicTransferAdapter.
KN4CK3R Mar 12, 2021
028cbd9
Moved into util package.
KN4CK3R Mar 13, 2021
8df4b21
Test if LFS endpoint is allowed.
KN4CK3R Mar 13, 2021
72b2a08
Merge branch 'master' into feature-lfs-clone
KN4CK3R Mar 13, 2021
d8844ba
Merge branch 'master' into feature-lfs-clone
KN4CK3R Mar 15, 2021
27ad4e4
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Mar 16, 2021
5b33ef4
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Mar 17, 2021
184a949
Added support for git://
KN4CK3R Mar 17, 2021
5a1f3ed
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Mar 18, 2021
b7774e5
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Mar 21, 2021
ba58438
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Mar 21, 2021
a727b46
Just use a static placeholder as the displayed url may be invalid.
KN4CK3R Mar 21, 2021
60a4177
Reverted to original code.
KN4CK3R Mar 21, 2021
2555639
Added "Advanced Settings".
KN4CK3R Mar 23, 2021
1eae7e1
Updated wording.
KN4CK3R Mar 23, 2021
70d0594
Added discovery info link.
KN4CK3R Mar 23, 2021
8b4fedd
Implemented suggestion.
KN4CK3R Mar 23, 2021
1c7181d
Fixed missing format parameter.
KN4CK3R Mar 25, 2021
2a80bd9
Added Pointer.IsValid().
KN4CK3R Mar 25, 2021
b49dc68
Always remove model on error.
KN4CK3R Mar 27, 2021
7aeeaf9
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Apr 5, 2021
32058a1
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Apr 6, 2021
71477b9
Merge branch 'master' of https://github.com/go-gitea/gitea into featu…
KN4CK3R Apr 7, 2021
a9b3ca2
Added suggestions.
KN4CK3R Apr 7, 2021
b5c1649
Use channel instead of array.
KN4CK3R Apr 8, 2021
70a4b72
Update routers/repo/migrate.go
zeripath Apr 8, 2021
45d560c
Merge branch 'master' into feature-lfs-clone
zeripath Apr 8, 2021
0a2593d
fmt
zeripath Apr 8, 2021
0592552
Merge branch 'master' into feature-lfs-clone
zeripath Apr 8, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cmd/serv.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ import (
"time"

"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/lfs"
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/pprof"
"code.gitea.io/gitea/modules/private"
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/services/lfs"

"github.com/dgrijalva/jwt-go"
jsoniter "github.com/json-iterator/go"
Expand Down
49 changes: 49 additions & 0 deletions integrations/api_repo_lfs_migrate_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package integrations

import (
"net/http"
"path"
"testing"

"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/lfs"
"code.gitea.io/gitea/modules/setting"
api "code.gitea.io/gitea/modules/structs"

"github.com/stretchr/testify/assert"
)

func TestAPIRepoLFSMigrateLocal(t *testing.T) {
defer prepareTestEnv(t)()

oldImportLocalPaths := setting.ImportLocalPaths
oldAllowLocalNetworks := setting.Migrations.AllowLocalNetworks
setting.ImportLocalPaths = true
setting.Migrations.AllowLocalNetworks = true

user := models.AssertExistsAndLoadBean(t, &models.User{ID: 1}).(*models.User)
session := loginUser(t, user.Name)
token := getTokenForLoggedInUser(t, session)

req := NewRequestWithJSON(t, "POST", "/api/v1/repos/migrate?token="+token, &api.MigrateRepoOptions{
CloneAddr: path.Join(setting.RepoRootPath, "migration/lfs-test.git"),
RepoOwnerID: user.ID,
RepoName: "lfs-test-local",
LFS: true,
})
resp := MakeRequest(t, req, NoExpectedStatus)
assert.EqualValues(t, http.StatusCreated, resp.Code)

store := lfs.NewContentStore()
ok, _ := store.Verify(lfs.Pointer{Oid: "fb8f7d8435968c4f82a726a92395be4d16f2f63116caf36c8ad35c60831ab041", Size: 6})
assert.True(t, ok)
ok, _ = store.Verify(lfs.Pointer{Oid: "d6f175817f886ec6fbbc1515326465fa96c3bfd54a4ea06cfd6dbbd8340e0152", Size: 6})
assert.True(t, ok)

setting.ImportLocalPaths = oldImportLocalPaths
setting.Migrations.AllowLocalNetworks = oldAllowLocalNetworks
}
5 changes: 3 additions & 2 deletions integrations/git_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (

"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/git"
"code.gitea.io/gitea/modules/lfs"
"code.gitea.io/gitea/modules/setting"
api "code.gitea.io/gitea/modules/structs"
"code.gitea.io/gitea/modules/util"
Expand Down Expand Up @@ -218,7 +219,7 @@ func rawTest(t *testing.T, ctx *APITestContext, little, big, littleLFS, bigLFS s
assert.NotEqual(t, littleSize, resp.Body.Len())
assert.LessOrEqual(t, resp.Body.Len(), 1024)
if resp.Body.Len() != littleSize && resp.Body.Len() <= 1024 {
assert.Contains(t, resp.Body.String(), models.LFSMetaFileIdentifier)
assert.Contains(t, resp.Body.String(), lfs.MetaFileIdentifier)
}
}

Expand All @@ -232,7 +233,7 @@ func rawTest(t *testing.T, ctx *APITestContext, little, big, littleLFS, bigLFS s
resp := session.MakeRequest(t, req, http.StatusOK)
assert.NotEqual(t, bigSize, resp.Body.Len())
if resp.Body.Len() != bigSize && resp.Body.Len() <= 1024 {
assert.Contains(t, resp.Body.String(), models.LFSMetaFileIdentifier)
assert.Contains(t, resp.Body.String(), lfs.MetaFileIdentifier)
}
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ref: refs/heads/master
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[core]
bare = false
repositoryformatversion = 0
filemode = false
symlinks = false
ignorecase = true
logallrefupdates = true
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Unnamed repository; edit this file 'description' to name the repository.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-checkout.\n"; exit 2; }
git lfs post-checkout "$@"
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-commit.\n"; exit 2; }
git lfs post-commit "$@"
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-merge.\n"; exit 2; }
git lfs post-merge "$@"
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/pre-push.\n"; exit 2; }
git lfs pre-push "$@"
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
dummy2
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
dummy1
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
546244003622c64b2fc3c2cd544d7a29882c8383
27 changes: 7 additions & 20 deletions integrations/lfs_getobject_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,6 @@ package integrations
import (
"archive/zip"
"bytes"
"crypto/sha256"
"encoding/hex"
"io"
"io/ioutil"
"net/http"
"net/http/httptest"
Expand All @@ -18,46 +15,36 @@ import (
"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/lfs"
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/storage"
"code.gitea.io/gitea/routers/routes"

gzipp "github.com/klauspost/compress/gzip"
"github.com/stretchr/testify/assert"
)

func GenerateLFSOid(content io.Reader) (string, error) {
h := sha256.New()
if _, err := io.Copy(h, content); err != nil {
return "", err
}
sum := h.Sum(nil)
return hex.EncodeToString(sum), nil
}

var lfsID = int64(20000)

func storeObjectInRepo(t *testing.T, repositoryID int64, content *[]byte) string {
oid, err := GenerateLFSOid(bytes.NewReader(*content))
pointer, err := lfs.GeneratePointer(bytes.NewReader(*content))
assert.NoError(t, err)
var lfsMetaObject *models.LFSMetaObject

if setting.Database.UsePostgreSQL {
lfsMetaObject = &models.LFSMetaObject{ID: lfsID, Oid: oid, Size: int64(len(*content)), RepositoryID: repositoryID}
lfsMetaObject = &models.LFSMetaObject{ID: lfsID, Pointer: pointer, RepositoryID: repositoryID}
} else {
lfsMetaObject = &models.LFSMetaObject{Oid: oid, Size: int64(len(*content)), RepositoryID: repositoryID}
lfsMetaObject = &models.LFSMetaObject{Pointer: pointer, RepositoryID: repositoryID}
}

lfsID++
lfsMetaObject, err = models.NewLFSMetaObject(lfsMetaObject)
assert.NoError(t, err)
contentStore := &lfs.ContentStore{ObjectStorage: storage.LFS}
exist, err := contentStore.Exists(lfsMetaObject)
contentStore := lfs.NewContentStore()
exist, err := contentStore.Exists(pointer)
assert.NoError(t, err)
if !exist {
err := contentStore.Put(lfsMetaObject, bytes.NewReader(*content))
err := contentStore.Put(pointer, bytes.NewReader(*content))
assert.NoError(t, err)
}
return oid
return pointer.Oid
}

func storeAndGetLfs(t *testing.T, content *[]byte, extraHeader *http.Header, expectedStatus int) *httptest.ResponseRecorder {
Expand Down
117 changes: 117 additions & 0 deletions integrations/lfs_local_endpoint_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package integrations

import (
"fmt"
"io/ioutil"
"net/url"
"os"
"path/filepath"
"testing"

"code.gitea.io/gitea/modules/lfs"

"github.com/stretchr/testify/assert"
)

func str2url(raw string) *url.URL {
u, _ := url.Parse(raw)
return u
}

func TestDetermineLocalEndpoint(t *testing.T) {
defer prepareTestEnv(t)()

root, _ := ioutil.TempDir("", "lfs_test")
defer os.RemoveAll(root)

rootdotgit, _ := ioutil.TempDir("", "lfs_test")
defer os.RemoveAll(rootdotgit)
os.Mkdir(filepath.Join(rootdotgit, ".git"), 0700)

lfsroot, _ := ioutil.TempDir("", "lfs_test")
defer os.RemoveAll(lfsroot)

// Test cases
var cases = []struct {
cloneurl string
lfsurl string
expected *url.URL
}{
// case 0
{
cloneurl: root,
lfsurl: "",
expected: str2url(fmt.Sprintf("file://%s", root)),
},
// case 1
{
cloneurl: root,
lfsurl: lfsroot,
expected: str2url(fmt.Sprintf("file://%s", lfsroot)),
},
// case 2
{
cloneurl: "https://git.com/repo.git",
lfsurl: lfsroot,
expected: str2url(fmt.Sprintf("file://%s", lfsroot)),
},
// case 3
{
cloneurl: rootdotgit,
lfsurl: "",
expected: str2url(fmt.Sprintf("file://%s", filepath.Join(rootdotgit, ".git"))),
},
// case 4
{
cloneurl: "",
lfsurl: rootdotgit,
expected: str2url(fmt.Sprintf("file://%s", filepath.Join(rootdotgit, ".git"))),
},
// case 5
{
cloneurl: rootdotgit,
lfsurl: rootdotgit,
expected: str2url(fmt.Sprintf("file://%s", filepath.Join(rootdotgit, ".git"))),
},
// case 6
{
cloneurl: fmt.Sprintf("file://%s", root),
lfsurl: "",
expected: str2url(fmt.Sprintf("file://%s", root)),
},
// case 7
{
cloneurl: fmt.Sprintf("file://%s", root),
lfsurl: fmt.Sprintf("file://%s", lfsroot),
expected: str2url(fmt.Sprintf("file://%s", lfsroot)),
},
// case 8
{
cloneurl: root,
lfsurl: fmt.Sprintf("file://%s", lfsroot),
expected: str2url(fmt.Sprintf("file://%s", lfsroot)),
},
// case 9
{
cloneurl: "",
lfsurl: "/does/not/exist",
expected: nil,
},
// case 10
{
cloneurl: "",
lfsurl: "file:///does/not/exist",
expected: str2url("file:///does/not/exist"),
},
}

for n, c := range cases {
ep := lfs.DetermineEndpoint(c.cloneurl, c.lfsurl)

assert.Equal(t, c.expected, ep, "case %d: error should match", n)
}
}
Loading