On Prem Azure Devops Migration fails #19772
Comments
|
Also tested on debian 11 with So git on Linux in my very simple test does simply not handle escaping of characters correctly, |
|
Tested with a different username and a plain password without any special characters in it - and its the same issue. I think git does not support proper http authentication as such, i.e. challenge and response. When I use the same password with Then wget properly encodes a response with the username/password the and request suceeds. So possibly its a git on linux vs azure devops on prem incompatibility - and what might need to be done instead is use ssh for git cloning, since that would fix the issue entirely. |
|
I can verify that adding a ssh public key to azure devops allows me to git clone. Basically I would guess that what needs to be done is
Success. If I knew how to program in GO and my UI skills weren't shit - I would create a pull request for this. |
6543
added
type/bug
type/upstream
|
related: #1635 |
|
Is this still an issue? I'v tested it with a pull and push mirror and both worked. |
Yes, I just tried to fire up my gitea installation - updated to 17.3 and same exact issue. Authentication error as I described above. What type of migration did you try to do? And please provide exact steps, was it http/https or other url you used?
|
|
@kdumontnu asked me about this issue, so I registered at Devops and created a repo. The Devops UI told me to use That resulted in After that I created a (mirror) migration for that repo and that worked too. |
You are using azure devops "cloud version" - this issue is about on prem dev ops. Also - I am not trying to set up a mirror - simply migrate a git repository from on prem azure devops to gitea. |
|
Oh, sorry about that. I will try with an on prem dev ops. |
No worries - but make sure you try a migration - and not try to set up any mirrors. |
|
Got it working. Basically the on prem Devops needs the I will not create a PR for that because this should be implemented as a generic "add custom headers to git requests" instead of just the These are the code changes I made: diff --git a/modules/git/repo.go b/modules/git/repo.go
index 8ba3ae4fd..f601b7300 100644
--- a/modules/git/repo.go
+++ b/modules/git/repo.go
@@ -98,16 +98,17 @@ func (repo *Repository) IsEmpty() (bool, error) {
// CloneRepoOptions options when clone a repository
type CloneRepoOptions struct {
- Timeout time.Duration
- Mirror bool
- Bare bool
- Quiet bool
- Branch string
- Shared bool
- NoCheckout bool
- Depth int
- Filter string
- SkipTLSVerify bool
+ Timeout time.Duration
+ Mirror bool
+ Bare bool
+ Quiet bool
+ Branch string
+ Shared bool
+ NoCheckout bool
+ Depth int
+ Filter string
+ SkipTLSVerify bool
+ AuthorizationHeader string
}
// Clone clones original repository to target path.
@@ -126,6 +127,9 @@ func CloneWithArgs(ctx context.Context, args []CmdArg, from, to string, opts Clo
if opts.SkipTLSVerify {
cmd.AddArguments("-c", "http.sslVerify=false")
}
+ if opts.AuthorizationHeader != "" {
+ cmd.AddArguments("-c").AddDynamicArguments("http.extraHeader=" + opts.AuthorizationHeader)
+ }
if opts.Mirror {
cmd.AddArguments("--mirror")
}
diff --git a/modules/migration/options.go b/modules/migration/options.go
index 1e92a1b0b..944d83bc1 100644
--- a/modules/migration/options.go
+++ b/modules/migration/options.go
@@ -11,13 +11,15 @@ import "code.gitea.io/gitea/modules/structs"
// this is for internal usage by migrations module and func who interact with it
type MigrateOptions struct {
// required: true
- CloneAddr string `json:"clone_addr" binding:"Required"`
- CloneAddrEncrypted string `json:"clone_addr_encrypted,omitempty"`
- AuthUsername string `json:"auth_username"`
- AuthPassword string `json:"-"`
- AuthPasswordEncrypted string `json:"auth_password_encrypted,omitempty"`
- AuthToken string `json:"-"`
- AuthTokenEncrypted string `json:"auth_token_encrypted,omitempty"`
+ CloneAddr string `json:"clone_addr" binding:"Required"`
+ CloneAddrEncrypted string `json:"clone_addr_encrypted,omitempty"`
+ AuthUsername string `json:"auth_username"`
+ AuthPassword string `json:"-"`
+ AuthPasswordEncrypted string `json:"auth_password_encrypted,omitempty"`
+ AuthToken string `json:"-"`
+ AuthTokenEncrypted string `json:"auth_token_encrypted,omitempty"`
+ UseAuthorizationHeader bool `json:"use_authorization_header"`
+ AuthorizationHeader string `json:"-"`
// required: true
UID int `json:"uid" binding:"Required"`
// required: true
diff --git a/modules/repository/repo.go b/modules/repository/repo.go
index de6de3bda..1e69acb67 100644
--- a/modules/repository/repo.go
+++ b/modules/repository/repo.go
@@ -74,10 +74,11 @@ func MigrateRepositoryGitData(ctx context.Context, u *user_model.User,
}
if err = git.Clone(ctx, opts.CloneAddr, repoPath, git.CloneRepoOptions{
- Mirror: true,
- Quiet: true,
- Timeout: migrateTimeout,
- SkipTLSVerify: setting.Migrations.SkipTLSVerify,
+ Mirror: true,
+ Quiet: true,
+ Timeout: migrateTimeout,
+ SkipTLSVerify: setting.Migrations.SkipTLSVerify,
+ AuthorizationHeader: opts.AuthorizationHeader,
}); err != nil {
return repo, fmt.Errorf("Clone: %w", err)
}
@@ -95,11 +96,12 @@ func MigrateRepositoryGitData(ctx context.Context, u *user_model.User,
}
if err := git.Clone(ctx, wikiRemotePath, wikiPath, git.CloneRepoOptions{
- Mirror: true,
- Quiet: true,
- Timeout: migrateTimeout,
- Branch: "master",
- SkipTLSVerify: setting.Migrations.SkipTLSVerify,
+ Mirror: true,
+ Quiet: true,
+ Timeout: migrateTimeout,
+ Branch: "master",
+ SkipTLSVerify: setting.Migrations.SkipTLSVerify,
+ AuthorizationHeader: opts.AuthorizationHeader,
}); err != nil {
log.Warn("Clone wiki: %v", err)
if err := util.RemoveAll(wikiPath); err != nil {
diff --git a/services/migrations/gitea_uploader.go b/services/migrations/gitea_uploader.go
index 8a7533b3d..5de25dfb6 100644
--- a/services/migrations/gitea_uploader.go
+++ b/services/migrations/gitea_uploader.go
@@ -7,6 +7,7 @@ package migrations
import (
"context"
+ "encoding/base64"
"fmt"
"io"
"os"
@@ -118,19 +119,24 @@ func (g *GiteaLocalUploader) CreateRepo(repo *base.Repository, opts base.Migrate
r.DefaultBranch = repo.DefaultBranch
r.Description = repo.Description
+ if opts.UseAuthorizationHeader {
+ opts.AuthorizationHeader = fmt.Sprintf("Authorization: Basic %s", base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s", opts.AuthUsername, opts.AuthPassword))))
+ }
+
r, err = repo_module.MigrateRepositoryGitData(g.ctx, owner, r, base.MigrateOptions{
- RepoName: g.repoName,
- Description: repo.Description,
- OriginalURL: repo.OriginalURL,
- GitServiceType: opts.GitServiceType,
- Mirror: repo.IsMirror,
- LFS: opts.LFS,
- LFSEndpoint: opts.LFSEndpoint,
- CloneAddr: repo.CloneURL, // SECURITY: we will assume that this has already been checked
- Private: repo.IsPrivate,
- Wiki: opts.Wiki,
- Releases: opts.Releases, // if didn't get releases, then sync them from tags
- MirrorInterval: opts.MirrorInterval,
+ RepoName: g.repoName,
+ Description: repo.Description,
+ OriginalURL: repo.OriginalURL,
+ GitServiceType: opts.GitServiceType,
+ Mirror: repo.IsMirror,
+ LFS: opts.LFS,
+ LFSEndpoint: opts.LFSEndpoint,
+ CloneAddr: repo.CloneURL, // SECURITY: we will assume that this has already been checked
+ Private: repo.IsPrivate,
+ Wiki: opts.Wiki,
+ Releases: opts.Releases, // if didn't get releases, then sync them from tags
+ MirrorInterval: opts.MirrorInterval,
+ AuthorizationHeader: opts.AuthorizationHeader,
}, NewMigrationHTTPTransport())
g.sameApp = strings.HasPrefix(repo.OriginalURL, setting.AppURL) |
|
Nice find @KN4CK3R - but what I don't understand is why other implementations do not require this - surely they all use username/password authentication. Other repositories probably just relies on the git binary doing the authentication for them. All basic authentication works like that. Server sends a challenge, client sends a "Authorization" header. And I am sorry you feel that its not worth it to fix this just because its "only" azure devops on prem that needs this - I can see other settings the the "MigrateOptions" - that are specific other other types, i.e. Wiki. Also I think it can also be a bug in the git client possibly, it should be able to handle the authentication, but possibly the azure devops server requests "NTLM" authenticaion and then git just says "nope" - and on windows perhaps there is a special handling of that that translates that into "BASIC" instead. It would be nice to see the actual headers the azure devops server sends to see if its possible to get a hint to why git cannot just provide basic authentication header automatically, as I am sure that is what it does to all other http/https git urls. |
|
Maybe you can find your answer in this thread: https://stackoverflow.com/questions/64800010/why-does-http-basic-auth-works-using-git-c-http-extraheader-authorization-ba |
|
My guess is that at some point urls with username/password in the url will stop working - and in my opinion it should be already. for authorization and never put it into the url, which also have the added benefit that no username/password combos are stored in git config files. |
The extra headers would be stored in the config file too. Maybe this will change in future when the secret store is added. |
Hi,
Any plans to support migration from on prem Azure DevOps (git) to gitea?
I have tried and it just fails - possibly because authentication in azure devops is not using basic authentication but something else?
It just logs:
I am 100% certain that the username/password combo is correct, since I use it on a daily basis - and also can access the repo from another service - using the exact same combo.
I also have this in my app.ini
My guess is that it is probably because I have a "special" character in my password and when I turn on debug logging I can see the url it tries to clone - and if I copy/paste that URL and do a manual git clone from the prompt - I also get the same:
If I do a
And manually type in the password - it just works.
I have no idea if its some kind of escaping that happens that messes up the http request inside git - or if its simply wrongly escaped from gitea's side.
Do I just have to accept this?
According to "people" on the internet, then it seems to have been escaped correctly according to https://en.wikipedia.org/wiki/Percent-encoding - but no dice - I have used other systems in the past that did not like my password.
To give a hint about what character that messes it up - its
#- which I know is bad in a url - but I refuse to change my password just because some systems do not like it.wget on the other hand just accepts the encoded url - so I am beginning to think its a bug in git.
Edit:
Just tested on a windows machine and git clone just works with the encoded url (git version 2.33) - on the gitea server (Rocky Linux) - git version 2.27.0 it fails - so either its a bug that is fixed in 2.33 - or an OS issue that is causing this.
Edit2:
Updated git to version 2.31 on the server - and its the same issue. So I am thinking its a git on linux issue.
Originally posted by @bjornbouetsmith in #8689 (comment)
The text was updated successfully, but these errors were encountered: