Migrate from SSH

[RickZeeland]

• Gitea version (or commit ref): gitea-1.1-windows-4.0-amd64.exe
• Git version:
• Operating system: Windows 10 Enterprise
• Database (use [x]):
  • [ x] PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • [x ] Not relevant
• Log gist:

Description

We have a GIT server which only supports SSH for cloning, not HTTP, example:

ssh://[email protected]/source/test.git

How can I migrate a repository in Gitea ?
Or is there some other way I can do this manually ?

[sapk]

You have multiple choice :

• If you only few repo (or some time) you could use migration interface (ex: https://try.gitea.io/repo/migrate). (Note: I think it doesn't support key auth)
• If you have access to server hosting gitea you could clone repo somewhere on it and import local path.

There must also be others methods to do it but they should be the easy one.

[RickZeeland]

@sapk : Thanks for your tips, but I already tried the Migrate form in Gitea, it does not accept SSH url's.
I have been struggling the whole day trying to find a way to clone and merge the repository, but to no avail. In theory it seems to be possible to merge from another remote repository, but I can not get it working. Copying the files and checking them in is not really an option, as this would mean loosing the tags history.

[sapk]

Have you try replacing ssh:// url part with git:// ?

[deanpcmad]

@RickZeeland how many repos do you want to transfer over? What you could do is just clone them and push them to your Gitea server

git clone [email protected]:abc/123.git
cd 123
git remote set-url origin [email protected]:org/repo.git
git push origin master
git push --tags

The git push --tags command will push all your tags

[RickZeeland]

@sapk : Thanks, I will try that out first next week when I'm at work again.

@deanpcmad : Looks good, It's only for one big repo. I tried something like you suggested locally with drive paths but got an error when pushing, had to go through some hoops to get it working:
git config --global receive.denyCurrentBranch ignore
Git gui - Repo1 - push to Repo2
Git gui - Repo2 - visualize all branch history - r-click on revision -
reset master branch to here - Reset type: hard

[RickZeeland]

@sapk : Nope, git: does not work.
@deanpcmad : It works like a charm, as long as you have a bare Gitea repository, see attached picture. Thanks !

[RickZeeland]

@hetykai I have not read anywhere that this has been fixed, so I'm afraid you will have to do it from Bash with GIT commands like shown above.
Example as shown by @deanpcmad :

git clone [email protected]:abc/123.git cd 123 git remote set-url origin [email protected]:org/repo.git git push origin master git push --tags

[RickZeeland]

@hetykai it could also have to do with the SSH configuration which is typically found in: C:\Projects\Gitea\custom\conf\app.ini

How to config SSH settings
https://discuss.gogs.io/t/how-to-config-ssh-settings/34
https://docs.gitea.io/en-us/config-cheat-sheet/

	DISABLE_SSH: Disables SSH feature when it’s not available.
	START_SSH_SERVER: Starts built-in SSH server when enabled.
	SSH_DOMAIN: Domain name of your ssh server.
	SSH_PORT: SSH port displayed in clone URL, in case yours is not 22.
	SSH_LISTEN_PORT: Port for the built-in SSH server. Defaults to SSH_PORT.

https://github.com/go-gitea/gitea/issues/2723

[pgodwin]

Any updates on this one? I've configured the SSH key on my user profile, but it doesn't seem to work properly with a git url. ie git://bitbucket.org:user/repo.git
I get the following error:

Migration failed: Clone: exit status 128 - fatal: unable to connect to bitbucket.org: bitbucket.org[0: 104.192.143.2]: errno=Invalid argument bitbucket.org[1: 104.192.143.3]: errno=Invalid argument bitbucket.org[2: 104.192.143.1]: errno=Invalid argument

[techknowlogick]

@pgodwin as of now HTTP(S) is still the only way to mirror/migrate git repos from other systems

[BNolet]

I'm confused, is this a conversation of the git:// URL prefix or not? If not, let me know so I can start an issue. Currently getting unable to connect and connection timed out

[RickZeeland]

@BNolet No this is not a conversation of the git:// URL prefix. Don't think anything has changed and it's still not possible to migrate from SSH in Gitea, the documentation at https://docs.gitea.io/en-us/ only mentions: "Clone with SSH/HTTP/HTTPS".

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[BNolet]

Still an issue

[ShoelaceMan]

Hey,

I'm seeing this as well, I think that it mostly has to do with being unable to specify a key to clone with, as if you import from an https:// link initially, and then edit the upstream url afterwards, it allows you to change it to an ssh:// link, but it gets permissions denied.

[6543]

what is the current state?

[lafriks]

Noone is currently working on this

[kyozhou]

it also confuse me

[halsafar]

This is blocking evaluating Gitea for work. We cannot import our repo which is SSH+Keys only. The fact file:/// is also blocked seems silly to me. Basically we have no path to migrate.

[TheoLassonder]

We'd really like to be able to do this also. We have a bunch of repos where the primaries are on various servers, and we'd like a mirror in Gitea.

I'm not sure why this is blocked by #12065 because it should work just fine if the user running Gitea can SSH to the remote servers where the repos are mirrored from (using an SSH key without password - no configuration required in Gitea at all).

[TheoLassonder]

Was just curious, and just had a go at hacking Gitea to support mirroring using password-less SSH. And it worked! The trouble is, I don't know much (or anything) about Go, so I mainly just commented stuff out that prevented the URL from being accepted until Gitea just went ahead and treated it as a plain Git migration. And that does work!

[TheoLassonder]

@zeripath It seems allowing SSH has similar security characteristics as allowing local file repos to be imported/mirrored. So it could be treated in the same way, that is, disabled by default but with a config parameter to allow it. It would help some people (including us) who now have to hack around this limitation using a local file mirror + a cron job to periodically update the local file mirror using SSH :-(

[TheoLassonder]

In fact, looking more closely at the changes needed to support it, it's down to a one line change: to accept the ssh:// as a valid URL, along with http://, https://, and git://.

Other than that, of course some text in the UI would need to be updated and some documentation as well.

Is there any interest in me creating a patch and push request for this?

[6543]

@TheoLassonder feel free to create a pull 👍

[mateusza]

What about using following workflow to allow SSH key-based authentication for repository migration/mirroring:

1. User starts new migration. Chooses "generic git" and "SSH key based authentication" instead of username and password.
2. Gitea generates temporary SSH key pair and displays the public key.
3. User manually adds the public key on the remote site. (i.e. as a read-only deploy key)
4. User provides the URL and remaining details regular way.
5. Gitea uses the private key to connect using SSH and clones the repository.
6. In case of mirroring mode - the key needs be kept in Gitea DB, otherwise it could be removed after migration is finished.

[lunny]

@mateusza That's what #12065 to do. It will store all users' secrets include private key.

[ansemjo]

Since #16437 is closed / merged with this one, I'll try to summarize my thoughts on this feature here.

I think it would make sense to split the issue of SSH migration or mirroring into two parts:

• support for the ssh:// protocol in general, with username and password authentication or system keys
• using pubkey authentication with keys stored in Gitea

---

If I understand correctly, the first part is what #12397 is about. To be fair, I had not thought about this issue:

As we have no way of checking that the user is allowed to use the keys that are available to Gitea we cannot assume that this is safe.

However, a) the further comments suggest that there are workarounds to prevent git from using the default keys – if any – in ~/.ssh/ of the Gitea user. And b) I don't think it is that much of a risk, to be honest, since placing keys into ~/.ssh/ of the Gitea user is a very conscious thing to do. If you feel it is a risk, then just don't generate any keys? Furthermore, in my selfhosted GitLab it always "just worked" to configure passwordless ssh:// mirroring, which would use the git user's available keys (as I described here). And GitLab certainly does not cater only to small users, so how much of an issue can this really be? Regardless, maybe this could then be a flag – whether to allow Gitea to use available keys or not?

Personally, I don't really see any issue that prevents supporting the ssh:// protocol with username and password / token at this point, similarly to how https:// is already supported .. Simply being able to allow Gitea to use available keys would already cover my use-case for this feature without needing to enable custom git hooks, which I see as a much greater risk. To this end, I share @TheoLassonder's sentiment above, that the protocol support should not be blocked by #12065.

---

Secondly, about the key storage. I don't think entropy is really a problem, since you really wouldn't be generating new keys very often. And unless you're on a tiny embedded platform with a readonly filesystem, even frequent key generation should not be a real problem today.

Generating (deploy) keys per repository and storing them in the database should also be reasonable. I haven't looked at the database model yet, and I havent looked at the work in #14483. But you're storing the secret token for https:// mirroring per-repository somewhere as well, after all. In this context, both tokens and per-repo private keys have very similar scopes, don't they?

Even if you don't want to deal with managing keys on-disk somewhere, you could maybe use Go's crypto/ssh/agent to directly load the private keys from the database into an agent and pass that to the git process to use?

---

I just stumbled upon this issue earlier today and I have only been using Gitea for a couple days now, so please excuse me if I have overlooked something. But the ssh:// protocol does appear to be a very sought-after feature and I was suprised that Gitea does not support it currently, so I wanted to summarize what I found regarding this feature so far.

[uPagge]

It's a pity that ssh is not supported yet :(
I wanted to push a mirror using ssh, but apparently this is not possible

[karolyi]

+1, this option would be nice to have.

[synsa]

+1, this as well. My company will only allow ssh:// for dealing with repos so being able to mirror push via ssh:// would be beneficial in allowing us to more widely adopt gitea over our internal git-web instance 😢

[jaketothepast]

I just hit this issue when trying to setup an SSH mirror to Github. Is there any WIP? I'd be happy to lend a hand

[ansemjo]

Since a lot of people seem to be finding this issue, I'll just copy verbatim my workaround from #16437 (comment):

---

I want to mirror a number of repositories to my GitHub account and don't want to create separate keys / tokens for each. So this is how I am currently doing it:

• login as the git/gitea user and generate a new key, e.g. with ssh-keygen -t ed25519 -C "gitea mirroring" -f ~/.ssh/id_ed25519 -N ""
• add this key to your GitHub account
• try ssh [email protected] once to accept the hostkey and make sure GitHub recognizes your key
• enable git hooks in gitea
• add the following post-receive hook to repositories:

#!/usr/bin/env bash
git push --mirror --quiet [email protected]:username/repository.git &>/dev/null &
echo "GitHub mirror initiated .."

---

Since I had quite a few of those repositories and I also wanted to have the option to skip mirroring on certain pushes, I expanded on the above and saved the following script in /usr/local/bin/github-mirror:

#!/usr/bin/env bash
# mirror a repository to github

usage() {
  echo "usage:"
  echo "  $0 username/reponame   - mirror to github"
  echo "  $0 git@hostname:...    - custom url"
  exit 1
}

mirror() {
  echo "mirror to $1 ..."
  git push --mirror --quiet "$1"
  exit $?
}

# repository argument required
if [[ -z ${1+defined} ]]; then
  echo "err: target repository required!"
  usage
fi

# check for push options
if [[ -n $GIT_PUSH_OPTION_COUNT ]]; then
  i=0
  while [[ $i -lt $GIT_PUSH_OPTION_COUNT ]]; do
    opt="GIT_PUSH_OPTION_$i"
    case "${!opt}" in
      skip|skipmirror)
        echo "Skip GitHub mirroring ..."; exit 0 ;;
      *) : ;;
    esac
    i=$((i+1))
  done
fi

# detect used scheme
# WARN: this is a very simple match ...
if [[ $1 =~ ^git@ ]]; then
  mirror "$1"
elif [[ $1 =~ ^[a-z0-9-]+/[a-zA-Z0-9_.-]+$ ]]; then
  mirror "[email protected]:$1.git"
else
  echo "err: unknown scheme"
  usage
fi

Use it in a post-receive hook like:

#!/usr/bin/env bash
github-mirror "ansemjo/$GITEA_REPO_NAME"

If your Gitea repository has the same name as the GitHub repository, you can just reuse the same snippet, because $GITEA_REPO_NAME is filled in when executing the hook.

You can skip mirroring on a push with a --push-option=skipmirror:

git push -o skipmirror

Note that I would still count this as a workaround ... but it has worked reliably for me.

[jaketothepast]

Thanks @ansemjo!

I'll gladly attempt a proper fix, or see if I can take over existing WIP to bring in this feature. Highly motivated now that I'm facing it