analysis on performance bottlenecks in repo migration

[noerw]

Repo migration from Github can take a long time (eg this gitea repo takes more than 24h on a small VPS (hetzner CPX11)).
It's unclear (to me) if this is incurred by ratelimits of external services (GH API), or due to inefficiencies in Giteas migration module.

The aim of this issue is to identify the bottlenecks involved. For now the primary method for this is to collect pprof profiles, to investigate in which routines most time is spent.
This partly also gives insight into which network, disk, DB operations take much time, but only indirectly. For that, analyzing DB query times specifically might be more helpful. (If somebody can outline a good process for that, a comment here would be appreciated ;)

I sampled some pprof activity during a migration of https://github.com/go-gitea/gitea, including all entities except for releases, gitea 1.14.0+dev-713-gec06eb112. which ran over several hours. You can find several pprof profiles that were sampled for 30 seconds - 30 minutes attached:
pprof.gitea.samples.cpu.00.zip

• To inspect them run go tool pprof -http :8080 <path to profile>
• To collect you own profiles, set ENABLE_PPROF = true under [server] in app.ini, then call go tool pprof -seconds 1800 0.0.0.0:6060

Actual analysis of these samples will follow in the upcoming days..

Server utilization graphs for the middle 12 hours of migration:

This looks to me like the higher-utilization phases each hour are related to a reset of the github ratelimiter window, so we're down to ~33% of potential performance just through github ratelimits

[lunny]

A possible resolution to break rate limit is to create multiple account and migrations support multiple tokens.

[zeripath]

quick and dirty look through those samples suggests just downloading and storing the releases is what's taking the time.

There's some delay due to encoding/json and it might be that using jsoniter here is a little quicker but I'm not certain it's huge.

[lunny]

#16070 may give some help for this issue.