cmd/serve: pprof cpu and memory profile dumps to disk #4560

xor-gate · 2018-07-30T17:39:06Z

Tracing memory usage with pprof dumps to disk for serve command which can help to find memory problems with issue #4450

lafriks · 2018-07-30T17:54:40Z

cmd/serv.go

@@ -143,6 +158,10 @@ func runServ(c *cli.Context) error {
 	username := strings.ToLower(rr[0])
 	reponame := strings.ToLower(strings.TrimSuffix(rr[1], ".git"))

+	if setting.EnablePprof {


I think it should be also allowed to specify from cli so that it would be possible to profile only single key by editing authorized_keys file

I agree, how would that look like then ?

Now I have:

command="/Users/jerry/src/code.gitea.io/gitea/gitea serv key-1 --config='/Users/jerry/src/code.gitea.io/gitea/custom/conf/app.ini'"

Then it would become:

command="/Users/jerry/src/code.gitea.io/gitea/gitea serv key-1 --config='/Users/jerry/src/code.gitea.io/gitea/custom/conf/app.ini'",environment="ENABLE_PPROF=true"

Which must work out of the box with OpenSSH right?

why not just:

command="/Users/jerry/src/code.gitea.io/gitea/gitea serv key-1 --enable-pprof --config='/Users/jerry/src/code.gitea.io/gitea/custom/conf/app.ini'"

Now I understand, latest commit has it added.

codecov-io · 2018-07-30T17:58:41Z

Codecov Report

Merging #4560 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4560   +/-   ##
=======================================
  Coverage   20.64%   20.64%           
=======================================
  Files         166      166           
  Lines       32247    32247           
=======================================
  Hits         6657     6657           
  Misses      24612    24612           
  Partials      978      978

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed3589f...0ab3e63. Read the comment docs.

xor-gate · 2018-07-30T18:36:36Z

I'm looking at my initial issue #4450, I see also very high CPU usage. Should I extend this PR also with cpu profile dump?

techknowlogick · 2018-07-30T19:46:56Z

custom/conf/app.ini.sample

@@ -185,6 +185,10 @@ STATIC_ROOT_PATH =
 APP_DATA_PATH = data
 ; Application level GZIP support
 ENABLE_GZIP = false
+; Application profiling (memory and cpu)
+; For "web" command it listens on localhost:6060
+; For "serve" command it dumps to disk at $TMPDIR/gitea_serve_pprof_<username>_id


Is it possible to define this path as a setting? If not that's fine.

Thats a nice addition, I will name it PPROF_FILE_PATH. Good? Should we only allow absolute paths or also expand e.g $TMPDIR ? Absolute paths are much easier to implement.

it should be treated same as other paths in Gitea

…its own module for reuse and clarity

xor-gate · 2018-08-01T10:14:19Z

Hi guys, it took some time but I have processed the review comments:

Added PPROF_DATA_PATH
Break pprof code into its own module

Hope this is fine now, feel free to comment.

lafriks · 2018-08-02T14:35:40Z

modules/setting/setting.go

@@ -773,6 +774,7 @@ func NewContext() {
 	AppDataPath = sec.Key("APP_DATA_PATH").MustString(path.Join(AppWorkPath, "data"))
 	EnableGzip = sec.Key("ENABLE_GZIP").MustBool()
 	EnablePprof = sec.Key("ENABLE_PPROF").MustBool(false)
+	PprofDataPath = sec.Key("PPROF_DATA_PATH").MustString(path.Join(AppWorkPath, "data/tmp/pprof"))


After this line you also need:

if !filepath.IsAbs(PprofDataPath) { PprofDataPath = filepath.Join(AppWorkPath, PprofDataPath) }

I agree, will fix that.

lafriks · 2018-08-02T14:36:37Z

modules/setting/setting.go

@@ -916,6 +918,12 @@ func NewContext() {
 		}
 	}

+	if PprofDataPath != "" {


this should be: if EnablePprof {

I think that will not work because we always have to create it. As there are two types of options:

Global ENABLE_PPROF and/or

Per key --enable-pprof from serve (then ENABLE_PPROF is not set the path does not exist in your case)

Than just add code to create it in second case?

I have decided to create the directories in one place (currently in cmd/serve.go). Felt a little weird mkdir in setting and in serv. Check my last commit.

xor-gate · 2018-08-07T10:02:40Z

@techknowlogick your review comment has been proccesed. Could you check if all changes are fine now?

sapk

Good PR.
The next step could be to add it to all command and put command name in dump name. This could be usefull to improve when gitea is called by ssh server.

techknowlogick · 2018-08-07T18:34:01Z

Thanks @xor-gate. My review was non-blocking and so now that @sapk has given a LG-TM, this can be merged.

I'll merge this likely later today, I'm just merging a few things in first.

xor-gate · 2018-10-15T08:51:38Z

I there any documentation edit necessary from my side before 1.6.0 is released?

lafriks · 2018-10-15T19:25:10Z

@xor-gate feel free to add section about pprof if you have time but it can be added any time, docs are built always from master so not directly tied to releases

Prepare for wild/v1.6 branch * BREAKING * Respect email privacy option in user search via API (go-gitea#4512) * Simply remove tidb and deps (go-gitea#3993) * Swagger.v1.json template (go-gitea#3572) * FEATURE * Pull request review/approval and comment on code (go-gitea#3748) * Added dependencies for issues (go-gitea#2196) (go-gitea#2531) * Add the ability to have built in themes in Gitea and provide dark theme arc-green (go-gitea#4198) * Add sudo functionality to the API (go-gitea#4809) * Add oauth providers via cli (go-gitea#4591) * Disable merging a WIP Pull request (go-gitea#4529) * Force user to change password (go-gitea#4489) * Add letsencrypt to Gitea (go-gitea#4189) * Add push webhook support for mirrored repositories (go-gitea#4127) * Add csv file render support defaultly (go-gitea#4105) * Add Recaptcha functionality to Gitea (go-gitea#4044) * BUGFIXES * Fix release creation via API (go-gitea#5076) * Remove links from topics in edit mode (go-gitea#5026) * Fix missing AppSubUrl in few more templates (fixup) (go-gitea#5021) * Fix missing AppSubUrl in some templates (go-gitea#5020) * Hide outdated comments in file view (go-gitea#5017) * Upgrade gopkg.in/testfixtures.v2 (go-gitea#4999) * Disable debug routes unless PPROF is enabled in configuration (go-gitea#4995) * Fix user menu item styling (go-gitea#4985) * Fix layout of the topics editing form (go-gitea#4971) * Fix null pointer dereference in ParseCommitWithSignature (go-gitea#4962) * Fix url in discord webhook (go-gitea#4953) * Detect charset and convert non UTF-8 files for display (go-gitea#4950) * Make sure to catch the right error so it is displayed on the UI (go-gitea#4945) * Fix(topics): don't redirect to explore page. (go-gitea#4938) * Fix bug forget to remove Stopwatch when remove repository (go-gitea#4928) * Fix bug when repo remained bare if multiple branches pushed in single push (go-gitea#4923) * Fix: Let's Encrypt configuration settings (go-gitea#4911) * Fix: Crippled diff (go-gitea#4726) (go-gitea#4900) * Fix trimming of markup section names (go-gitea#4863) * Issues api allow pulls and fix go-gitea#4832 (go-gitea#4852) * Do not autocreate directory for new users/orgs (go-gitea#4828) (go-gitea#4849) * Fix redirect with non-ascii branch names (go-gitea#4764) (go-gitea#4810) * Fix missing release title in webhook (go-gitea#4783) (go-gitea#4796) * User shouldn't be able to approve or reject his/her own PR (go-gitea#4729) * Make sure to reset commit count in the cache on mirror syncing (go-gitea#4720) * Fixed bug where team with admin privelege type doesn't get any unit (go-gitea#4719) * Fix incorrect caption of webhook setting (go-gitea#4701) (go-gitea#4717) * Allow WIP marker to contains < or > (go-gitea#4709) * Hide org/create menu item in Dashboard if user has no rights (go-gitea#4678) (go-gitea#4680) * Site admin could create repos even MAX_CREATION_LIMIT=0 (go-gitea#4645) * Fix custom templates being ignored (go-gitea#4638) * Fix starring icon after semantic ui update (go-gitea#4628) * Fix Split-View line adjustment (go-gitea#4622) * Fix integer constant overflows in tests (go-gitea#4616) * Push whitelist now doesn't apply to branch deletion (go-gitea#4601) (go-gitea#4607) * Fix bugs when too many IN variables (go-gitea#4594) * Fix failure on creating pull request with assignees (go-gitea#4419) (go-gitea#4583) * Fix panic issue on update avatar email (go-gitea#4580) (go-gitea#4581) * Fix status code label for a successful webhook (go-gitea#4540) * An inactive user shouldn't be able to be added as a collaborator (go-gitea#4535) * Don't fail silently if trying to add a collaborator twice (go-gitea#4533) * Fix incorrect MergeWhitelistTeamIDs check in CanUserMerge function (go-gitea#4519) (go-gitea#4525) * Fix out-of-transaction query in removeOrgUser (go-gitea#4521) (go-gitea#4522) * Fix migration from older releases (go-gitea#4495) * Accept 'Data:' in commit graph (go-gitea#4487) * Update xorm to latest version and fix correct `user` table referencing in sql (go-gitea#4473) * Relative URLs for LibreJS page (go-gitea#4460) * Redirect to correct page after using scratch token (go-gitea#4458) * Fix column droping for MSSQL that need new transaction for that (go-gitea#4440) * Replace src with raw to fix image paths (go-gitea#4377) * Add default merge options when creating new repository (go-gitea#4369) * Fix docker build (go-gitea#4358) * Fixes repo membership check in API (go-gitea#4341) * Dep upgrade mysql lib (go-gitea#4161) * Fix some issues with special chars in branch names (go-gitea#3767) * Responsive design fixes (go-gitea#4508) * ENHANCEMENT * Fix milestones sorted wrongly (go-gitea#4987) * Allow api to create tags for releases if they don't exist (go-gitea#4890) * Fix go-gitea#4877 to follow the OpenID Connect Audiences spec (go-gitea#4878) * Enforce token on api routes [fixed critical security issue go-gitea#4357] (go-gitea#4840) * Update legacy branch and tag URLs in dashboard to new format (go-gitea#4812) * Slack webhook channel name cannot be empty or just contain an hashtag (go-gitea#4786) * Add whitespace handling to PR-comparsion (go-gitea#4683) * Make reverse proxy auth optional (go-gitea#4643) * MySQL TLS (go-gitea#4642) * Make sure to set PR split view when creating/previewing a pull request (go-gitea#4617) * Log user in after a successful sign up (go-gitea#4615) * Fix typo IsPullReuqestBroken -> IsPullRequestBroken (go-gitea#4578) * Allow admin toggle forcing a password change for newly created users (go-gitea#4563) * Update jQuery to v1.12.4 (go-gitea#4551) * Env var GITEA_PUSHER_EMAIL (go-gitea#4516) * Feat(repo): support search repository by topic name (go-gitea#4505) * Small improvements to dependency UI (go-gitea#4503) * Make max commits in graph configurable (go-gitea#4498) * Add valid for lfs oid (go-gitea#4461) * Add shortcut to save wiki page (go-gitea#4452) * Allow administrator to create repository for any organization (go-gitea#4368) * Fix repository last updated time update when delete a user who watched the repo (go-gitea#4363) * Switch plaintext scratch tokens to use hash instead (go-gitea#4331) * Increase default TOTP secret size to 320 bits (go-gitea#4287) * Keep preseeded database password (go-gitea#4284) * Implemented hover text showing user FullName (go-gitea#4261) * Add ability to delete a token (go-gitea#4235) * Fix typos in i18n variable names. (go-gitea#4080) * Api: repos/search: add parameters to control the sort order (go-gitea#3964) * Add missing path in the Docker app.ini template (go-gitea#2181) * Add file name and branch to page title (go-gitea#4902) * Offline use of google fonts (go-gitea#4872) * Add missing History link to directory listings v2 (go-gitea#4829) * Locale for Edit and Remove due date issue (go-gitea#4802) * Disable 'May Import Local Repository' when is disabled by setting (Is… (go-gitea#4780) * API /admin/users/{username} missing parameter (go-gitea#4775) * Display error when adding a user to a team twice (go-gitea#4746) * Remove UsePrivilegeSeparation from the Docker sshd_config, see go-gitea#2876 (go-gitea#4722) * Focus title input when clicking helper link (go-gitea#4696) * Add vendor to user reserved words and format words list according alphabet (go-gitea#4685) * Add gitea/issues link to 500 page (go-gitea#4654) * Hide home button when landing page is not set to home (go-gitea#4651) * Remove link to GitHub issues in 404 template (go-gitea#4639) * Cmd/serve: pprof cpu and memory profile dumps to disk (go-gitea#4560) * Add flash message after an account has been successfully activated (go-gitea#4510) * Prevent html entity escaping on delete branch (go-gitea#4471) * Locale for button Edit on protected branch (go-gitea#4442) * Update notification icon (go-gitea#4343) * Added front-end topics validation (go-gitea#4316) * Don't display buttons if there are no system notifications (go-gitea#4280) * Issue due date api (go-gitea#3890) * SECURITY * Improve URL validation for external wiki and external issues (go-gitea#4710) * Make cookies HttpOnly and obey COOKIE_SECURE flag (go-gitea#4706) * Don't disclose emails of all users when sending out emails (go-gitea#4664) * Check that repositories can only be migrated to own user or organizations (go-gitea#4366) * TRANSLATION * Fix punctuation in English translation (go-gitea#4958) * Fix translation (go-gitea#4355)

xor-gate mentioned this pull request Jul 30, 2018

Serv command consumes all server resources #4450

Closed

2 tasks

xor-gate changed the title ~~WIP cmd/serve: pprof memory profile dumps to disk~~ cmd/serve: pprof memory profile dumps to disk Jul 30, 2018

cmd/serve: pprof memory profile dumps to disk (usefull for #4450)

96ec67f

lafriks reviewed Jul 30, 2018

View reviewed changes

bkcsoft added lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Jul 30, 2018

cmd/serve: Allow pprof per-key

4808ca3

techknowlogick reviewed Jul 30, 2018

View reviewed changes

lafriks approved these changes Jul 30, 2018

View reviewed changes

bkcsoft added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Jul 30, 2018

cmd/serve: Add setting for PPROF_DATA_PATH and break pprof code into …

8893394

…its own module for reuse and clarity

xor-gate changed the title ~~cmd/serve: pprof memory profile dumps to disk~~ cmd/serve: pprof cpu and memory profile dumps to disk Aug 1, 2018

lafriks requested changes Aug 2, 2018

View reviewed changes

xor-gate and others added 4 commits August 5, 2018 23:14

Process last review comments

3c19346

Merge branch 'master' into issue-4450

0eaa416

cmd/serv: Fix typo

da76139

Merge branch 'issue-4450' of github.com:xor-gate/gitea into issue-4450

0ce9b27

lafriks approved these changes Aug 6, 2018

View reviewed changes

sapk approved these changes Aug 7, 2018

View reviewed changes

bkcsoft added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Aug 7, 2018

Merge branch 'master' into issue-4450

5159a19

Merge branch 'master' into issue-4450

0ab3e63

techknowlogick merged commit b1bc08e into go-gitea:master Aug 7, 2018

lafriks added the type/miscellaneous label Aug 7, 2018

lafriks added this to the 1.6.0 milestone Aug 7, 2018

lafriks added the type/changelog Adds the changelog for a new Gitea version label Nov 10, 2018

sapk mentioned this pull request Sep 5, 2019

Error 502 after version upgrade #8041

Closed

5 tasks

go-gitea locked and limited conversation to collaborators Nov 24, 2020

delvh removed the type/changelog Adds the changelog for a new Gitea version label Oct 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/serve: pprof cpu and memory profile dumps to disk #4560

cmd/serve: pprof cpu and memory profile dumps to disk #4560

xor-gate commented Jul 30, 2018 •

edited

Loading

lafriks Jul 30, 2018 •

edited

Loading

xor-gate Jul 30, 2018

xor-gate Jul 30, 2018

lafriks Jul 30, 2018

xor-gate Jul 30, 2018

codecov-io commented Jul 30, 2018 •

edited

Loading

xor-gate commented Jul 30, 2018

techknowlogick Jul 30, 2018

xor-gate Jul 31, 2018

lafriks Jul 31, 2018

xor-gate commented Aug 1, 2018 •

edited

Loading

lafriks Aug 2, 2018

xor-gate Aug 2, 2018

lafriks Aug 2, 2018

xor-gate Aug 2, 2018 •

edited

Loading

lafriks Aug 2, 2018 •

edited

Loading

xor-gate Aug 5, 2018

xor-gate commented Aug 7, 2018

sapk left a comment

techknowlogick commented Aug 7, 2018

xor-gate commented Oct 15, 2018

lafriks commented Oct 15, 2018

cmd/serve: pprof cpu and memory profile dumps to disk #4560

cmd/serve: pprof cpu and memory profile dumps to disk #4560

Conversation

xor-gate commented Jul 30, 2018 • edited Loading

lafriks Jul 30, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Jul 30, 2018 • edited Loading

Codecov Report

xor-gate commented Jul 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xor-gate commented Aug 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xor-gate Aug 2, 2018 • edited Loading

Choose a reason for hiding this comment

lafriks Aug 2, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xor-gate commented Aug 7, 2018

sapk left a comment

Choose a reason for hiding this comment

techknowlogick commented Aug 7, 2018

xor-gate commented Oct 15, 2018

lafriks commented Oct 15, 2018

xor-gate commented Jul 30, 2018 •

edited

Loading

lafriks Jul 30, 2018 •

edited

Loading

codecov-io commented Jul 30, 2018 •

edited

Loading

xor-gate commented Aug 1, 2018 •

edited

Loading

xor-gate Aug 2, 2018 •

edited

Loading

lafriks Aug 2, 2018 •

edited

Loading