Reduce repo indexer disk usage #3452
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lafriks
reviewed
Feb 3, 2018
modules/indexer/issue.go
Outdated
| @@ -70,9 +73,15 @@ func createIssueIndexer() error { | |||
| mapping := bleve.NewIndexMapping() | |||
| docMapping := bleve.NewDocumentMapping() | |||
|
|
|||
| numericFieldMapping := bleve.NewNumericFieldMapping() | |||
| numericFieldMapping.Store = false | |||
| numericFieldMapping.IncludeInAll = false | |||
| docMapping.AddFieldMappingsAt("RepoID", bleve.NewNumericFieldMapping()) | |||
There was a problem hiding this comment.
Should this use numericFieldMapping?
tboerger
added
the
lgtm/need 2
ethantkoenig
force-pushed
the
repo_indexer_disk_usage
branch
3 times, most recently
from
43870d9
to
c90f7af
Compare
Codecov Report
@@ Coverage Diff @@
## master #3452 +/- ##
==========================================
- Coverage 35.67% 35.67% -0.01%
==========================================
Files 281 281
Lines 40697 40671 -26
==========================================
- Hits 14519 14508 -11
+ Misses 24031 24020 -11
+ Partials 2147 2143 -4
Continue to review full report at Codecov.
|
lafriks
approved these changes
Feb 4, 2018
tboerger
added
lgtm/need 1
appleboy
suggested changes
Feb 4, 2018
| "strconv" | ||
|
|
||
| "code.gitea.io/gitea/modules/setting" |
|
@appleboy Done |
appleboy
approved these changes
Feb 5, 2018
tboerger
added
lgtm/done
|
@ethantkoenig please resolve conflicts |
ethantkoenig
force-pushed
the
repo_indexer_disk_usage
branch
from
b2d1420
to
55a3db8
Compare
|
@lafriks Resolved |
aswild
added a commit
to aswild/gitea
that referenced
this pull request
Jul 6, 2018
* SECURITY * Limit uploaded avatar image-size to 4096x3072 by default (go-gitea#4353) * Do not allow to reuse TOTP passcode (go-gitea#3878) * FEATURE * Add cli commands to regen hooks & keys (go-gitea#3979) * Add support for FIDO U2F (go-gitea#3971) * Added user language setting (go-gitea#3875) * LDAP Public SSH Keys synchronization (go-gitea#1844) * Add topic support (go-gitea#3711) * Multiple assignees (go-gitea#3705) * Add protected branch whitelists for merging (go-gitea#3689) * Global code search support (go-gitea#3664) * Add label descriptions (go-gitea#3662) * Add issue search via API (go-gitea#3612) * Add repository setting to enable/disable health checks (go-gitea#3607) * Emoji Autocomplete (go-gitea#3433) * Implements generator cli for secrets (go-gitea#3531) * ENHANCEMENT * Add more webhooks support and refactor webhook templates directory (go-gitea#3929) * Add new option to allow only OAuth2/OpenID user registration (go-gitea#3910) * Add option to use paged LDAP search when synchronizing users (go-gitea#3895) * Symlink icons (go-gitea#1416) * Improve release page UI (go-gitea#3693) * Add admin dashboard option to run health checks (go-gitea#3606) * Add branch link in branch list (go-gitea#3576) * Reduce sql query times in retrieveFeeds (go-gitea#3547) * Option to enable or disable swagger endpoints (go-gitea#3502) * Add missing licenses (go-gitea#3497) * Reduce repo indexer disk usage (go-gitea#3452) * Enable caching on assets and avatars (go-gitea#3376) * Add repository search ordered by stars/forks. Forks column in admin repo list (go-gitea#3969) * Add Environment Variables to Docker template (go-gitea#4012) * LFS: make HTTP auth period configurable (go-gitea#4035) * Add config path as an optionial flag when changing pass via CLI (go-gitea#4184) * Refactor User Settings sections (go-gitea#3900) * Allow square brackets in external issue patterns (go-gitea#3408) * Add Attachment API (go-gitea#3478) * Add EnableTimetracking option to app settings (go-gitea#3719) * Add config option to enable or disable log executed SQL (go-gitea#3726) * Shows total tracked time in issue and milestone list (go-gitea#3341) * TRANSLATION * Improve English grammar and consistency (go-gitea#3614) * DEPLOYMENT * Allow Gitea to run as different USER in Docker (go-gitea#3961) * Provide compressed release binaries (go-gitea#3991) * Sign release binaries (go-gitea#4188)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reduces disk usage of the repo (i.e. code) indexer:
_allfield (which meant that we were previously storing everything twice)I saw as roughly 3x (1.5GB -> 500MB) reduction in disk usage as a result of these changes (of course, mileage will vary depending on what type of text/code you are indexing).
Also introduces a migration-like versions to the issue and repo indexers to facilitate changes (which will typically require rebuilding the index).
Yes, this PR shamelessly pulls in https://github.com/ethantkoenig/rupture as a dependency to facilitate tracking indexer versions and migrations; I am aware of no other alternatives.