Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[qm-elk] Add support for QM data enrichment #926

Closed
wants to merge 10 commits into from

Conversation

vchrombie
Copy link
Member

@vchrombie vchrombie commented Jul 30, 2020

This PR contains the following enrichers

  • gitlabqm
  • gitqm
  • pipermailqm
  • meetupqm
  • githubqm

metrics discussion: metric labelled issues
more information: vchrombie/gsoc#6

micro-mordred configurations

project.json

{
  "potioncraft": {
    "gitqm": [
      "https://gitlab.com/Bitergia/devops/indexwarrior"
    ] ,
    "gitlabqm:issue": [
      "https://gitlab.com/Bitergia/devops%2Findexwarrior"
    ] ,
    "gitlabqm:merge": [
      "https://gitlab.com/Bitergia/devops%2Findexwarrior"
    ] ,
    "pipermailqm": [
      "https://lists.linuxfoundation.org/pipermail/grimoirelab-discussions/"
    ] ,
    "meetupqm": [
      "GitLab-Meetup-Group"
    ] ,
    "githubqm:issue": [
      "https://github.com/chaoss/grimoirelab"
    ] ,
    "githubqm:pull": [
      "https://github.com/chaoss/grimoirelab"
    ]
  }
}

setup.cfg

[gitqm]
raw_index = gitqm_raw
enriched_index = gitqm_enriched
category = commit

[gitlabqm:issue]
api-token = xxxx
raw_index = gitlabqm_issues_raw
enriched_index = gitlabqm_issues_enriched
sleep-for-rate = true
category = issue
no-archive = true

[gitlabqm:merge]
api-token = xxxx
raw_index = gitlabqm_mrs_raw
enriched_index = gitlabqm_mrs_enriched
sleep-for-rate = true
category = merge_request
no-archive = true

[pipermailqm]
raw_index = pipermailqm_raw
enriched_index = pipermailqm_enriched

[meetupqm]
raw_index = meetupqm_raw
enriched_index = meetupqm_enriched
api-token = xxxx
sleep-for-rate = true
sleep-time = "300"
no-archive = true

[githubqm:issue]
raw_index = githubqm_issues_raw
enriched_index = githubqm_issues_enriched
api-token = xxxx
sleep-for-rate = true
sleep-time = 300
category = issue
no-archive = true

[githubqm:pull]
raw_index = githubqm_prs_raw
enriched_index = githubqm_prs_enriched
api-token = xxxx
sleep-for-rate = true
sleep-time = 300
category = pull_request
no-archive = true

@vchrombie
Copy link
Member Author

Hi everyone,
I have updated the PR with the suggested changes and also the latest enricher work.

I will add the tests, schema and also format the project structure to similar of the SCMS structure tomorrow.

@coveralls
Copy link

coveralls commented Aug 2, 2020

Pull Request Test Coverage Report for Build 2375

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 47 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-3.9%) to 78.272%

Files with Coverage Reduction New Missed Lines %
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/github.py 2 75.06%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/utils.py 45 66.42%
Totals Coverage Status
Change from base Build 2373: -3.9%
Covered Lines: 8451
Relevant Lines: 10797

💛 - Coveralls

This commit adds the gitlabqm enricher. The base file, qmenrich.py
is added. The connectors for the enricher are also updated.

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
This commit adds the gitlabqm enricher. The connectors
for the enricher are also updated.

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
This commit adds the support for four additional metrics
- number of issue comments
- number of merge request comments
- number of issues attended
- number of merge requests attended

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
This commit is in reference to the issues raised
- fix the date normalizing errors and also problems
related to the timezone
- made the '.git' string optional
- fix the value error

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
This commit adds the pipermailqm enricher. The connectors
of the enricher are also updated.

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
This commit adds the meetupqm enricher. The connectors
of the enricher are also updated.

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
This commit moves all the qm enrichers to a dedicated
module to improve maintainability.

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
- Fix a few implementation flaws in all the enrichers
- Fix the typo for comments in meetup

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
This commit adds the githubqm enricher. The connectors
for the enricher are also updated.

Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>
@vchrombie
Copy link
Member Author

The work will be continued at glab-prosoul/grimoirelab-elk.

@vchrombie vchrombie closed this Sep 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants