[Bug][gitextractor] Error running pipeline beta5

[Capone1983]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Moved from beta4 to beta5. All bitbucket and github repos when step gitextractor starts - fails now.

attached stack trace -- stack trace: | github.com/apache/incubator-devlake/server/services.runPipeline | /app/server/services/pipeline_runner.go:79 | [...repeated from below...] Wraps: (2) Error running pipeline 261. Wraps: (3) attached stack trace -- stack trace: | github.com/apache/incubator-devlake/server/services.RunTasksStandalone | /app/server/services/task.go:217 | github.com/apache/incubator-devlake/server/services.(*pipelineRunner).runPipelineStandalone.func1 | /app/server/services/pipeline_runner.go:42 | github.com/apache/incubator-devlake/core/runner.runPipelineTasks | /app/core/runner/run_pipeline.go:90 | github.com/apache/incubator-devlake/core/runner.RunPipeline | /app/core/runner/run_pipeline.go:54 | github.com/apache/incubator-devlake/server/services.(*pipelineRunner).runPipelineStandalone | /app/server/services/pipeline_runner.go:38 | github.com/apache/incubator-devlake/server/services.runPipeline | /app/server/services/pipeline_runner.go:76 | github.com/apache/incubator-devlake/server/services.RunPipelineInQueue.func1 | /app/server/services/pipeline.go:347 | runtime.goexit | /usr/local/go/src/runtime/asm_amd64.s:1598 Wraps: (4) attached stack trace | -- stack trace: | | github.com/apache/incubator-devlake/server/services.RunTasksStandalone.func1 | | /app/server/services/task.go:189 | Wraps: (2) Error running task 26893. | Wraps: (3) attached stack trace | -- stack trace: | | github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks | | /app/core/runner/run_task.go:329 | | github.com/apache/incubator-devlake/core/runner.RunPluginTask | | /app/core/runner/run_task.go:163 | | github.com/apache/incubator-devlake/core/runner.RunTask | | /app/core/runner/run_task.go:137 | | github.com/apache/incubator-devlake/server/services.runTaskStandalone | | /app/server/services/task_runner.go:113 | | github.com/apache/incubator-devlake/server/services.RunTasksStandalone.func1 | | /app/server/services/task.go:187 | Wraps: (4) subtask Clone Git Repo ended unexpectedly | Wraps: (5) attached stack trace | -- stack trace: | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(*GitcliCloner).execGitCloneCommand | | /app/plugins/gitextractor/parser/clone_gitcli.go:144 | | [...repeated from below...] | Wraps: (6) failed to fetch all branches from the remote server | Wraps: (7) attached stack trace | -- stack trace: | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(*GitcliCloner).execCommand | | /app/plugins/gitextractor/parser/clone_gitcli.go:238 | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(*GitcliCloner).execGitCommandIn | | /app/plugins/gitextractor/parser/clone_gitcli.go:227 | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(*GitcliCloner).execGitCloneCommand | | /app/plugins/gitextractor/parser/clone_gitcli.go:143 | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(*GitcliCloner).CloneRepo | | /app/plugins/gitextractor/parser/clone_gitcli.go:92 | | github.com/apache/incubator-devlake/plugins/gitextractor/tasks.CloneGitRepo | | /app/plugins/gitextractor/tasks/repo_cloner.go:57 | | github.com/apache/incubator-devlake/core/runner.runSubtask | | /app/core/runner/run_task.go:405 | | github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks | | /app/core/runner/run_task.go:327 | | github.com/apache/incubator-devlake/core/runner.RunPluginTask | | /app/core/runner/run_task.go:163 | | github.com/apache/incubator-devlake/core/runner.RunTask | | /app/core/runner/run_task.go:137 | | github.com/apache/incubator-devlake/server/services.runTaskStandalone | | /app/server/services/task_runner.go:113 | | github.com/apache/incubator-devlake/server/services.RunTasksStandalone.func1 | | /app/server/services/task.go:187 | | runtime.goexit | | /usr/local/go/src/runtime/asm_amd64.s:1598 | Wraps: (8) git cmd [git fetch --depth=1 origin -c http.sslVerify=false] in /tmp/gitextractor2334407235 failed: error: unknown switch c' | | usage: git fetch [] [ [...]] | | or: git fetch [] | | or: git fetch --multiple [] [( | )...] | | or: git fetch --all [] | | | | -v, --verbose be more verbose | | -q, --quiet be more quiet | | --all fetch from all remotes | | --set-upstream set upstream for git pull/fetch | | -a, --append append to .git/FETCH_HEAD instead of overwriting | | --upload-pack path to upload pack on remote end | | -f, --force force overwrite of local reference | | -m, --multiple fetch from multiple remotes | | -t, --tags fetch all tags and associated objects | | -n do not fetch all tags (--no-tags) | | -j, --jobs number of submodules fetched in parallel | | -p, --prune prune remote-tracking branches no longer on remote | | -P, --prune-tags prune local tags no longer on remote and clobber changed tags | | --recurse-submodules[=] | | control recursive fetching of submodules | | --dry-run dry run | | --write-fetch-head write fetched references to the FETCH_HEAD file | | -k, --keep keep downloaded pack | | -u, --update-head-ok allow updating of HEAD ref | | --progress force progress reporting | | --depth deepen history of shallow clone | | --shallow-since | | deepen history of shallow repository based on time | | --shallow-exclude | | deepen history of shallow clone, excluding rev | | --deepen deepen history of shallow clone | | --unshallow convert to a complete repository | | --update-shallow accept refs that update .git/shallow | | --refmap specify fetch refmap | | -o, --server-option | | option to transmit | | -4, --ipv4 use IPv4 addresses only | | -6, --ipv6 use IPv6 addresses only | | --negotiation-tip | | report that we have only objects reachable from this object | | --filter object filtering | | --auto-maintenance run 'maintenance --auto' after fetching | | --auto-gc run 'maintenance --auto' after fetching | | --show-forced-updates | | check for forced-updates on all updated branches | | --write-commit-graph write the commit-graph after fetching | | --stdin accept refspecs from stdin | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *withstack.withStack (8) *errutil.leafError Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError`

What do you expect to happen

Extract of git info should be successful.

How to reproduce

Move from beta4 to beta5

Anything else

No response

Version

v1.0.1-beta5@ebddfe3

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hey @Capone1983! I'm here to help you with any bugs, questions, or contributions you have in mind. Let's tackle this issue together!

The issue you're encountering with the gitextractor step failing after moving from beta4 to beta5 is likely due to a permission problem with your tokens. The error message "error: unknown switch c" suggests that there might be an issue with the Git command being executed.

To resolve this issue, you should:

1. Verify that you are using a supported authentication method.
2. Ensure that the tokens have the correct permissions for the entities you want to collect.

Refer to the corresponding plugin's Config UI manuals for the supported authentication methods and required token permissions. For example, you can find the necessary information for configuring GitHub tokens here.

Additionally, you can refer to the Configuration and Blueprint Troubleshooting guide for more details on common error codes and their solutions. Specifically, the guide mentions that a 403 error is usually caused by the permission of your tokens, and it provides steps to ensure you have the correct permissions ^{[1][2][3][4][5]}.

Also, ensure that you have libgit2 version 1.3.0 installed, as it is required to run and debug the gitextractor plugin on your local machine ^[6][7][8][9].

To continue talking to Dosu, mention @dosu.

[Capone1983]

We should not have to change anything when moving to different versions.
Reverted back to beta4 and its working fine again.

[Startrekzky]

@Capone1983 Got it. We'll look at it asap.

[denodaeus]

Also encountering this issue on beta5:

Mtime="2024-08-05 14:10:04" level=error msg=" [pipeline service] [pipeline #12] run tasks failed\n\tcaused by: Error running task 81.\n\tWraps: (2) subtask Clone Git Repo ended unexpectedly\n\tWraps: (3) failed to fetch all branches from the remote server\n\tWraps: (4) git cmd [git fetch --depth=1 origin -c http.sslVerify=false] in /tmp/gitextractor3731976518 failed: error: unknown switch c'\n\t | usage: git fetch [] [ [...]]\n\t | or: git fetch [] \n\t | or: git fetch --multiple [] [( | )...]\n\t | or: git fetch --all []\n\t |\n\t | -v, --verbose be more verbose\n\t | -q, --quiet be more quiet\n\t | --all fetch from all remotes\n\t | --set-upstream set upstream for git pull/fetch\n\t | -a, --append append to .git/FETCH_HEAD instead of overwriting\n\t | --upload-pack path to upload pack on remote end\n\t | -f, --force force overwrite of local reference\n\t | -m, --multiple fetch from multiple remotes\n\t | -t, --tags fetch all tags and associated objects\n\t | -n do not fetch all tags (--no-tags)\n\t | -j, --jobs number of submodules fetched in parallel\n\t | -p, --prune prune remote-tracking branches no longer on remote\n\t | -P, --prune-tags prune local tags no longer on remote and clobber changed tags\n\t | --recurse-submodules[=]\n\t | control recursive fetching of submodules\n\t | --dry-run dry run\n\t | --write-fetch-head write fetched references to the FETCH_HEAD file\n\t | -k, --keep keep downloaded pack\n\t | -u, --update-head-ok allow updating of HEAD ref\n\t | --progress force progress reporting\n\t | --depth dee deepen history of shallow repository based on time\n\t | --shallow-exclude \n\t | deepen history of shallow clone, excluding rev\n\t | --deepen deepen history of shallow clone\n\t | --unshallow convert to a complete repository\n\t | --update-shallow accept refs that update .git/shallow\n\t | --refmap specify fetch refmap\n\t | -o, --server-option \n\t | option to transmit\n\t | -4, --ipv4 use IPv4 addresses only\n\t | -6, --ipv6 use IPv6 addresses only\n\t | --negotiation-tip \n\t | report that we have only objects reachable from this object\n\t | --filter object filtering\n\t | --auto-maintenance run 'maintenance --auto' after fetching\n\t | --auto-gc run 'maintenance --auto' after fetching\n\t | --show-forced-updates\n\t | check for forced-updates on all updated branches\n\t | --write-commit-graph write the commit-graph after fetching\n\t | --stdin accept refspecs from stdin\n\tError types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *hintdetail.withDetail (4) *errors.errorString\n\t"
time="2024-08-05 14:10:04" level=info msg=" [pipeline service] [pipeline #12] return error"`

I noticed that due to some self signed certs issues with my on-prem Bitbucket Server instance, the -c is being passed in with an argument after with sslVerify=false. git -c expects a configuration string. I also launched the worker container to make sure the -c argument was available it it's version of git.

[d4x1]

@denodaeus Can you run git fetch --depth=1 origin -c http.sslVerify=false sucessfully in your local repo?
I think this cli should be git -c http.sslVerify=false clone_or_fetch.

[Startrekzky]

@dosu Could you stop labeling GitHub issues with tags starting with pr-type/?

[dosubot]

The pr-type/ labels are used to categorize pull requests based on their nature, such as pr-type/bug-fix or pr-type/feature-development ^[1].

However, there is no information available about any automated workflows or bots that add these labels, nor are there specific guidelines or documentation on labeling GitHub issues in the repository ^[2][1].

If you need to stop labeling issues with pr-type/ tags, you may need to manually adjust your labeling practices or update any existing automation scripts that apply these labels.

To continue talking to Dosu, mention @dosu.

[devstein]

Hey @Startrekzky apologies for the confusion here. It looks like Dosu's issue and PR label settings weren't set up correctly. I just fixed it, so Dosu won't manage pr-type tags or labels moving or other irrelevant issues labels moving forwarrd.

If you have any other issues or question, feel free to ping me in our community slack

[denodaeus]

Any idea when you'll have a beta6 with this change? beta4 has a bunch of broken dashboards with bitbucket server, and beta5 won't run for me because of the above

[klesh]

Hi, All. PR #7941 should fix the ssl verify problem.