Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JIRA extra_fields override description_template #1075

Closed
EquineOntology opened this issue Nov 8, 2024 · 10 comments · Fixed by #1079
Closed

JIRA extra_fields override description_template #1075

EquineOntology opened this issue Nov 8, 2024 · 10 comments · Fixed by #1079

Comments

@EquineOntology
Copy link
Contributor

Hello!

I just started using taskwarrior & bugwarrior and they're really neat. I'd like to use the JIRA integration to pull the assignee's email and assign it as a tag, but I'm encountering some funky behaviour - it seems that as soon as I define jira.extra_fields, the dict of extra fields overrides description_template, and I'm wondering if there are any known workarounds.

Without extra_fields: description works

[general]
targets = eq_jira
inline_links = True
annotation_comments = False
replace_tags = True
static_tags = mgmt,tl,ic
static_fields = priority,project

[eq_jira]
service = jira
jira.base_uri = <redacted>
jira.username = <redacted>
jira.password = <redacted>
jira.import_labels_as_tags = False
jira.import_sprints_as_tags = False
jira.query = <redacted>
jira.description_template = {{jiraid}}
jira.project_template =
jira.default_priority =

CleanShot 2024-11-08 at 14 33 28@2x

With extra_fields: description is overwritten

# identical setup as above except for the addition of extra_fields and add_tags
jira.extra_fields = test_field:customfield_10019
# test_field will contain gibberish, that's expected, I'm using it to show the issue
jira.add_tags = {{test_field}}

In this setup it seems like the map of extra_fields overrides the description?

CleanShot 2024-11-08 at 14 26 36@2x

Other notes

  • I'm positive the issue is not with jira.add_tags: as long as I don't add extra_fields, just using tags doesn't override the description
  • I'm using the latest version, not 1.8.0 (I installed from source), as 1.8.0 doesn't support jira.extra_fields.

Any help appreciated!

@ryneeverett ryneeverett added this to the release-next milestone Nov 9, 2024
@ryneeverett
Copy link
Collaborator

In this setup it seems like the map of extra_fields overrides the description?

Very odd. Since you have description_template configured, the map of extra_fields will be merged into the templating context (variables such as jiraid) but it seems like the template itself is changing. Odder yet, there doesn't seem to be anything in your configuration which aligns with the output we're seeing -- it is as if description_template = test_field:"{{test_field}}".

My hunch is that the underlying cause is a configuration parsing error. Could you run bugwarrior ini2toml and post the (redacted) output?

@EquineOntology
Copy link
Contributor Author

Certainly! Turns out it was a bit of a process, so I'll include that as well in case it's helpful for bugwarrior in general. I can't quite figure out how to make the configuration parser happy to run that last bugwarrior pull though :)

1. bugwarrior ini2toml

(FYI bugwarrior was installed yesterday by pulling the latest develop and running python setup.py install)

➜  bugwarrior ini2toml
Install extra dependencies to use this command:
    pip install bugwarrior[ini2toml]

2. pip install "bugwarrior[ini2toml]"

Seems like I was missing ini2toml, tomlkit and configupdater

➜  pip install "bugwarrior[ini2toml]"
Looking in indexes: https://gitlab.com/api/v4/groups/10866812/-/packages/pypi/simple
Requirement already satisfied: bugwarrior[ini2toml] in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages/bugwarrior-1.8.0-py3.10.egg (1.8.0)
Requirement already satisfied: click in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from bugwarrior[ini2toml]) (8.1.7)
Requirement already satisfied: dogpile.cache>=0.5.3 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from bugwarrior[ini2toml]) (1.3.3)
Requirement already satisfied: jinja2>=2.7.2 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from bugwarrior[ini2toml]) (3.1.4)
Requirement already satisfied: lockfile>=0.9.1 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from bugwarrior[ini2toml]) (0.12.2)
Requirement already satisfied: pydantic[email]>=2 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from bugwarrior[ini2toml]) (2.8.2)
Requirement already satisfied: python-dateutil in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from bugwarrior[ini2toml]) (2.9.0.post0)
Requirement already satisfied: pytz in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from bugwarrior[ini2toml]) (2024.2)
Requirement already satisfied: requests in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from bugwarrior[ini2toml]) (2.32.3)
Requirement already satisfied: taskw>=0.8 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from bugwarrior[ini2toml]) (2.0.0)
Requirement already satisfied: tomli in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from bugwarrior[ini2toml]) (2.0.1)
Requirement already satisfied: typing-extensions in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from bugwarrior[ini2toml]) (4.12.2)
Collecting ini2toml[full]
  Downloading ini2toml-0.15-py3-none-any.whl (55 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.7/55.7 kB 512.6 kB/s eta 0:00:00
Requirement already satisfied: decorator>=4.0.0 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from dogpile.cache>=0.5.3->bugwarrior[ini2toml]) (5.1.1)
Requirement already satisfied: stevedore>=3.0.0 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from dogpile.cache>=0.5.3->bugwarrior[ini2toml]) (5.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from jinja2>=2.7.2->bugwarrior[ini2toml]) (2.1.5)
Requirement already satisfied: annotated-types>=0.4.0 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from pydantic[email]>=2->bugwarrior[ini2toml]) (0.7.0)
Requirement already satisfied: pydantic-core==2.20.1 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from pydantic[email]>=2->bugwarrior[ini2toml]) (2.20.1)
Requirement already satisfied: email-validator>=2.0.0 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages/email_validator-2.2.0-py3.10.egg (from pydantic[email]>=2->bugwarrior[ini2toml]) (2.2.0)
Requirement already satisfied: kitchen in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from taskw>=0.8->bugwarrior[ini2toml]) (1.2.6)
Requirement already satisfied: setuptools>=59.6 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from ini2toml[full]->bugwarrior[ini2toml]) (65.5.0)
Requirement already satisfied: packaging>=20.7 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from ini2toml[full]->bugwarrior[ini2toml]) (24.1)
Collecting tomlkit<2,>=0.10.0
  Downloading tomlkit-0.13.2-py3-none-any.whl (37 kB)
Collecting configupdater<4,>=3.0.1
  Downloading ConfigUpdater-3.2-py2.py3-none-any.whl (34 kB)
Requirement already satisfied: six>=1.5 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from python-dateutil->bugwarrior[ini2toml]) (1.16.0)
Requirement already satisfied: certifi>=2017.4.17 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->bugwarrior[ini2toml]) (2024.7.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->bugwarrior[ini2toml]) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->bugwarrior[ini2toml]) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->bugwarrior[ini2toml]) (2.2.2)
Requirement already satisfied: dnspython>=2.0.0 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages/dnspython-2.7.0-py3.10.egg (from email-validator>=2.0.0->pydantic[email]>=2->bugwarrior[ini2toml]) (2.7.0)
Requirement already satisfied: pbr>=2.0.0 in /Users/christian/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from stevedore>=3.0.0->dogpile.cache>=0.5.3->bugwarrior[ini2toml]) (6.1.0)
Installing collected packages: tomlkit, ini2toml, configupdater
Successfully installed configupdater-3.2 ini2toml-0.15 tomlkit-0.13.2

3. Run ini2toml

➜  bugwarrior ini2toml
[general]
targets = ["eq_jira"]
inline_links = true
annotation_comments = false
replace_tags = true
static_tags = ["mgmt", "tl", "ic"]
static_fields = ["priority", "project"]

[eq_jira]
service = "jira"
base_uri = <redacted>
username = <redacted>
password = <redacted>
import_labels_as_tags = false
import_sprints_as_tags = false
query = <redacted>
extra_fields = "test_field:customfield_10019"
add_tags = ["{{test_field}}"]
description_template = "{{jiraid}}"
project_template = ""
default_priority = ""

4. Run pull

➜  bugwarrior pull
[eq_jira]
base_uri <-expected prefix '"jira"': did you mean '"jira".base_uri'?

5. Add service prefixes & re-run pull

# everything above this is the same
[eq_jira]
service = "jira"
jira.base_uri = <redacted>
jira.username = <redacted>
jira.password = <redacted>
jira.import_labels_as_tags = false
jira.import_sprints_as_tags = false
jira.query = <redacted>
jira.extra_fields = "test_field:customfield_10019"
jira.add_tags = ["{{test_field}}"]
jira.description_template = "{{jiraid}}"
jira.project_template = ""
jira.default_priority = ""
➜  bugwarrior pull
Validation error found in /Users/christian/.bugwarriorrc
See https://bugwarrior.readthedocs.io

No section: '["eq_jira"]'

And here is where I'm stuck - I can't figure out how to make it happy. I double-checked the TOML docs and even asked ChatGPT, and the syntax seems correct. I also tried a bunch of permutations for the quotes in targets = ["eq_jira"] and the [eq_jira] section header, but no dice.

@ryneeverett
Copy link
Collaborator

Sorry for the additional process, I had forgotten that ini2toml had dependencies which weren't included by default, but at least that part was self-explanatory.

I'm assuming that between steps 3 and 4 you replaced the contents of your bugwarriorrc file with the output of bugwarrior ini2toml? I forgot to write any documentation about toml configuration (#1001) so that's an understandable error. I just wanted to see the output because it reveals how bugwarrior is interpreting your ini configuration.

You either need to revert your configuration file to what you had before copying in the ini2toml output, or rename it .bugwarrior.toml and revert it to the ini2toml output before you removed the service prefixes. (Service prefixes are required in ini but not toml.) What you have right now is halfway between ini and toml and neither parser is going to like it.

Unfortunately the ini2toml output doesn't give me any clues as to the original bug here, but it mostly rules out configuration parsing as the underlying problem.

@EquineOntology
Copy link
Contributor Author

I'm assuming that between steps 3 and 4 you replaced the contents of your bugwarriorrc file with the output of bugwarrior ini2toml?

Correct! And as you expected, once I set up the toml config correctly the output was the same -
CleanShot 2024-11-10 at 17 43 24@2x

@ryneeverett
Copy link
Collaborator

We don't have a great way to test a configuration from end to end, but I tried to do so in two passes:

  • from configuration file to parsed configuration
  • from parsed configuration to the record inserted into taskwarrior

configuration file to parsed configuration

  • I copied your ini2toml output into my configuration file with the minimal adjustments to make it valid.
configuration file
[general]
targets = ["eq_jira"]
inline_links = true
annotation_comments = false
replace_tags = true
static_tags = ["mgmt", "tl", "ic"]
static_fields = ["priority", "project"]

[eq_jira]
service = "jira"
base_uri = "https://example.com"
username = "<redacted>"
password = "<redacted>"
import_labels_as_tags = false
import_sprints_as_tags = false
query = "<redacted>"
extra_fields = "test_field:customfield_10019"
add_tags = ["{{test_field}}"]
description_template = "{{jiraid}}"
project_template = ""
default_priority = ""
  • I threw a breakpoint at the beginning of JiraService.__init__, ran bugwarrior pull --dry-run --debug, and verified that everything was parsed as expected:
log
(Pdb) pp dict(self.config)
{'PAT': '',
 'add_tags': ['{{test_field}}'],
 'also_unassigned': False,
 'annotations_template': None,
 'base_uri': AnyUrl('https://example.com', scheme='https', host='example.com', tld='com', host_type='domain'),
 'body_length': 9223372036854775807,
 'default_priority': '',
 'depends_template': None,
 'description_template': '{{jiraid}}',
 'due_template': None,
 'end_template': None,
 'entry_template': None,
 'extra_fields': [JiraExtraField(label='test_field',
                                 keys=['customfield_10019'])],
 'id_template': None,
 'imask_template': None,
 'import_labels_as_tags': False,
 'import_sprints_as_tags': False,
 'label_template': '{{label}}',
 'mask_template': None,
 'modified_template': None,
 'only_if_assigned': '',
 'parent_template': None,
 'password': '<redacted>',
 'priority_template': None,
 'project_template': '',
 'query': '<redacted>',
 'recur_template': None,
 'scheduled_template': None,
 'service': 'jira',
 'start_template': None,
 'status_template': None,
 'tags_template': None,
 'target': 'eq_jira',
 'templates': {'description': '{{jiraid}}', 'project': ''},
 'until_template': None,
 'urgency_template': None,
 'use_cookies': False,
 'username': '<redacted>',
 'uuid_template': None,
 'verify_ssl': True,
 'version': 5,
 'wait_template': None}

parsed configuration to the record inserted into taskwarrior

  • I modified the jira tests to be as similar to your configuration as possible.
  • The tests passed and performed as expected.
  • Specifically notable is TestJiraIssue.test_issues.
    • You can see that the description is the same as the jiraid as expected.
    • There isn't much point in examining a higher level of abstraction than this -- this method calls collect.TaskConstructor.get_taskwarrior_record(), which is the same method called by collect.aggregate_issues which is called by command.pull.

insertion of record into taskwarrior

My tentative conclusion is that bugwarrior is trying to insert the correct data into taskwarrior and that is where the problem is occurring. Bugwarrior currently uses taskw to do this, which ultimately uses the command line. This generally works really well but some characters aren't always properly escaped and the result is often that taskwarrior misinterprets the arguments.

In your case, it appears that "test_field:"..." is supposed to be inserted as an extra field, but taskwarrior is interpreting it as the final argument and thus the description. I'm not sure what is happening with the actual description or any other fields that are supposed to be inserted but seem to be getting cut off -- you might have some shell errors in your logs (if taskw isn't suppressing them). You could get some confirmation of this theory by introspecting a task (e.g. task 29) and checking if it has the test_field UDA. This theory predicts it will be missing.

I kind of suspect the pipe or colon in your test_field are causing the problem -- perhaps at the point in the shell command where it is inserted as a tag. You could get some confirmation of this possibility by removing the add_tags configuration. If your test_field's are only a problem when inserted as tags, we would expect this to correct the description. It's not that add_tags is inherently buggy -- as you said you'd used it independently -- but it's possible that the particular strings of your test_field are causing the buggy behavior when they are inserted as tags.

@EquineOntology
Copy link
Contributor Author

EquineOntology commented Nov 11, 2024

Thank you for the detailed follow-up! Here's my own :)

Pipe symbol as a potential root cause

I got really excited when you mentioned this! But then I remembered the original fields I was using didn't have pipes either :( I ran a test using the issue type (Epic/Bug/Story/Task etc.) to make sure and indeed, same result
CleanShot 2024-11-11 at 12 20 25@2x

CleanShot 2024-11-11 at 12 14 25@2x

test_field missing from task info

Spot on!

CleanShot 2024-11-11 at 12 18 50@2x

Removing add_tags

Interestingly, commenting add_tags in the config file doesn't actually help. That might be a natural consequence of the first point though.

CleanShot 2024-11-11 at 12 25 55@2x


Is there a way to see the command that is passed to taskw? I tried running bugwarrior pull with DEBUG log level but the output didn't offer any hints.

@ryneeverett
Copy link
Collaborator

test_field missing from task info

Spot on!

Great! I think that confirms that we're looking in the right direction.

Is there a way to see the command that is passed to taskw? I tried running bugwarrior pull with DEBUG log level but the output didn't offer any hints.

Unfortunately there isn't a great way, no. You could do strace -f bugwarrior pull though and search through the (quite verbose) output.

@ryneeverett ryneeverett mentioned this issue Nov 13, 2024
@EquineOntology
Copy link
Contributor Author

EquineOntology commented Nov 13, 2024

Hello again! I took some time over the last couple of days to set up a debugger and follow the data. TLDR I don't think this is a bugwarrior issue, as you initially suspected. In my understanding bugwarrior is behaving correctly, i.e. at no point is it overriding description with the wrong data.

The problem appears when the list of changes is created here:

  1. the value of modifications ends up being ['description:"EQPD-19227"', 'tags:"Story"', 'test_field:"Story"']
  2. which means the args passed to taskw through _execute are ('4d90ffc4-8274-47cb-94ba-38de67060d33', 'modify', 'description:"EQPD-19227"', 'tags:"Story"', 'test_field:"Story"')

It simply seems like task is taking all elements it doesn't recognize as parts of description. Consider the following tests (FYI in all cases, the tag is set correctly as Story):

Correct/Wrong Command Description after running Case
task id:123 modify description:"EQPD-19227" tags:"Story" EQPD-19227 Baseline
task id:123 modify description:"EQPD-19227" tags:"Story" test_field:"Story" test_field:Story Add non-standard argument
task id:123 modify a_test_field:"Story" description:"EQPD-19227" tags:"Story" a_test_field:Story Check if order of arguments has any effect (no)
task id:123 modify a_test_field:"Story" description:"EQPD-19227" tags:"Story" test_field:"Story" a_test_field:Story test_field:Story Check behaviour with multiple non-standard arguments

The only question I have is whether the non-standard argument that appear in the command being executed are supposed to be part of the command. Given the involvement of taskw in the process my hunch is yes, and if that is the case this doesn't seem like a bugwarrior issue, so I'd be inclined to close this.


Ok, this is entirely outside of bugwarrior but just to provide closure: I don't know C++ very well, but if I follow the code correctly this seems like a case of "reasonable approach to unknown arguments causing an unintended consequence in my specific case" in taskwarrior.

  1. CmdModify::execute calls Task::modify.
  2. Task::modify treats unknown arguments as normal words, and those get appended to the description. I think that's an explanation that matches the examples in the table above.

@ryneeverett given that I was ultimately trying to do the same as #826, is there some UDA configuration darck magick I can perform to prevent my custom field from being regarded as an "unknown" one? If not, I can pick up the mantle of #826, at this point I'm way too invested to give up 😉

@EquineOntology
Copy link
Contributor Author

EquineOntology commented Nov 13, 2024

Hahaha well, turns out it was very simple - just define a UDA for the specific custom field I want to integrate.

In .bugwarrior.toml

extra_fields = "jiraassignee:assignee.emailAddress"
add_tags = ["{{jiraassignee}}"]

And in .taskrc

uda.jiraassignee.type=string
uda.jiraassignee.label=Jira assignee

And now it works like a charm :D

@ryneeverett I think the lesson here is that I should update this section of the docs to clarify a bit. Does that seem like an ok solution?

@ryneeverett
Copy link
Collaborator

@ryneeverett I think the lesson here is that I should update this section of the docs to clarify a bit. Does that seem like an ok solution?

Yes, sounds good!

It would also be nice if we could add a validator to the extra_fields option to check that the uda's are configured if the (optional) taskrc option is set. However, it isn't clear to me that there is a reasonable way to access the "Main Section" configuration from a service's validators, so this is probably not practical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants