Variable substitution in URL only works for finished builds #26

jelco · 2016-03-21T16:03:24Z

In my setup I have some variable components of not just the payload but also the URL. I have been able to pass these values to the plugin through build parameters and configure substitution in the URL using the ${varname} syntax. However, this only seems to work for the events triggered by a completed build (success and failure). I would also like to trigger a hook when a build is started, but that hook doesn't seem to perform proper variable substitution in the URL yet:

Error in event handler: java.lang.IllegalArgumentException: Invalid uri 'https://api.bitbucket.org/2.0/repositories/organization/%system.teamcity.projectName%/commit/%system.build.vcs.number%/statuses/build': Invalid URL encoding (enable debug to see stacktrace)

Note that the configuration has the URL listed as follows:

https://api.bitbucket.org/2.0/repositories/organization/${project}/commit/${revision}/statuses/build

Meaning that there is a certain extent of replacement going on (e.g. ${project} to %system.teamcity.projectName%) but not complete enough. Perhaps it's just a matter of that two-pass variable substitution that I've seen mentioned? Note that directly entering the percent-style variable names in the XML file always results in errors, even in the success/failure hooks, so I'm guessing it's a primary pass that is missing (that substitutes the percent-style variables with their actual values) instead of a secondary pass (for the dollar-style substitution).

(The API URLs have been anonymized.)

The text was updated successfully, but these errors were encountered:

netwolfuk · 2016-03-23T08:04:34Z

Hi @jelco , which template and format are you using?

jelco · 2016-03-23T08:14:13Z

Tailored JSON without a template. Here, I'll just include the full specification of this particular webhook (with some anonymization):

<webhook url="https://api.bitbucket.org/2.0/repositories/organization/${project}/commit/${revision}/statuses/build" enabled="true" format="tailoredjson" template="none">
  <states>
    <state type="buildFailed" enabled="false" />
    <state type="beforeBuildFinish" enabled="false" />
    <state type="buildFinished" enabled="false" />
    <state type="buildStarted" enabled="true" />
    <state type="buildFixed" enabled="false" />
    <state type="buildBroken" enabled="false" />
    <state type="buildSuccessful" enabled="false" />
    <state type="buildInterrupted" enabled="false" />
    <state type="responsibilityChanged" enabled="false" />
  </states>
  <build-types enabled-for-all="true" enabled-for-subprojects="true" />
  <parameters>
    <param name="body" value="{&quot;state&quot;: &quot;INPROGRESS&quot;, &quot;key&quot;: &quot;${key}&quot;, &quot;name&quot;: &quot;${name}&quot;, &quot;url&quot;: &quot;${url}&quot;, &quot;description&quot;: &quot;${buildStatus}&quot;}" />
  </parameters>
  <auth enabled="true" type="userpass" preemptive="true">
    <auth-parameters>
      <param name="password" value="password" />
      <param name="username" value="username" />
    </auth-parameters>
  </auth>
</webhook>

I'm running the custom-templates branch (the latest version you supplied with the pre-emptive authentication).

netwolfuk · 2016-03-23T09:41:46Z

I'm having trouble getting your example to work for any build state. Can you confirm that it works for you for the finished states. (I note that the finished states are set to false above).

jelco · 2016-03-23T11:43:12Z

Ah, I understand the confusion. I have three separate webhooks (since the content is different per hook). The full configuration is as follows:

<webhooks enabled="true">
  <webhook url="https://api.bitbucket.org/2.0/repositories/organization/${project}/commit/${revision}/statuses/build" enabled="true" format="tailoredjson" template="none">
    <states>
      <state type="buildFailed" enabled="false" />
      <state type="beforeBuildFinish" enabled="false" />
      <state type="buildFinished" enabled="false" />
      <state type="buildStarted" enabled="true" />
      <state type="buildFixed" enabled="false" />
      <state type="buildBroken" enabled="false" />
      <state type="buildSuccessful" enabled="false" />
      <state type="buildInterrupted" enabled="false" />
      <state type="responsibilityChanged" enabled="false" />
    </states>
    <build-types enabled-for-all="true" enabled-for-subprojects="true" />
    <parameters>
      <param name="body" value="{&quot;state&quot;: &quot;INPROGRESS&quot;, &quot;key&quot;: &quot;${key}&quot;, &quot;name&quot;: &quot;${name}&quot;, &quot;url&quot;: &quot;${url}&quot;, &quot;description&quot;: &quot;${buildStatus}&quot;}" />
    </parameters>
    <auth enabled="true" type="userpass" preemptive="true">
      <auth-parameters>
        <param name="password" value="password" />
        <param name="username" value="username" />
      </auth-parameters>
    </auth>
  </webhook>
  <webhook url="https://api.bitbucket.org/2.0/repositories/organization/${project}/commit/${revision}/statuses/build" enabled="true" format="tailoredjson" template="none">
    <states>
      <state type="buildFailed" enabled="false" />
      <state type="beforeBuildFinish" enabled="false" />
      <state type="buildFinished" enabled="true" />
      <state type="buildStarted" enabled="false" />
      <state type="buildFixed" enabled="false" />
      <state type="buildBroken" enabled="false" />
      <state type="buildSuccessful" enabled="true" />
      <state type="buildInterrupted" enabled="false" />
      <state type="responsibilityChanged" enabled="false" />
    </states>
    <build-types enabled-for-all="true" enabled-for-subprojects="true" />
    <parameters>
      <param name="body" value="{&quot;state&quot;: &quot;SUCCESSFUL&quot;, &quot;key&quot;: &quot;${key}&quot;, &quot;name&quot;: &quot;${name}&quot;, &quot;url&quot;: &quot;${url}&quot;, &quot;description&quot;: &quot;${buildStatus}&quot;}" />
    </parameters>
    <auth enabled="true" type="userpass" preemptive="true">
      <auth-parameters>
        <param name="password" value="password" />
        <param name="username" value="username" />
      </auth-parameters>
    </auth>
  </webhook>
  <webhook url="https://api.bitbucket.org/2.0/repositories/organization/${project}/commit/${revision}/statuses/build" enabled="true" format="tailoredjson" template="none">
    <states>
      <state type="buildFailed" enabled="true" />
      <state type="beforeBuildFinish" enabled="false" />
      <state type="buildFinished" enabled="true" />
      <state type="buildStarted" enabled="false" />
      <state type="buildFixed" enabled="false" />
      <state type="buildBroken" enabled="false" />
      <state type="buildSuccessful" enabled="false" />
      <state type="buildInterrupted" enabled="false" />
      <state type="responsibilityChanged" enabled="false" />
    </states>
    <build-types enabled-for-all="true" enabled-for-subprojects="true" />
    <parameters>
      <param name="body" value="{&quot;state&quot;: &quot;FAILED&quot;, &quot;key&quot;: &quot;${key}&quot;, &quot;name&quot;: &quot;${name}&quot;, &quot;url&quot;: &quot;${url}&quot;, &quot;description&quot;: &quot;${buildStatus}&quot;}" />
    </parameters>
    <auth enabled="true" type="userpass" preemptive="true">
      <auth-parameters>
        <param name="password" value="password" />
        <param name="username" value="username" />
      </auth-parameters>
    </auth>
  </webhook>
</webhooks>

netwolfuk · 2016-03-23T11:44:52Z

I have been able to reproduce the problem. Using the following URL:
http://localhost:8111/2.0/repositories/organization/${project}/commit/${revision}/statuses/build

And the properties on the build as so:
webhook.project %system.teamcity.projectName%
webhook.revision %system.build.vcs.number%

Before the build starts, TeamCity has not done any change gathering, (and I assume, resolved any parameters).

I suspect that is probably caused by the same as this issue in which Pavel states:

Before changes are collected in build, we actually do not know what will be the display name of the branch. This is why on buildStarted event you get , but on changesLoaded event if changes were collected without errors, you'll see master. So far this is by design.

So the correct answer is probably for me to add "changesLoaded" build event to the listener.

netwolfuk · 2016-03-23T11:54:19Z

Created #28, and will close this ticket.

netwolfuk · 2016-03-23T13:45:51Z

ChangesLoaded support added see commit ca4e0cf
It has just built and is ready to download

See if that gives you what you want.

jelco · 2016-03-23T16:23:56Z

I disabled the buildStarted state line and added this line to the configuration:

<state type="changesLoaded" enabled="true" />

Unfortunately, the result doesn't change a whole lot. I can definitely notice that the hook is triggered at a later stage so it looks like it's picking up the other event just fine, but the error isn't any different:

Error in event handler: java.lang.IllegalArgumentException: Invalid uri 'https://api.bitbucket.org/2.0/repositories/organization/%system.teamcity.projectName%/commit/%system.build.vcs.number%/statuses/build': Invalid URL encoding (enable debug to see stacktrace)

netwolfuk · 2016-03-23T19:11:34Z

That's a real shame. The first one should just be available as ${projectName} as that's a tcwebhooks PayloadContent bean property. I can't remember if I added the vcs stuff to the bean yet though or how you'd access it. I'll have a dig into it and see.

netwolfuk · 2016-05-21T10:30:29Z

https://youtrack.jetbrains.com/issue/TW-45285

Addresses issue #26

netwolfuk · 2021-10-17T08:35:49Z

Hi @Lurenlzm @jelco
I have added experimental support for forcing TeamCity to resolve variables before it would normally do so. This means that (in my limited testing) certain variables can be resolved before the build started event fires.

To test it, please download the plugin and rest plugin from the issue_26-force_resolve_variables branch. Just login as a guest to download the artifacts.

This branch adds support for a new settings on WebHook Project Parameters called "Forced Resolve".

To use this feature:

Install a 1.2.0-alpha8 or later tcWebHooks plugin and tcWebHooks REST plugin.
Create a WebHooks Parameter for your project. The page to create them is on the bottom of the webhook edit page.
Set the name you want to use in your webhook template or URL. eg, my_var_name
Set the value to refer to a TeamCity variable using Teamcity Syntax eg, %my_parameter%. Note the use of % not ${}.
Set the Force Reload option to Attempt to resolve variable contents early via TeamCity.
Refer to ${my_var_name} in your webhook configuration (eg, template, URL or other webhook setting).

tcWebHooks will ask TeamCity to early resolve that parameter before the webhook is executed.

jan0sch · 2021-10-27T13:24:50Z

Hi, at first thanks for your work. ❤️
I just tried this out but still get the malformed url error message.

Configuration of a custom parameter as described in your screenshot:

Used in the URL via ${CommitHash} but getting the error message:

999 :: Unexpected exception. Please log a bug on GitHub tcplugins/tcWebHooks. Exception was: Malformed escape pair at index 66: https://XXX/api/v1/repos/XXX/XXX/statuses/%build.vcs.number%

For me it looks like the variable is not substituted but the reference to the TeamCity variable (%vcs.build.number% in my case) is just taken as is.

netwolfuk · 2021-10-27T18:03:49Z

Hi @jan0sch thanks so much for taking the time to test this.

I presume this is for build started. Is it the same for changes loaded?

I'm not at my computer so I can't remember the exact text, but there should be some debug logging in the teamcity-server.log for something like WebHookExtraParameters indicating that it's trying to force resolve that variable. Can you please check is that's running?

Thanks again for your time on this.

NEdwards-Sharkmob · 2021-11-07T21:55:04Z

Hi @Lurenlzm @jelco I have added experimental support for forcing TeamCity to resolve variables before it would normally do so. This means that (in my limited testing) certain variables can be resolved before the build started event fires.

To test it, please download the plugin and rest plugin from the issue_26-force_resolve_variables branch. Just login as a guest to download the artifacts.

This branch adds support for a new settings on WebHook Project Parameters called "Forced Resolve".

To use this feature:

Install a 1.2.0-alpha8 or later tcWebHooks plugin and tcWebHooks REST plugin.

Create a WebHooks Parameter for your project. The page to create them is on the bottom of the webhook edit page.

Set the name you want to use in your webhook template or URL. eg, my_var_name

Set the value to refer to a TeamCity variable using Teamcity Syntax eg, %my_parameter%. Note the use of % not ${}.

Set the Force Reload option to Attempt to resolve variable contents early via TeamCity.

Refer to ${my_var_name} in your webhook configuration (eg, template, URL or other webhook setting).

tcWebHooks will ask TeamCity to early resolve that parameter before the webhook is executed.

I installed this version as I've been experiencing the same issue.

I found that using the force early resolve fixed the issue I was having where my %build.vcs.number% didn't resolve in Changes Loaded, but it didn't resolve for Build Succeeded or Build Failed events. I was able to verify this with the test webhooks feature.

However, when I tried using ${build.vcs.number} as the value in my project webhook parameter, this worked for all 3 (changes loaded, build succeeded, build failed) - verified via the test feature and in use. Not sure if this would solve the issue being described here, but figured it would be worth mentioning. It would be handy to know if issues could result from doing this too.

netwolfuk · 2021-11-07T23:27:09Z

Hi @NEdwards-Sharkmob

Thanks for taking the time to test this.

I made an assumption that we'd only want to force resolve for early build stages. From memory, I don't think the force resolve tries to resolve the values in the build finished event. However for consistency, I should add that in.

Please also bear in mind that testing these sorts of variables in the UI based on a finished build often gives misleading results because variables have been resolved by TeamCity during the build and will appear resolved even if the event being tested is an early one. I see you mentioned that you did also test with a real build, but I just wanted to point that out for other people on this thread.

I guess when we're talking about change related metadata it makes sense that those values wont resolve until changes are loaded. I don't know enough about the inner workings of TeamCity to answer that authoritatively.

I'll check the force resolving on later build events and add if required.

netwolfuk · 2021-11-12T18:47:23Z

Hi all. A new build is available. This enables the resolver to run on the following:
state.equals(BuildStateEnum.BUILD_FINISHED) || state.equals(BuildStateEnum.BUILD_SUCCESSFUL) || state.equals(BuildStateEnum.BUILD_FAILED) || state.equals(BuildStateEnum.BUILD_FIXED) || state.equals(BuildStateEnum.BUILD_BROKEN)

Available from the issue_26-force_resolve_variables link.

jan0sch · 2021-11-16T05:57:48Z

Variable resolution seems to work now. Thank you very much.

* Add support for "Force Resolve" feature of WebHook Project Parameters Addresses issue #26 * Force resolve variables on the Build Finished events

netwolfuk · 2023-03-11T00:38:46Z

1.2.0 has been released. Closing this issue

* Add support for "Force Resolve" feature of WebHook Project Parameters Addresses issue #26 * Force resolve variables on the Build Finished events

netwolfuk closed this as completed Mar 23, 2016

netwolfuk reopened this Mar 23, 2016

netwolfuk mentioned this issue May 30, 2016

Resolving of build parameters seems to not work in 1.1-alpha5.81.121 #34

Closed

netwolfuk mentioned this issue Aug 18, 2016

teamcity variables do not work for 'build started' trigger w/ tailored JSON #46

Closed

jelco mentioned this issue Aug 2, 2017

Variable substitution fails for explicitly specified branches #72

Open

netwolfuk mentioned this issue Apr 18, 2018

Variables not populated in the Started event #102

Closed

netwolfuk added a commit that referenced this issue Oct 17, 2021

Add support for "Force Resolve" feature of WebHook Project Parameters

f42970e

Addresses issue #26

netwolfuk mentioned this issue Dec 29, 2021

Issue 26 force resolve variables #193

Merged

netwolfuk added a commit that referenced this issue Dec 29, 2021

Issue 26 force resolve variables (#193)

0f8e285

* Add support for "Force Resolve" feature of WebHook Project Parameters Addresses issue #26 * Force resolve variables on the Build Finished events

netwolfuk closed this as completed Mar 11, 2023

netwolfuk added a commit that referenced this issue Apr 10, 2024

Issue 26 force resolve variables (#193)

050e81d

* Add support for "Force Resolve" feature of WebHook Project Parameters Addresses issue #26 * Force resolve variables on the Build Finished events

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Variable substitution in URL only works for finished builds #26

Variable substitution in URL only works for finished builds #26

jelco commented Mar 21, 2016

netwolfuk commented Mar 23, 2016

jelco commented Mar 23, 2016

netwolfuk commented Mar 23, 2016

jelco commented Mar 23, 2016

netwolfuk commented Mar 23, 2016

netwolfuk commented Mar 23, 2016

netwolfuk commented Mar 23, 2016

jelco commented Mar 23, 2016

netwolfuk commented Mar 23, 2016

netwolfuk commented May 21, 2016

netwolfuk commented Oct 17, 2021

jan0sch commented Oct 27, 2021

netwolfuk commented Oct 27, 2021 •

edited

Loading

NEdwards-Sharkmob commented Nov 7, 2021

netwolfuk commented Nov 7, 2021 •

edited

Loading

netwolfuk commented Nov 12, 2021

jan0sch commented Nov 16, 2021

netwolfuk commented Mar 11, 2023

Variable substitution in URL only works for finished builds #26

Variable substitution in URL only works for finished builds #26

Comments

jelco commented Mar 21, 2016

netwolfuk commented Mar 23, 2016

jelco commented Mar 23, 2016

netwolfuk commented Mar 23, 2016

jelco commented Mar 23, 2016

netwolfuk commented Mar 23, 2016

netwolfuk commented Mar 23, 2016

netwolfuk commented Mar 23, 2016

jelco commented Mar 23, 2016

netwolfuk commented Mar 23, 2016

netwolfuk commented May 21, 2016

netwolfuk commented Oct 17, 2021

jan0sch commented Oct 27, 2021

netwolfuk commented Oct 27, 2021 • edited Loading

NEdwards-Sharkmob commented Nov 7, 2021

netwolfuk commented Nov 7, 2021 • edited Loading

netwolfuk commented Nov 12, 2021

jan0sch commented Nov 16, 2021

netwolfuk commented Mar 11, 2023

netwolfuk commented Oct 27, 2021 •

edited

Loading

netwolfuk commented Nov 7, 2021 •

edited

Loading