WEbHook removed after server updated

[anatoly-kryzhanovsky]

Expected Behavior

Configured webhooks should preserve after TeamCity updated

Current Behavior

After update all webhooks are removed

Steps to Reproduce (for bugs)

1. configure webhooks for several build configuration
2. update teamcity via web interface
3. view web hook for build configuration

Your Environment

• tcWebHooks Version: 1.1.374.403
• TeamCity Version: 2021.1+
• TeamCity server Operating System: Windows 2016 Server
• Are you using a WebHook Template?: No

[netwolfuk]

Hi @anatoly-kryzhanovsky I just did a test and installed a new TeamCity 2020.2.3 (in Docker).
I installed the tcWebHooks plugin and setup a webhook.
Then I updated TeamCity to 2021.2.2 (build 99660), and the webhook configuration is still there.

Can you take a look in the BuildServer data directory and see if the webhook is configured in the plugin-settings.xml for your project? For more details on how to view the file from within the TeamCity UI, see issue #189

Check the plugin-settings.xml.1 and other numbered files to see if they were there in previous versions.

[anatoly-kryzhanovsky]

ok, just now i update from 2021.2.2 (99660) to 2021.2.3 (99711)
i checked plugin-settings in pluginData for several projects - webhooks present before update and i can see them thought web interface
but after upgrade i can't set them in web interface, but they still present in plugin-settings.xml

may be i can provide more information? some logs?

UPD
i check several project and found very strange thing: for some of them webhooks are still displayed in web interface, but for some missed (but exists in xml file)

[netwolfuk]

tcWebHooks logs to TeamCity's server log. I think it's called teamcity-server.log.

You could have a look in the log for when TeamCity started up and you can see events in the log with the word webhook.

It should say that it's loading webhooks for project xyz. Does it have any errors there?

You could enable debug logging on TeamCity server and restart TeamCity to get more logging.

I'm not at the computer at the moment so I can't remember the exact log messages sorry.

[anatoly-kryzhanovsky]

i checked teamcity-server.log, i found related records, but all of them are info, not error

[netwolfuk]

ok, these are what I am expecting to see...

INFO -   jetbrains.buildServer.SERVER - WebHookProjectSettingsFactory: re-reading settings for project1

and if there are errors loading the config, it will be near those messages.

Can you please copy a section from plugin-settings.xml in a project where the webhooks are working, and also the same section from another plugin-settings.xml where they are not displayed in the UI?

Please make sure to remove the URL or anything else that's private. If you prefer, you can email me the information. My gmail address is the same username as on github.

[netwolfuk]

Another thing to check. If you edit webhooks in the UI, do the changes appear in the plugin-settings.xml file for that project?

[anatoly-kryzhanovsky]

this is content for worked hook:

<?xml version="1.0" encoding="UTF-8"?>
<settings>
  <webhooks enabled="true">
    <webhook url="https://<redacted>" enabled="true" format="jsontemplate" template="slack.com">
      <states>
        <state type="buildBroken" enabled="false" />
        <state type="changesLoaded" enabled="false" />
        <state type="buildFailed" enabled="true" />
        <state type="buildStarted" enabled="true" />
        <state type="buildFixed" enabled="false" />
        <state type="buildFinished" enabled="true" />
        <state type="responsibilityChanged" enabled="false" />
        <state type="buildInterrupted" enabled="true" />
        <state type="beforeBuildFinish" enabled="false" />
        <state type="buildSuccessful" enabled="true" />
      </states>
      <build-types enabled-for-all="true" enabled-for-subprojects="true" />
    </webhook>
  </webhooks>
</settings>

and this is example for configuration that missed from ui:

<?xml version="1.0" encoding="UTF-8"?>
<settings>
  <webhooks enabled="true">
    <webhook url="https://<radacted>" enabled="true" format="jsontemplate" template="slack.com-compact">
      <states>
        <state type="changesLoaded" enabled="false" />
        <state type="beforeBuildFinish" enabled="false" />
        <state type="responsibilityChanged" enabled="false" />
        <state type="buildInterrupted" enabled="false" />
        <state type="buildFailed" enabled="true" />
        <state type="buildBroken" enabled="false" />
        <state type="buildStarted" enabled="true" />
        <state type="buildFixed" enabled="false" />
        <state type="buildFinished" enabled="true" />
        <state type="buildSuccessful" enabled="true" />
      </states>
      <build-types enabled-for-all="false" enabled-for-subprojects="false">
        <build-type id="bt73" />
        <build-type id="bt60" />
        <build-type id="bt61" />
      </build-types>
    </webhook>
  </webhooks>
</settings>

hm.. now i see some pattern - hooks that preserved after upgrade configured for project whole, and hooks that are missed configured for build type.
and now i see that there are three record in settings, but i have only one hook...
may be id of build type regenerated on server restart?

[anatoly-kryzhanovsky]

for additional information: we are using configuration as code based on Kotlin DSL

[netwolfuk]

ah! That's really interesting. Thank you so much for this detailed information. I will do some more investigation over the weekend.

Internally, there is a cache, and I wonder if it's not loading into the cache because it can't resolve the bt id. I will try to reproduce the issue you have raised, and look into fixing it.

Thanks so much for taking the time to raise this. I'll get back to you asap.

[netwolfuk]

for additional information: we are using configuration as code based on Kotlin DSL

ok, that might be a factor too. I'll make sure to test using this style of config. Thank you!

[netwolfuk]

I can reproduce this issue, by changing bt1 to bt21 (non-existing build) and restarting teamcity.

Thanks, I will see if I can load the configuration in a more robust way, and show a warning if they don't match up as expected. I suspect these bt numbers did change from the kotlin DSL.

[anatoly-kryzhanovsky]

thank you, will be waiting for news!

[netwolfuk]

I should really be using the id of the buildType. The problem is that these IDs can be changed by the user, and then I have to listen for these change events and then find all references and update them.

These bt numbers are not supposed to change, so it seemed safer to use them. However this bug would indicate that they can in fact change. Or maybe a build has been deleted in TeamCity? Is that possible?

[anatoly-kryzhanovsky]

no, i did not change any build configuration. this is happened on server restart. may be i need to define explicit id in my dsl file... i need to read docs about that

[netwolfuk]

Would you consider running 1.2 alpha version?

Looking at the 1.1 code, it's going to be a lot of work to fix this and it will all be thrown away because 1.2 manages configuration of webhooks in a very different way.

[netwolfuk]

Btw, it's not fixed in 1.2, but should be a lot easier to fix on that version.

[netwolfuk]

Hi @anatoly-kryzhanovsky I've not yet figured out why the BT ids have changed in your teamcity, but I have raised a support ticket with TeamCity to understand how that can happen.

If I understand your issue correctly, the webhook is configured but the BT numbers no longer match. These "build specific webhooks" are really just the same as project webhooks, but with only specific builds selected.

It's therefore possible to edit the webhooks for your project and select the correct builds.

For me, I access /webhooks/index.html?projectId=TcPlugins
I get a list of three webhooks as per the following screenshot.

The first webhook configuration appears to be assigned to one build type, but when I click on it, the BT number does not match, so no builds are selected.

If I reselect the correct build(s), and save the webhook, it will appear again on the webhooks tab on the selected build.

Until I can confirm why the build IDs have changed, this will allow you to configure your webhooks to work again.

[anatoly-kryzhanovsky]

thank you for your investigation.
yes, the reason of issue is look correctly and i think that it is teamcity issue (or may be feature?)

so, i think we need to wait response from JetBrains. can you provide issue link?

[netwolfuk]

Sure. The ticket is https://youtrack.jetbrains.com/issue/TW-75142

[netwolfuk]

I am fairly confident that if this ever happens again, it will be resolved by https://github.com/tcplugins/tcWebHooks/releases/tag/v2.0.1