TeamCity server cpu and memory spike after enabling tcwebhooks about tcwebhooks

[dom747]

We have our own ami built using packer

[arthursmel]

Since the plugin was disabled, we no longer can access the webhook configurations for each project.
We were using the following template:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<webhook-templates>
    <webhook-template id="tc-pr-verify-webhook" enabled="true" rank="100" format="jsonTemplate">
        <template-description>PR Verify Webhook</template-description>
        <template-tool-tip></template-tool-tip>
        <preferred-date-format></preferred-date-format>
        <templates max-id="0">
            <template id="0">
                <template-text use-for-branch-template="true">{
    'event_key': '${notifyType}',
    'build_id': '${buildId}',
    'build_tags': '${buildTags}',
    'build_text': '${text}',
    'build_url': '${buildStatusUrl}'
}</template-text>
                <branch-template-text></branch-template-text>
                <states>
                    <state type="buildInterrupted" enabled="true"/>
                    <state type="buildSuccessful" enabled="true"/>
                    <state type="buildFailed" enabled="true"/>
                    <state type="buildFixed" enabled="true"/>
                    <state type="buildBroken" enabled="true"/>
                </states>
            </template>
        </templates>
    </webhook-template>
</webhook-templates>

[netwolfuk]

Hi @dom747 . Thank you for the detailed bug report.

Have you previously been running any of the tcWebHooks pre-release 1.2.0 versions (eg, Alpha, or Release Candidate), or were you previously running 1.1.x ?

I am trying to determine if the change is because of the AMIs, or because of a recent tcWebHooks upgrade. The ExtraParameters logic has changed in 1.2.0 and above, but has been in alpha a year at least. If you happened to try any of those versions that would help me pinpoint the issue.

If you're running Centos, have you changed any of the SELinux configurations? By default SELinux is enabled. I'm not sure that will matter in this case. Just interesting to know.

I can't seem to find ticket 4905074 on the jertbrains youtrack instance. I will email their support team and ask to get access to your support ticket. Is that ok?

The webhook XML file is located on the server in BuildServer/config/projects/yourProject/pluginData/plugin-settings.xml

[netwolfuk]

Executing webhooks in threads was added in 1.2.0 also.

You could try disabling threading in tcWebHooks by creating a <webhooks> section in your BuildServer/config/main-config.xml

It looks like this...

<?xml version="1.0" encoding="UTF-8"?>
<server>
  <webhooks useThreadedExecutor="false">
  </webhooks>
</server>

[dom747]

Thanks for the reply. We have been using 1.2.0 for a while, I believe. We have not changed any SELinux configurations. Yes it's ok for you to get information from them.
Thanks.
I could try your suggestion. The challenge is that it took 2 hours to get the server back to a working state after my last attempt to enable the plugin. We have 1000 AWS agents and they all need an update, and the server got stuck until I restarted 3 times.
My jetbrains ticket was a support ticket , not on their YouTrack.

[netwolfuk]

Wow, that's awesome!. I have not tested it on 1000 agents. At that scale, it could be a concurrency issue with the ExtraParameters object. I am trying to figure out how this could be an issue based on the number of agents, but I suspect it's that my code is consuming all the threads on the ThreadPool Executor that TeamCity allows plugins to access.

If it's the same thread pool, maybe my code is starving it for all the other threads, including the threads that communicate with the agents.

I feel like the root cause could be something with the change to the lastest centos. I will create some VMs and try to replicate the issue. My personal budget won't scale to 1000 agents though :-).

Are you using a public AMI? If so, I could use the same ones as you for testing.

[netwolfuk]

I've reached out to a couple of mates at TeamCity, but don't have a contact for "support".
Could you please email me (my same username as here but at gmail) with a TeamCity support email address?

Also, if you can share you packer script (remove anything that's private), or at least the base AMI that packer is using. Perhaps via email too, rather than on this public forum. Thanks!

[netwolfuk]

Release 2.0.0. Release Candidate 1 released to try to resolve the issues associated with this issue