Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buildbot Frontend not reachable #352

Closed
malikwirin opened this issue Dec 14, 2024 · 14 comments
Closed

Buildbot Frontend not reachable #352

malikwirin opened this issue Dec 14, 2024 · 14 comments

Comments

@malikwirin
Copy link

My configuration isn't based on the clan infra. But I realized that they are very similar.

{ config, lib, ... }:

let
  sopsCfg = config.sops;
  domain = config.networking.domain;
in
{
  services = {
    buildbot-nix.master = {
      enable = true;
      domain = "buildbot.${domain}";

      workersFile = sopsCfg.templates."buildbot-workers.json".path;
      admins = [ "malik" ];
      outputsPath = "/var/www/buildbot/nix-outputs";

      authBackend = "gitea";
      gitea = {
        enable = true;
        tokenFile = sopsCfg.secrets."codeberg-token".path;
        instanceUrl = "https://codeberg.org";
        oauthId = "redacted";
        oauthSecretFile = sopsCfg.secrets."cb-buildbot-secret".path;
        webhookSecretFile = sopsCfg.secrets."buildbot-webhook".path;
        topic = "build-with-buildbot";
      };
    };

    buildbot-master = {
      buildbotUrl = lib.mkForce "https://${config.services.buildbot-nix.master.domain}";
    };
    
    nginx.virtualHosts."${config.services.buildbot-nix.master.domain}" = {
      useACMEHost = domain;
      addSSL = true;
    };
  };
}

But I always see the following instead of the buildbot frontend when trying to.
Bildschirmfoto vom 2024-12-14 18-34-23
Could this be related to harmonia#481?

@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nginx-subdomains-are-all-messed-up/57418/1

@Mic92
Copy link
Member

Mic92 commented Dec 14, 2024

Check journalctl -u buildbot-master -f for logs.

@MagicRB
Copy link
Contributor

MagicRB commented Dec 14, 2024

For context, 503 means the gateway, i.e. the backend couldn't be reached. The Nginx config contains a proxy_pass directive and it will throw a 503 if the port is unreachable, specifically when it returns a Connection Refused, on a Connection Timeout you get a 504

@malikwirin
Copy link
Author

Check journalctl -u buildbot-master -f for logs.

Dez 16 10:05:10 hetzbox systemd[1]: Starting Buildbot Continuous Integration Server....
Dez 16 10:05:14 hetzbox buildbot-master-pre-start[488717]: updating existing installation
Dez 16 10:05:14 hetzbox buildbot-master-pre-start[488717]: not touching existing buildbot.tac
Dez 16 10:05:14 hetzbox buildbot-master-pre-start[488717]: creating buildbot.tac.new instead
Dez 16 10:05:14 hetzbox buildbot-master-pre-start[488717]: creating /var/lib/buildbot/master/master.cfg.sample
Dez 16 10:05:14 hetzbox buildbot-master-pre-start[488717]: creating database (postgresql://@/buildbot)
Dez 16 10:05:14 hetzbox buildbot-master-pre-start[488717]: buildmaster configured in /var/lib/buildbot/master
Dez 16 10:05:15 hetzbox systemd[1]: Started Buildbot Continuous Integration Server..
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:15+0100 [-] Loading /var/lib/buildbot/master/buildbot.tac...
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:18+0100 [-] Loaded.
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:18+0100 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 24.7.0 (/nix/store/zv1kaq7f1q20x62kbjv6pfjygw5jmwl6-python3-3.12.7/bin/python3.12 3.12.7) starting up.
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:18+0100 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:18+0100 [-] Starting BuildMaster -- buildbot.version: 4.1.0
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:18+0100 [-] Loading configuration from '/nix/store/9v7k84jr8sniapsdq0srgxknwn1024h0-master.cfg'
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:18+0100 [twisted.logger._global#warn] /nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/pydantic/_internal/_config.py:341: builtins.UserWarning: Valid config keys have changed in V2:
Dez 16 10:05:18 hetzbox twistd[488723]:         * 'fields' has been removed
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:18+0100 [-] while starting BuildMaster
Dez 16 10:05:18 hetzbox twistd[488723]:         Traceback (most recent call last):
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/twisted/internet/defer.py", line 1074, in _runCallbacks
Dez 16 10:05:18 hetzbox twistd[488723]:             current.result = callback(  # type: ignore[misc]
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/twisted/internet/defer.py", line 1960, in _gotResultInlineCallbacks
Dez 16 10:05:18 hetzbox twistd[488723]:             _inlineCallbacks(r, gen, status, context)
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/twisted/internet/defer.py", line 2010, in _inlineCallbacks
Dez 16 10:05:18 hetzbox twistd[488723]:             result = context.run(
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/twisted/python/failure.py", line 549, in throwExceptionIntoGenerator
Dez 16 10:05:18 hetzbox twistd[488723]:             return g.throw(self.value.with_traceback(self.tb))
Dez 16 10:05:18 hetzbox twistd[488723]:         --- <exception caught here> ---
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/buildbot/master.py", line 252, in startService
Dez 16 10:05:18 hetzbox twistd[488723]:             self.config = yield threads.deferToThreadPool(
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/twisted/python/threadpool.py", line 269, in inContext
Dez 16 10:05:18 hetzbox twistd[488723]:             result = inContext.theWork()  # type: ignore[attr-defined]
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/twisted/python/threadpool.py", line 285, in <lambda>
Dez 16 10:05:18 hetzbox twistd[488723]:             inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/twisted/python/context.py", line 117, in callWithContext
Dez 16 10:05:18 hetzbox twistd[488723]:             return self.currentContext().callWithContext(ctx, func, *args, **kw)
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/twisted/python/context.py", line 82, in callWithContext
Dez 16 10:05:18 hetzbox twistd[488723]:             return func(*args, **kw)
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/buildbot/config/master.py", line 133, in loadConfig
Dez 16 10:05:18 hetzbox twistd[488723]:             config = MasterConfig.loadFromDict(config_dict, filename)
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/buildbot/config/master.py", line 270, in loadFromDict
Dez 16 10:05:18 hetzbox twistd[488723]:             config.run_configurators(filename, config_dict)
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/buildbot/config/master.py", line 301, in run_configurators
Dez 16 10:05:18 hetzbox twistd[488723]:             interfaces.IConfigurator(configurator).configure(config_dict)
Dez 16 10:05:18 hetzbox twistd[488723]:           File "/nix/store/n5qg1m1d7jqcvg7da7vnrc3waqz7s8ny-python3-3.12.7-env/lib/python3.12/site-packages/buildbot_nix/__init__.py", line 1824, in configure
Dez 16 10:05:18 hetzbox twistd[488723]:             reload_builder = backend.create_reload_builder([worker_names[0]])
Dez 16 10:05:18 hetzbox twistd[488723]:         builtins.IndexError: list index out of range
Dez 16 10:05:18 hetzbox twistd[488723]:
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:18+0100 [-] BuildMaster startup failed
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:18+0100 [-] BuildMaster is stopped
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:18+0100 [twisted.internet.base#info] Main loop terminated.
Dez 16 10:05:18 hetzbox twistd[488723]: 2024-12-16T10:05:18+0100 [twisted.scripts._twistd_unix.UnixAppLogger#info] Server Shut Down.
Dez 16 10:05:19 hetzbox systemd[1]: buildbot-master.service: Deactivated successfully.
Dez 16 10:05:19 hetzbox systemd[1]: buildbot-master.service: Consumed 7.934s CPU time, 120.5M memory peak, 39.4M read from disk.

@MagicRB
Copy link
Contributor

MagicRB commented Dec 16, 2024

I'll take a look at this later today, now I know where the bug is probably

@Mic92
Copy link
Member

Mic92 commented Dec 16, 2024

Have you not configured any workers by chance?

@malikwirin
Copy link
Author

Have you not configured any workers by chance?

The Buildbot worker is also failing with reference to twisted.

Dez 16 10:23:09 hetzbox twistd[489059]:     endpoint = clientFromString(reactor, connection_string)
Dez 16 10:23:09 hetzbox twistd[489059]:   File "/nix/store/165y4y7b2s5nsyznqgiv4dnxb7nnndb8-python3-3.12.7-env/lib/python3.12/site-packages/twisted/internet/endpoints.py", line 2133, in clientFromString
Dez 16 10:23:09 hetzbox twistd[489059]:     plugin = _matchPluginToPrefix(
Dez 16 10:23:09 hetzbox twistd[489059]:   File "/nix/store/165y4y7b2s5nsyznqgiv4dnxb7nnndb8-python3-3.12.7-env/lib/python3.12/site-packages/twisted/internet/endpoints.py", line 1737, in _matchPluginToPrefix
Dez 16 10:23:09 hetzbox twistd[489059]:     raise ValueError(f"Unknown endpoint type: '{endpointType}'")
Dez 16 10:23:09 hetzbox twistd[489059]: builtins.ValueError: Unknown endpoint type: 'buildbot.$topdomain'
Dez 16 10:23:09 hetzbox twistd[489059]: Starting worker hetzbox-000
Dez 16 10:23:09 hetzbox twistd[489059]: Failed to load application: Unknown endpoint type: 'buildbot.$topdomain'
Dez 16 10:23:09 hetzbox systemd[1]: buildbot-worker.service: Main process exited, code=exited, status=1/FAILURE
Dez 16 10:23:09 hetzbox systemd[1]: buildbot-worker.service: Failed with result 'exit-code'.

I obfuscated the domain. But because its a free domain provider (for not much longer) I witnessed some instabilities in resolving the name.

The worker is configured like this:

{ config, ... }:

let
  masterCfg = config.services.buildbot-nix.master;
  masterPort = config.services.buildbot-master.port;
in
{
  services.buildbot-nix.worker = {
    enable = true;
    name = config.networking.hostName;
    masterUrl = "${masterCfg.domain}:${(toString masterPort)}";
    workerPasswordFile = config.sops.secrets."hetzbox-buildbot-worker-password".path;
  };
}

@Mic92
Copy link
Member

Mic92 commented Dec 16, 2024

I mean in your buildbot-workers.json file.

@malikwirin
Copy link
Author

I mean in your buildbot-workers.json file.

The buildbot-workers.json is configured as a sops template. Could users and groups be wrong?

{ inputs, config, ... }:

let
  hostName = config.networking.hostName;
  cfg = config.sops;
  bb-master-user = config.users.users.buildbot;
  bb-worker-user = config.users.users.buildbot-worker;
  buildbot-group = bb-master-user.group;
  bbMasterPort = config.services.buildbot-master.port;
in
{
  sops = {
    secrets = {
      codeberg-token = {
        group = buildbot-group;
      };
      cb-buildbot-secret = {
        owner = bb-master-user.name;
      };
      buildbot-webhook = {
        owner = bb-master-user.name;
      };
      hetzbox-buildbot-worker-password = {
        owner = bb-worker-user.name;
        group = buildbot-group;
      };
    };

    templates = {
      "buildbot-workers.json" = {
        content = ''
          [
            {
              "name": "${hostName}",
              "password": "${cfg.placeholder.hetzbox-buildbot-worker-password}",
              "host": "${hostName}",
              "port": ${(toString bbMasterPort)}
            }
          ]
        '';
        owner = bb-master-user.name;
      };
    };
  };
}

@Mic92
Copy link
Member

Mic92 commented Dec 16, 2024

It's called "pass" instead of password and there is no host or port in this file.
I added some model validation here: https://github.com/nix-community/buildbot-nix/pull/356/files
However this doesn't explain the error yet, you were seeing.
Ah you need to specify number of cores as well in the "cores" field.

@Mic92
Copy link
Member

Mic92 commented Dec 16, 2024

Use this as a template:

{ "name": "eve", "pass": "XXXXXXXXXXXXXXXXXXXX", "cores": 16 }

@malikwirin
Copy link
Author

#355 made the cause of the problem more clear

Dez 16 11:16:51 hetzbox twistd[508175]:           File "/nix/store/9d6mhasyhr070iihg35p5fa3hq7vzn2v-python3-3.12.7-env/lib/python3.12/site-packages/buildbot_nix/__init__.py", line 1780, in configure
Dez 16 11:16:51 hetzbox twistd[508175]:             raise BuildbotNixError(msg)
Dez 16 11:16:51 hetzbox twistd[508175]:         buildbot_nix.errors.BuildbotNixError: No workers configured in buildbot-nix-workers
Dez 16 11:16:51 hetzbox twistd[508175]:
Dez 16 11:16:51 hetzbox twistd[508175]: 2024-12-16T11:16:51+0100 [-] BuildMaster startup failed
Dez 16 11:16:51 hetzbox twistd[508175]: 2024-12-16T11:16:51+0100 [-] BuildMaster is stopped
Dez 16 11:16:51 hetzbox twistd[508175]: 2024-12-16T11:16:51+0100 [twisted.internet.base#info] Main loop terminated.
Dez 16 11:16:51 hetzbox twistd[508175]: 2024-12-16T11:16:51+0100 [twisted.scripts._twistd_unix.UnixAppLogger#info] Server Shut Down.
Dez 16 11:16:52 hetzbox systemd[1]: buildbot-master.service: Deactivated successfully.
Dez 16 11:16:52 hetzbox systemd[1]: buildbot-master.service: Consumed 8.627s CPU time, 145.4M memory peak, 63.6M read from disk.

@malikwirin
Copy link
Author

It's called "pass" instead of password and there is no host or port in this file. I added some model validation here: https://github.com/nix-community/buildbot-nix/pull/356/files However this doesn't explain the error yet, you were seeing. Ah you need to specify number of cores as well in the "cores" field.

Thanks a lot. My issue is resolved as the frontend is now reachable. I cannot remember where I got the wrong workers.json config from.

I am not really sure what cores means in this context. So i took half the number of available threads.

@Mic92
Copy link
Member

Mic92 commented Dec 16, 2024

It's called "pass" instead of password and there is no host or port in this file. I added some model validation here: https://github.com/nix-community/buildbot-nix/pull/356/files However this doesn't explain the error yet, you were seeing. Ah you need to specify number of cores as well in the "cores" field.

Thanks a lot. My issue is resolved as the frontend is now reachable. I cannot remember where I got the wrong workers.json config from.

I am not really sure what cores means in this context. So i took half the number of available threads.

It's number of concurrent build jobs effectively. If you allocate less, than you will see some errors in your buildbot worker because half the worker instances won't be able to connect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants