github-bot broken? about build

[richardlau]

Is the machine still in sync with its playbook? If it's not, I suggest we remove the Node.js installed via apt and instead just download it from nodejs.org and extract it to /usr/local

I've downloaded Node.js 14.21.3 onto the machine and extracted it to /usr/local. Had to edit /lib/systemd/system/github-bot.service so that it used /usr/local/bin/node instead of /usr/bin/node. It looks like the bot has now been able to start:

root@infra-rackspace-debian8-x64-1:/home/iojs# systemctl status github-bot
● github-bot.service - github-bot
   Loaded: loaded (/lib/systemd/system/github-bot.service; enabled)
   Active: active (running) since Wed 2024-05-15 17:52:29 UTC; 1s ago
 Main PID: 19176 (node)
   CGroup: /system.slice/github-bot.service
           └─19176 /usr/local/bin/node server.js

May 15 17:52:29 infra-rackspace-debian8-x64-1 systemd[1]: Started github-bot.
May 15 17:52:29 infra-rackspace-debian8-x64-1 node[19176]: {"name":"bot","hostname":"infra-rackspace-debian8-x64-1","pid":19176,"level":30,"msg":"Loading: ./scripts/ping.js","time":"2024-05-15T1...591Z","v":0}
May 15 17:52:29 infra-rackspace-debian8-x64-1 node[19176]: {"name":"bot","hostname":"infra-rackspace-debian8-x64-1","pid":19176,"level":30,"msg":"Loading: ./scripts/node-ping-owners.js","time":"...595Z","v":0}
May 15 17:52:29 infra-rackspace-debian8-x64-1 node[19176]: {"name":"bot","hostname":"infra-rackspace-debian8-x64-1","pid":19176,"level":30,"msg":"Loading: ./scripts/jenkins-status.js","time":"20...687Z","v":0}
May 15 17:52:29 infra-rackspace-debian8-x64-1 node[19176]: {"name":"bot","hostname":"infra-rackspace-debian8-x64-1","pid":19176,"level":30,"msg":"Loading: ./scripts/event-relay.js","time":"2024-...688Z","v":0}
May 15 17:52:29 infra-rackspace-debian8-x64-1 node[19176]: {"name":"bot","hostname":"infra-rackspace-debian8-x64-1","pid":19176,"level":30,"msg":"Listening on port 3333","time":"2024-05-15T17:52...693Z","v":0}
Hint: Some lines were ellipsized, use -l to show in full.
root@infra-rackspace-debian8-x64-1:/home/iojs#

[targos]

nodejs/github-bot@c5ba723

Landed 18 hours ago.

[Trott]

nodejs/github-bot#511 for a temporary workaround

[Trott]

@richardlau Should we land nodejs/github-bot#514 and keep an eye on things?

[richardlau]

post-build-status-update at first glance looks green, but clicking on the jobs and looking at the "Pipeline steps", e.g. https://ci.nodejs.org/job/post-build-status-update/2261443/flowGraphTable/, shows that the "httpRequest" step failed.

The console output can't find the build log, e.g.

java.io.FileNotFoundException: /var/lib/jenkins/jobs/post-build-status-update/builds/2261443/log (No such file or directory)
	at java.base/java.io.RandomAccessFile.open0(Native Method)
...

I'll see if I can find it on the Jenkins server (I think they get compressed there).

[richardlau]

On ci:

root@infra-digitalocean-ubuntu14-x64-1:~# ls -al /var/lib/jenkins/jobs/post-build-status-update/builds/2261443
total 976
drwxr-xr-x   3 jenkins jenkins   4096 May 14 07:56 .
drwxr-xr-x 102 jenkins jenkins 933888 May 14 08:37 ..
-rw-r--r--   1 jenkins jenkins  35675 May 14 07:56 build.xml
-rw-r--r--   1 jenkins jenkins      0 May 14 07:56 changelog1986113076434187246.xml
-rw-r--r--   1 jenkins jenkins   6021 May 14 07:56 log.gz
-rw-r--r--   1 jenkins jenkins    102 May 14 07:56 log-index
drwxr-xr-x   2 jenkins jenkins   4096 May 14 07:56 workflow
root@infra-digitalocean-ubuntu14-x64-1:~#

From the uncompressed log:

[2024-05-14T11:56:22.706Z] Fetching upstream changes from https://github.com/nodejs/build.git
[2024-05-14T11:56:22.706Z]  > git --version # timeout=10
[2024-05-14T11:56:22.711Z]  > git --version # 'git version 2.34.1'
[2024-05-14T11:56:22.711Z]  > git fetch --tags --force --progress -- https://github.com/nodejs/build.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2024-05-14T11:56:23.047Z]  > git rev-parse origin/main^{commit} # timeout=10
[2024-05-14T11:56:23.258Z]  > git config core.sparsecheckout # timeout=10
[2024-05-14T11:56:23.264Z]  > git read-tree -mu HEAD # timeout=10
[2024-05-14T11:56:23.272Z]  > git checkout -f e2397a4d6638c7c644464be8ae219f8580ee6100 # timeout=10
[2024-05-14T11:56:23.963Z] HttpMethod: POST
[2024-05-14T11:56:23.963Z] URL: http://github-bot.nodejs.org:3333/node/jenkins/end
[2024-05-14T11:56:23.964Z] Content-Type: application/json; charset=UTF-8
[2024-05-14T11:56:23.995Z] Sending request to url: http://github-bot.nodejs.org:3333/node/jenkins/end
[2024-05-14T11:56:24.193Z] Treating class org.apache.http.conn.HttpHostConnectException(Connect to github-bot.nodejs.org:3333 [github-bot.nodejs.org/23.253.100.79] failed: Connection refused) as 408 Request Timeout

[richardlau]

The bot isn't running -- on gh-bot:

root@infra-rackspace-debian8-x64-1:~# systemctl status github-bot
● github-bot.service - github-bot
   Loaded: loaded (/lib/systemd/system/github-bot.service; enabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2024-05-14 13:00:13 UTC; 8s ago
  Process: 29335 ExecStart=/usr/bin/node server.js (code=exited, status=1/FAILURE)
 Main PID: 29335 (code=exited, status=1/FAILURE)

May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: code: 'MODULE_NOT_FOUND',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: requireStack: [
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/glob.js',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/index.js',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/server.js'
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: ]
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: }
May 14 13:00:13 infra-rackspace-debian8-x64-1 systemd[1]: github-bot.service: main process exited, code=exited, status=1/FAILURE
May 14 13:00:13 infra-rackspace-debian8-x64-1 systemd[1]: Unit github-bot.service entered failed state.
root@infra-rackspace-debian8-x64-1:~#

[richardlau]

From journalctl

-- Logs begin at Tue 2024-05-14 10:38:04 UTC, end at Tue 2024-05-14 13:02:12 UTC. --
May 14 10:38:04 infra-rackspace-debian8-x64-1 systemd[1]: Stopping github-bot...
May 14 10:38:04 infra-rackspace-debian8-x64-1 systemd[1]: Starting github-bot...
May 14 10:38:04 infra-rackspace-debian8-x64-1 systemd[1]: Started github-bot.
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: internal/modules/cjs/loader.js:883
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: throw err;
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: ^
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: Error: Cannot find module 'node:path'
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: Require stack:
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/node_modules/glob/dist/commonjs/glob.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/node_modules/glob/dist/commonjs/index.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/server.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Function.Module._load (internal/modules/cjs/loader.js:725:27)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module.require (internal/modules/cjs/loader.js:952:19)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at require (internal/modules/cjs/helpers.js:88:18)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Object.<anonymous> (/home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js:28:21)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module._compile (internal/modules/cjs/loader.js:1063:30)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module.load (internal/modules/cjs/loader.js:928:32)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Function.Module._load (internal/modules/cjs/loader.js:769:14)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module.require (internal/modules/cjs/loader.js:952:19) {
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: code: 'MODULE_NOT_FOUND',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: requireStack: [
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/glob.js',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/index.js',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/server.js'
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: ]
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: }
May 14 10:38:05 infra-rackspace-debian8-x64-1 systemd[1]: github-bot.service: main process exited, code=exited, status=1/FAILURE
May 14 10:38:05 infra-rackspace-debian8-x64-1 systemd[1]: Unit github-bot.service entered failed state.

which is #3214 and the inability to update Node.js there until we update the OS all over again.

[targos]

Can't we update Node.js to 14.21.3 (the last 14.x version), or at least 14.18? isaacs/node-glob@57f5551

[Trott]

Oof, sorry I did this with that glob update. That sure seemed safe.... 🥴

[Trott]

If upgrading to Node.js 14.18 or later is trouble, and even if it's not, a quick fix would be to revert the glob update.

[Trott]

If upgrading to Node.js 14.18 or later is trouble, and even if it's not, a quick fix would be to revert the glob update.

Oops, probably not because the bot will still install the latest glob and this was introduced in a patch release. I'll open a PR to pin glob instead, which isn't great, but will fix things if the Node.js upgrade isn't something that can happen quickly.

[richardlau]

FWIW apt update offers no update or newer verson of Node.js 14.

[Trott]

Hmm, pinning glob doesn't seem to have fixed it. I'm not sure how to look at the log file to confirm that glob is still the problem, but I'll reopen this.

[targos]

Is the machine still in sync with its playbook? If it's not, I suggest we remove the Node.js installed via apt and instead just download it from nodejs.org and extract it to /usr/local

[richardlau]

hmm. bot is still failing to start.

It seems like the deployment still has path-scurry 1.11.1 installed.

# npm ls
[email protected] /home/iojs/github-bot
├─┬ @octokit/[email protected]
│ ├─┬ @octokit/[email protected]
│ │ ├─┬ @octokit/[email protected]
│ │ │ └── @octokit/[email protected] deduped
│ │ ├─┬ @octokit/[email protected]
│ │ │ ├── @octokit/[email protected] deduped
│ │ │ ├── @octokit/[email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├─┬ @octokit/[email protected]
│ │ │ ├─┬ @octokit/[email protected]
│ │ │ │ ├── @octokit/[email protected] deduped
│ │ │ │ ├── [email protected] deduped
│ │ │ │ └── [email protected] deduped
│ │ │ ├── @octokit/[email protected] deduped
│ │ │ ├── @octokit/[email protected] deduped
│ │ │ ├── [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ └─┬ [email protected]
│ │ │ │   ├── [email protected]
│ │ │ │   └── [email protected]
│ │ │ └── [email protected] deduped
│ │ ├─┬ @octokit/[email protected]
│ │ │ ├── @octokit/[email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └─┬ [email protected]
│ │ │   └── [email protected]
│ │ ├─┬ @octokit/[email protected]
│ │ │ └── @octokit/[email protected]
│ │ ├── [email protected]
│ │ └── [email protected]
│ ├─┬ @octokit/[email protected]
│ │ ├── @octokit/[email protected]
│ │ └── @octokit/[email protected] deduped
│ ├── @octokit/[email protected]
│ └─┬ @octokit/[email protected]
│   ├── @octokit/[email protected] deduped
│   └── [email protected]
├─┬ [email protected]
│ └── [email protected]
├─┬ [email protected]
│ └── [email protected]
├─┬ [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ └─┬ [email protected]
│ │   ├─┬ [email protected]
│ │   │ ├── [email protected]
│ │   │ └── [email protected] deduped
│ │   ├─┬ [email protected]
│ │   │ ├── [email protected] deduped
│ │   │ ├─┬ [email protected]
│ │   │ │ └── [email protected] deduped
│ │   │ ├── [email protected]
│ │   │ └── [email protected]
│ │   └── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ └─┬ [email protected]
│ │   └── [email protected]
│ └── [email protected]
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected]
│ │ └─┬ [email protected]
│ │   └─┬ [email protected]
│ │     ├─┬ [email protected]
│ │     │ ├── [email protected] deduped
│ │     │ └── [email protected] deduped
│ │     ├── [email protected] deduped
│ │     ├─┬ [email protected]
│ │     │ └─┬ [email protected]
│ │     │   ├── [email protected] deduped
│ │     │   └── [email protected]
│ │     ├── [email protected] deduped
│ │     └── [email protected]
│ └── [email protected]
├─┬ [email protected]
│ ├─┬ @types/[email protected]
│ │ └── @types/[email protected]
│ ├─┬ @types/[email protected]
│ │ ├─┬ @types/[email protected]
│ │ │ ├─┬ @types/[email protected]
│ │ │ │ └── @types/[email protected] deduped
│ │ │ └── @types/[email protected] deduped
│ │ ├─┬ @types/[email protected]
│ │ │ ├── @types/[email protected] deduped
│ │ │ ├── @types/[email protected] deduped
│ │ │ └── @types/[email protected]
│ │ ├── @types/[email protected]
│ │ └─┬ @types/[email protected]
│ │   ├── @types/[email protected]
│ │   └── @types/[email protected] deduped
│ └── [email protected]
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ └─┬ [email protected]
│ │   └── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ ├── [email protected]
│ └─┬ [email protected]
│   └─┬ [email protected]
│     └─┬ [email protected]
│       └── [email protected]
├─┬ [email protected]
│ └── [email protected]
├── [email protected]
├── [email protected]
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├─┬ [email protected]
│ │ └── [email protected] deduped
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ └── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ └── [email protected]
└─┬ [email protected]
  ├─┬ [email protected]
  │ ├── [email protected] deduped
  │ └── [email protected]
  ├─┬ [email protected]
  │ ├─┬ @isaacs/[email protected]
  │ │ ├─┬ [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ └── [email protected] deduped
  │ │ ├─┬ string-width-cjs@npm:[email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ └─┬ [email protected]
  │ │ │   └── [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ └── [email protected]
  │ │ ├─┬ strip-ansi-cjs@npm:[email protected]
  │ │ │ └── [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected] deduped
  │ │ │ └── [email protected] deduped
  │ │ └─┬ wrap-ansi-cjs@npm:[email protected]
  │ │   ├─┬ [email protected]
  │ │   │ └─┬ [email protected]
  │ │   │   └── [email protected]
  │ │   ├─┬ [email protected]
  │ │   │ ├── [email protected]
  │ │   │ ├── [email protected] deduped
  │ │   │ └── [email protected] deduped
  │ │   └─┬ [email protected]
  │ │     └── [email protected]
  │ └── @pkgjs/[email protected]
  ├─┬ [email protected]
  │ └─┬ [email protected]
  │   └── [email protected]
  ├── [email protected]
  └─┬ [email protected]
    ├── [email protected]
    └── [email protected]

Maybe trying to get a manual Node.js 14 installed as a temporary solution would be worth pursuing.

[richardlau]

httpRequest step in the pipeline (see #3720 (comment)) is passing now and I can see status checks being posted back to PRs.