It is required to finally determine the value of the parameter net.core.somaxconn about frameworkbenchmarks

[NateBrady23]

@michaelhixson:

net.ipv4.tcp_max_syn_backlog = 65535
net.core.somaxconn = 65535
net.ipv4.tcp_tw_reuse = 1

Putting it into the wiki

[michaelhixson]

The net.ipv4.tcp_tw_reuse = 1 is a change from what we had in 2020 at least. See #4092 (comment)

At that time, we noticed that changing it to 1 would "solve" issues for a few frameworks, but we decided not to change it because the real problem was those frameworks not reusing their own connections. If we want to go back on that decision, it's fine, but setting this to 1 would be something brand new for us.

[NateBrady23]

Those were the settings in vagrant. We never adjusted those settings in Citrine. net.core.somaxconn has always been 4096 in this environment as far as I'm aware.

Though I'm happy to leave this open to have a discussion about changing these.

[remittor]

@nbrady-techempower

We never adjusted those settings in Citrine. net.core.somaxconn has always been 4096

This is unlikely, because until 2023, projects using uWSGI were correctly launched (they had into config listen = 65535).
Then something happened and this appeared in the run logs: run.log

On the CI system, everything always started without errors.

[remittor]

It was possible to find out more precisely when everything broke.

2022-01-22: OK https://tfb-status.techempower.com/unzip/results.2022-01-27-10-45-01-139.zip/results/20220122165234/uwsgi/run

2022-01-31: Fail https://tfb-status.techempower.com/unzip/results.2022-02-05-08-28-48-272.zip/results/20220131174100/uwsgi/run

There is a possibility that this is due to this event: #7078

[remittor]

I even managed to find out the exact values of this parameter.

2022-01-22: https://tfb-status.techempower.com/unzip/results.2022-01-27-10-45-01-139.zip/results/20220122165234/racket/run

racket: 7:M 26 Jan 11:04:56.441 # WARNING: The TCP backlog setting of 65535 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 16384.
racket: 7:M 26 Jan 11:04:56.441 # Server started, Redis version 3.2.6

2022-01-31: https://tfb-status.techempower.com/unzip/results.2022-02-05-08-28-48-272.zip/results/20220131174100/racket/run

racket: 8:M 11 Apr 11:06:39.480 # WARNING: The TCP backlog setting of 65535 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 4096.
racket: 8:M 11 Apr 11:06:39.480 # Server started, Redis version 3.2.6

That is, the value has changed from 16384 to 4096.

---

And here is the proof that the CI system have net/core/somaxconn = 65535
https://github.com/TechEmpower/FrameworkBenchmarks/actions/runs/4547822068/jobs/8018189920#step:9:652

racket: 8:M 28 Mar 21:44:29.038 # Server started, Redis version 3.2.6

There is no warning message.

[NateBrady23]

It's possible the machines were commissioned with those set and that first upgrade reverted config values. I'll let this run finish and we'll see about updating them.

Edit: I'm not in the office today but I should be able to take care of this tomorrow (Friday) and we'll cut this current run short.

[NateBrady23]

Unfortunately couldn't get in to get to this today, but coming first thing Monday!

[NateBrady23]

Citrine is updated

[michaelhixson]

@nbrady-techempower What new values did you set?

[remittor]

...we decided not to change it because the real problem was those frameworks not reusing their own connections.

780 * 60 sec = 46800 sec = 780 min = 13 hours
13 / 160 = 0.08 => 8%

Average execution time for 6 tests: 12 min
60 / (12*60) = 0.08 => 8%

And here you need to choose an option:

1. greater realism
2. testing acceleration

[remittor]

Citrine is updated

@nbrady-techempower , too early you closed this issue.
After the imi-workerman test, everything completely broke down.

    "ihp": "20230427061559", 
    "imi": "20230427063439", 
    "imi-swoole-mysql-raw": "20230427064707", 
    "imi-swoole-pgsql": "20230427065927", 
    "imi-swoole-pgsql-raw": "20230427071149", 
    "imi-workerman": "ERROR: Problem starting imi-workerman", 
    "imi-workerman-mysql-raw": "ERROR: Problem starting imi-workerman-mysql-raw", 
    "imi-workerman-pgsql": "ERROR: Problem starting imi-workerman-pgsql", 
    "imi-workerman-pgsql-raw": "ERROR: Problem starting imi-workerman-pgsql-raw", 
    "inverno": "ERROR: Problem starting inverno", 
    "inverno-postgres": "ERROR: Problem starting inverno-postgres", 
    "isocket-nio": "ERROR: Problem starting isocket-nio", 
    "japronto": "ERROR: Problem starting japronto", 
    "javalin": "ERROR: Problem starting javalin", 
    "javalin-postgres": "ERROR: Problem starting javalin-postgres", 
    "jawn": "20230427074702", 
    "jersey": "20230427075034", 
    "jester": "20230427075707", 
    "jetty": "20230427080238", 
    "jetty-servlet": "20230427080809",

It's too bad that the run.log can't be viewed now.

UPD:
My guess is that this may be caused by too many threads/processes being started.
Look this:

FrameworkBenchmarks/frameworks/PHP/imi/config/config.php

Lines 52 to 55 in 24c6e25

	'configs' => [
	// 支持设置 Workerman 参数
	'count' => (int) shell_exec('nproc') * 4,
	],

14(cores) * 2(sockets) * 4 = 112 proof link
Although I can not understand how this can affect the operation of the network part.

[remittor]

here is another place where everything broke:

    "ronykit": "20230429155022", 
    "ronykit-prefork": "20230429155612", 
    "routerling": "20230429160913", 
    "sailsjs": "20230429161412", 
    "salvo": "ERROR: Problem starting salvo", 
    "salvo-diesel": "ERROR: Problem starting salvo-diesel", 
    "salvo-moka": "ERROR: Problem starting salvo-moka", 
    "salvo-mongo": "ERROR: Problem starting salvo-mongo", 
    "salvo-mongo-raw": "ERROR: Problem starting salvo-mongo-raw", 
    "salvo-pg": "ERROR: Problem starting salvo-pg", 
    "salvo-pg-pool": "ERROR: Problem starting salvo-pg-pool", 
    "salvo-sqlx": "ERROR: Problem starting salvo-sqlx", 
    "sanic": "ERROR: Framework is not accepting requests from client machine", 
    "scalene": "20230429165232", 
    "scorper": "20230429165834", 
    "servant": "ERROR: Problem starting servant", 
    "servant-psql-simple": "ERROR: Problem starting servant-psql-simple", 
    "servicestack-v6": "20230429170709", 
    "servicetalk": "20230429171318", 
    "servlet": "20230429171901", 
    "servlet-lib-native": "ERROR: Problem starting servlet-lib-native", 
    "servlet-mysql": "20230429173136", 
    "servlet-postgresql": "20230429174439", 
    "servlet3": "20230429175027",

[remittor]

build imi-workerman:

imi-workerman: No releases available for package "pecl.php.net/event"
imi-workerman: install failed
imi-workerman: Removing intermediate container b5c9940ba4a5
imi-workerman: Docker build failed; terminating
imi-workerman: Traceback (most recent call last):
imi-workerman:   File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
imi-workerman:     raise Exception(token['errorDetail']['message'])
imi-workerman: Exception: The command '/bin/sh -c pecl install event &&     echo "extension=event.so" > /usr/local/etc/php/conf.d/event.ini' returned a non-zero code: 1 

build imi-workerman-mysql-raw

imi-workerman-mysql-raw: Step 1/17 : FROM php:8.2-cli
imi-workerman-mysql-raw: Docker build failed; terminating
imi-workerman-mysql-raw: Traceback (most recent call last):
imi-workerman-mysql-raw:   File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
imi-workerman-mysql-raw:     raise Exception(token['errorDetail']['message'])
imi-workerman-mysql-raw: Exception: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
imi-workerman-mysql-raw: Build time: 15s 

build inverno:

inverno: Step 1/7 : FROM maven:3.8.2-openjdk-16 as maven
inverno: Docker build failed; terminating
inverno: Traceback (most recent call last):
inverno:   File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
inverno:     raise Exception(token['errorDetail']['message'])
inverno: Exception: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
inverno: Build time: 15s 

build japronto:

japronto: Step 1/6 : FROM python:3.8.12
japronto: Docker build failed; terminating
japronto: Traceback (most recent call last):
japronto:   File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
japronto:     raise Exception(token['errorDetail']['message'])
japronto: Exception: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
japronto: Build time: 15s 

build salvo:

salvo: Step 5/7 : RUN cargo build --release
salvo:  ---> Running in 992b0ce6a390
salvo:     Updating git repository `https://github.com/salvo-rs/salvo.git`
salvo:     Updating crates.io index
salvo: error: failed to select a version for the requirement `salvo = "^0.38"`
salvo: candidate versions found which didn't match: 0.39.1, 0.37.9, 0.37.8, ...
salvo: location searched: crates.io index
salvo: required by package `salvo_test v0.15.0 (/salvo)`
salvo: Removing intermediate container 992b0ce6a390
salvo: Docker build failed; terminating
salvo: Traceback (most recent call last):
salvo:   File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
salvo:     raise Exception(token['errorDetail']['message'])
salvo: Exception: The command '/bin/sh -c cargo build --release' returned a non-zero code: 101
salvo: Build time: 1m 20s 

Сreating containers sometimes ends with an error.

Stat for docker errors:
results.2023-02-05-04-03-06-716.zip : 31
results.2023-04-06-08-48-50-340.zip : 6
results.2023-04-19-19-51-19-699.zip : 6
results.2023-04-30-18-31-55-286.zip : 36

Stat for log with text net/http: request canceled while waiting for connection (:
results.2023-02-05-04-03-06-716.zip : 1
results.2023-04-06-08-48-50-340.zip : 0
results.2023-04-19-19-51-19-699.zip : 0
results.2023-04-30-18-31-55-286.zip : 10

I assume that resource docker.io was unavailable for some time on April 27th.

[remittor]

@nbrady-techempower
Looks like I was worrying for nothing. Probably on April 27, just the docker server was unavailable for some time.

[NateBrady23]

@remittor That's good news! Thanks for staying on top of it.

[remittor]

@nbrady-techempower

Citrine is updated

Something new settings are not applied.

Look this: https://tfb-status.techempower.com/unzip/results.2023-05-26-18-27-11-854.zip/results/20230520161617/racket/run

racket: 8:M 25 May 2023 09:46:44.997 # WARNING: The TCP backlog setting of 65535 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 4096.
racket: 8:M 25 May 2023 09:46:44.997 # Server initialized
racket: 8:M 25 May 2023 09:46:44.997 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

[NateBrady23]

Wow, my mistake! It's set permanently now.