Coder Social home page Coder Social logo

Comments (18)

NateBrady23 avatar NateBrady23 commented on June 16, 2024 1

@michaelhixson:

net.ipv4.tcp_max_syn_backlog = 65535
net.core.somaxconn = 65535
net.ipv4.tcp_tw_reuse = 1

Putting it into the wiki

from frameworkbenchmarks.

michaelhixson avatar michaelhixson commented on June 16, 2024 1

The net.ipv4.tcp_tw_reuse = 1 is a change from what we had in 2020 at least. See #4092 (comment)

At that time, we noticed that changing it to 1 would "solve" issues for a few frameworks, but we decided not to change it because the real problem was those frameworks not reusing their own connections. If we want to go back on that decision, it's fine, but setting this to 1 would be something brand new for us.

from frameworkbenchmarks.

NateBrady23 avatar NateBrady23 commented on June 16, 2024

Those were the settings in vagrant. We never adjusted those settings in Citrine. net.core.somaxconn has always been 4096 in this environment as far as I'm aware.

Though I'm happy to leave this open to have a discussion about changing these.

from frameworkbenchmarks.

remittor avatar remittor commented on June 16, 2024

@nbrady-techempower

We never adjusted those settings in Citrine. net.core.somaxconn has always been 4096

This is unlikely, because until 2023, projects using uWSGI were correctly launched (they had into config listen = 65535).
Then something happened and this appeared in the run logs: run.log

On the CI system, everything always started without errors.

from frameworkbenchmarks.

remittor avatar remittor commented on June 16, 2024

It was possible to find out more precisely when everything broke.

2022-01-22: OK https://tfb-status.techempower.com/unzip/results.2022-01-27-10-45-01-139.zip/results/20220122165234/uwsgi/run

2022-01-31: Fail https://tfb-status.techempower.com/unzip/results.2022-02-05-08-28-48-272.zip/results/20220131174100/uwsgi/run

There is a possibility that this is due to this event: #7078

from frameworkbenchmarks.

remittor avatar remittor commented on June 16, 2024

I even managed to find out the exact values of this parameter.

2022-01-22: https://tfb-status.techempower.com/unzip/results.2022-01-27-10-45-01-139.zip/results/20220122165234/racket/run

racket: 7:M 26 Jan 11:04:56.441 # WARNING: The TCP backlog setting of 65535 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 16384.
racket: 7:M 26 Jan 11:04:56.441 # Server started, Redis version 3.2.6

2022-01-31: https://tfb-status.techempower.com/unzip/results.2022-02-05-08-28-48-272.zip/results/20220131174100/racket/run

racket: 8:M 11 Apr 11:06:39.480 # WARNING: The TCP backlog setting of 65535 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 4096.
racket: 8:M 11 Apr 11:06:39.480 # Server started, Redis version 3.2.6

That is, the value has changed from 16384 to 4096.


And here is the proof that the CI system have net/core/somaxconn = 65535
https://github.com/TechEmpower/FrameworkBenchmarks/actions/runs/4547822068/jobs/8018189920#step:9:652

racket: 8:M 28 Mar 21:44:29.038 # Server started, Redis version 3.2.6

There is no warning message.

from frameworkbenchmarks.

NateBrady23 avatar NateBrady23 commented on June 16, 2024

It's possible the machines were commissioned with those set and that first upgrade reverted config values. I'll let this run finish and we'll see about updating them.

Edit: I'm not in the office today but I should be able to take care of this tomorrow (Friday) and we'll cut this current run short.

from frameworkbenchmarks.

NateBrady23 avatar NateBrady23 commented on June 16, 2024

Unfortunately couldn't get in to get to this today, but coming first thing Monday!

from frameworkbenchmarks.

NateBrady23 avatar NateBrady23 commented on June 16, 2024

Citrine is updated

from frameworkbenchmarks.

michaelhixson avatar michaelhixson commented on June 16, 2024

@nbrady-techempower What new values did you set?

from frameworkbenchmarks.

remittor avatar remittor commented on June 16, 2024

...we decided not to change it because the real problem was those frameworks not reusing their own connections.

780 * 60 sec = 46800 sec = 780 min = 13 hours
13 / 160 = 0.08 => 8%

Average execution time for 6 tests: 12 min
60 / (12*60) = 0.08 => 8%

And here you need to choose an option:

  1. greater realism
  2. testing acceleration

from frameworkbenchmarks.

remittor avatar remittor commented on June 16, 2024

Citrine is updated

@nbrady-techempower , too early you closed this issue.
After the imi-workerman test, everything completely broke down.

    "ihp": "20230427061559", 
    "imi": "20230427063439", 
    "imi-swoole-mysql-raw": "20230427064707", 
    "imi-swoole-pgsql": "20230427065927", 
    "imi-swoole-pgsql-raw": "20230427071149", 
    "imi-workerman": "ERROR: Problem starting imi-workerman", 
    "imi-workerman-mysql-raw": "ERROR: Problem starting imi-workerman-mysql-raw", 
    "imi-workerman-pgsql": "ERROR: Problem starting imi-workerman-pgsql", 
    "imi-workerman-pgsql-raw": "ERROR: Problem starting imi-workerman-pgsql-raw", 
    "inverno": "ERROR: Problem starting inverno", 
    "inverno-postgres": "ERROR: Problem starting inverno-postgres", 
    "isocket-nio": "ERROR: Problem starting isocket-nio", 
    "japronto": "ERROR: Problem starting japronto", 
    "javalin": "ERROR: Problem starting javalin", 
    "javalin-postgres": "ERROR: Problem starting javalin-postgres", 
    "jawn": "20230427074702", 
    "jersey": "20230427075034", 
    "jester": "20230427075707", 
    "jetty": "20230427080238", 
    "jetty-servlet": "20230427080809",

image

It's too bad that the run.log can't be viewed now.

UPD:
My guess is that this may be caused by too many threads/processes being started.
Look this:

'configs' => [
// 支持设置 Workerman 参数
'count' => (int) shell_exec('nproc') * 4,
],

14(cores) * 2(sockets) * 4 = 112 proof link
Although I can not understand how this can affect the operation of the network part.

from frameworkbenchmarks.

remittor avatar remittor commented on June 16, 2024

here is another place where everything broke:

    "ronykit": "20230429155022", 
    "ronykit-prefork": "20230429155612", 
    "routerling": "20230429160913", 
    "sailsjs": "20230429161412", 
    "salvo": "ERROR: Problem starting salvo", 
    "salvo-diesel": "ERROR: Problem starting salvo-diesel", 
    "salvo-moka": "ERROR: Problem starting salvo-moka", 
    "salvo-mongo": "ERROR: Problem starting salvo-mongo", 
    "salvo-mongo-raw": "ERROR: Problem starting salvo-mongo-raw", 
    "salvo-pg": "ERROR: Problem starting salvo-pg", 
    "salvo-pg-pool": "ERROR: Problem starting salvo-pg-pool", 
    "salvo-sqlx": "ERROR: Problem starting salvo-sqlx", 
    "sanic": "ERROR: Framework is not accepting requests from client machine", 
    "scalene": "20230429165232", 
    "scorper": "20230429165834", 
    "servant": "ERROR: Problem starting servant", 
    "servant-psql-simple": "ERROR: Problem starting servant-psql-simple", 
    "servicestack-v6": "20230429170709", 
    "servicetalk": "20230429171318", 
    "servlet": "20230429171901", 
    "servlet-lib-native": "ERROR: Problem starting servlet-lib-native", 
    "servlet-mysql": "20230429173136", 
    "servlet-postgresql": "20230429174439", 
    "servlet3": "20230429175027",

from frameworkbenchmarks.

remittor avatar remittor commented on June 16, 2024

build imi-workerman:

imi-workerman: No releases available for package "pecl.php.net/event"
imi-workerman: install failed
imi-workerman: Removing intermediate container b5c9940ba4a5
imi-workerman: Docker build failed; terminating
imi-workerman: Traceback (most recent call last):
imi-workerman:   File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
imi-workerman:     raise Exception(token['errorDetail']['message'])
imi-workerman: Exception: The command '/bin/sh -c pecl install event &&     echo "extension=event.so" > /usr/local/etc/php/conf.d/event.ini' returned a non-zero code: 1 

build imi-workerman-mysql-raw

imi-workerman-mysql-raw: Step 1/17 : FROM php:8.2-cli
imi-workerman-mysql-raw: Docker build failed; terminating
imi-workerman-mysql-raw: Traceback (most recent call last):
imi-workerman-mysql-raw:   File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
imi-workerman-mysql-raw:     raise Exception(token['errorDetail']['message'])
imi-workerman-mysql-raw: Exception: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
imi-workerman-mysql-raw: Build time: 15s 

build inverno:

inverno: Step 1/7 : FROM maven:3.8.2-openjdk-16 as maven
inverno: Docker build failed; terminating
inverno: Traceback (most recent call last):
inverno:   File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
inverno:     raise Exception(token['errorDetail']['message'])
inverno: Exception: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
inverno: Build time: 15s 

build japronto:

japronto: Step 1/6 : FROM python:3.8.12
japronto: Docker build failed; terminating
japronto: Traceback (most recent call last):
japronto:   File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
japronto:     raise Exception(token['errorDetail']['message'])
japronto: Exception: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
japronto: Build time: 15s 

build salvo:

salvo: Step 5/7 : RUN cargo build --release
salvo:  ---> Running in 992b0ce6a390
salvo:     Updating git repository `https://github.com/salvo-rs/salvo.git`
salvo:     Updating crates.io index
salvo: error: failed to select a version for the requirement `salvo = "^0.38"`
salvo: candidate versions found which didn't match: 0.39.1, 0.37.9, 0.37.8, ...
salvo: location searched: crates.io index
salvo: required by package `salvo_test v0.15.0 (/salvo)`
salvo: Removing intermediate container 992b0ce6a390
salvo: Docker build failed; terminating
salvo: Traceback (most recent call last):
salvo:   File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 55, in __build
salvo:     raise Exception(token['errorDetail']['message'])
salvo: Exception: The command '/bin/sh -c cargo build --release' returned a non-zero code: 101
salvo: Build time: 1m 20s 

Сreating containers sometimes ends with an error.

Stat for docker errors:
results.2023-02-05-04-03-06-716.zip : 31
results.2023-04-06-08-48-50-340.zip : 6
results.2023-04-19-19-51-19-699.zip : 6
results.2023-04-30-18-31-55-286.zip : 36

Stat for log with text net/http: request canceled while waiting for connection (:
results.2023-02-05-04-03-06-716.zip : 1
results.2023-04-06-08-48-50-340.zip : 0
results.2023-04-19-19-51-19-699.zip : 0
results.2023-04-30-18-31-55-286.zip : 10

I assume that resource docker.io was unavailable for some time on April 27th.

from frameworkbenchmarks.

remittor avatar remittor commented on June 16, 2024

@nbrady-techempower
Looks like I was worrying for nothing. Probably on April 27, just the docker server was unavailable for some time.

from frameworkbenchmarks.

NateBrady23 avatar NateBrady23 commented on June 16, 2024

@remittor That's good news! Thanks for staying on top of it.

from frameworkbenchmarks.

remittor avatar remittor commented on June 16, 2024

@nbrady-techempower

Citrine is updated

Something new settings are not applied.

Look this: https://tfb-status.techempower.com/unzip/results.2023-05-26-18-27-11-854.zip/results/20230520161617/racket/run

racket: 8:M 25 May 2023 09:46:44.997 # WARNING: The TCP backlog setting of 65535 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 4096.
racket: 8:M 25 May 2023 09:46:44.997 # Server initialized
racket: 8:M 25 May 2023 09:46:44.997 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

from frameworkbenchmarks.

NateBrady23 avatar NateBrady23 commented on June 16, 2024

Wow, my mistake! It's set permanently now.

from frameworkbenchmarks.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.