Comments (5)
This turns out to be rather significant for us since we use the memory storage engine. Kernel-level automatic NUMA balancing "kernel.numa_balancing" doesn't help us (enough), basically we discovered that hot spots (certain processes would be much hotter CPU-wise than others) are largely the result of the kernel scheduling the process to a node and having memory allocation local to that node and then subsequently moving the process to another node (the latency impacts are significant in that case and difficult to debug).
[fdbserver.4500]
command = /usr/bin/numactl -m 0 -N 0 /usr/sbin/fdbserver
We are directly managing core and memory affinity with numactl hence.
from foundationdb.
I'd guess the difference will be small on an otherwise-unloaded machine unless you manage to make an fdbserver allocate some memory on the wrong NUMA domain. That should definitely cause noticeable overhead.
from foundationdb.
The performance effects of this sort of tuning tend to be fairly hardware and OS version specific, so I would recommend diverse testing before making this the default behavior out of the box. But an fdbserver parameter (and thus, config file option) to set affinity might be low risk and convenient.
Ideally operating systems should be able to do a reasonable job placing essentially single-threaded programs, even on NUMA machines. But I think setting affinity is a legitimate solution to a particular situation with network card interrupt delivery. It rings a bell for me. There might have been something written on this subject at some point in FDB history, if the Apple team wants to try to dig it up.
from foundationdb.
We messed with pinning the network queues to cores, etc. It only made minor differences with highly unusual test workloads. Never a practical improvement. I agree with @davidscherer 's suggestion of adding an option for CPU affinity.
from foundationdb.
The numactl command can also handle the set-cpu-affinity work.
Unless we want to start a (fdbserver) process on a particular core, it seems using the linux tools to set the affinity is enough.
from foundationdb.
Related Issues (20)
- DR fails atomicity in 7.3.43 HOT 4
- Use std::filesystem::path in Platform.h HOT 7
- Remove path code in platform.h
- Automatic idempotency ids have insufficient entropy
- release package for linux arm64 HOT 16
- Batch GRV Rate Limit Exceeded is not always thrown HOT 2
- Add an API to support splitting a given range into shards individually HOT 6
- Apple analysis HOT 1
- Client side does not merge conflict ranges HOT 2
- Problems in retrieving conflicting keys after setting transaction option ReportConflictingKeys HOT 1
- SQLite in StorageServer deadlocked after the node was disconnected and resumed. HOT 4
- fbcli in python
- Building FoundationDB would fail if CMake uses Makefile HOT 3
- Descriptions of experimental features HOT 1
- VS Code warns "foundationdb/devel:centos7-latest" is unsupported Dev Container HOT 3
- fdbbackup segfaults on v7.1.63 (works on v7.1.61) HOT 1
- Go: getting a one-time error 1039 since removal of the R/O transactions commit HOT 3
- Why is the permission of files generated in datadir -rw—-S—-?How can I modify the permission of this files? HOT 2
- Why is the permission of files generated in datadir -rw—-S—-?How can I modify the permission of this files
- [C API] FDB failed to insert (keys, values) during the single transaction when the total size of the values exceeded 10MB HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from foundationdb.