Comments (9)
root> SELECT count() FROM partitions where node_id=0
+---------+
| count() |
+---------+
| 2052 |
+---------+
1 rows in set (221.00 msec)
root>
root> SELECT count() FROM partitions where node_id=1
+---------+
| count() |
+---------+
| 2188 |
+---------+
1 rows in set (233.00 msec)
root> SELECT count() FROM partitions where node_id=2
+---------+
| count() |
+---------+
| 2050 |
+---------+
1 rows in set (228.00 msec)
root> SELECT count() FROM partitions where node_id=3
+---------+
| count() |
+---------+
| 2187 |
+---------+
1 rows in set (10.00 msec)
root>
from logdevice.
The error logs on logdeviced shouldn't cause issues. It means that you have a lot of partitions (You can read more about what's that in the new docs we've landed https://logdevice.io/docs/LogsDB.html). Typically, a partition is 15 minutes (by default) worth of data, when you exceed a certain configurable soft limit (by default 2000), the partition durations increase to decrease the number of partitions. This warning, tells you about this expansion. You can read more about this setting (rocksdb-partition-count-soft-limit) in https://logdevice.io/docs/Settings.html#logsdb and adjust if you need to. So that's regarding the logdeviced error you've seeing and I think it might not be related. Btw, what is the retention of the logs you're writing to logdevice (according to my calculations it's higher than 25 days?).
The second part about failing to read logs. So given that partitions are being created, it means that logs are being written to those partitions. Your cluster might be write available but it could have lost read availability and this could happen for multiple reasons. So I'd need some more debugging informations from you:
1- To confirm that logs are being written, can you please run: select * from sequencers where log_id=<put your id here>
? If you run this multiple times, and the field next_lsn
is changing, it means that you're successfully writing records. The other field I'd be interested in, is last_released
which is what is the last LSN the readers are allowed to read. If that's far from the log tail, it means you're facing read availability issues.
2- Stuck recoveries (https://logdevice.io/docs/Recovery.html) can block read availability. So to check if there's any pending recoveries, run: select * from recoveries
.
3- What's your replication factor? Can you tolerate the loss of some nodes? Can you run select * from cluster_state
to check if there are any unhealthy nodes? Can you also run select * from shard_authoritative_status
to find if there's any unavailable nodes or data loss?
4- Client logs can be helpful as well, by default the loglevel is ERROR in ldcat and ldtail, but if you can lower it down to INFO here (https://github.com/facebookincubator/LogDevice/blob/4a9eb30ec8c706b972d8d1616a7cf69eafe0cfc3/logdevice/examples/cat.cpp#L48) and provide me with the log that would be helpful. I'll file an issue to make this easily configurable via cmdline flags.
Once you provide me with this debug information, we can investigate it more :)
from logdevice.
@MohamedBassem I am really sorry, it's the "writer" bug, this "writer" include a Third-party log library which has a bug,and I run it in the background(didn't find crash in the first time).
Sorry for bothered you.
The debug tricks you mentioned are very useful to me. I will try to learn more about the cluster SQL info. Is there a way to count how many records on a logid?
In the future when the test is done, I will create a larger cluster :
20 servers
12TB * 12 disks on one server
2*10Gb network cards on one server
This servers, are they suitable for logdevice ?
The doc says that "RocksDB works best if you have less than 2000 partitions in a shard", and "rocksdb-partition-size-limit: Default is 6G","rocksdb-partition-duration:15 minutes"
I should adjust a configuration that is appropriate for our application. Any suggestions?
from logdevice.
@zhengxiaochuan-3 no problem. Glad that you found where the problem is.
I will try to learn more about the cluster SQL
Here's the documentation of all the tables that we have: https://logdevice.io/docs/LDQuery.html. There's a lot of them, but you can use them as a reference.
Is there a way to count how many records on a logid?
There might be a way to do it with ldquery, but I think your best option is to count by reading them.
This servers, are they suitable for logdevice ?
Yeah definitely.
I should adjust a configuration that is appropriate for our application
What's the retention do you have in mind? How big are your records?
As for the configuration, I'm not an expert in rocksdb, @al13n321 can you advise on that?
from logdevice.
What's the retention do you have in mind? How big is your records?
- My record is about 1KB~128KB
- About 3000~5000 client write to LogDevice concurrently.
- One client may write 50~100 LogIDs (each LogID has a bufferWriter) to LogDevice concurrently.
- One bufferWrite may write 20~100 records per second
from logdevice.
What about the retention of the logs? just collecting data for Mike
You can find that in the backlog field when calling logs show
.
from logdevice.
Currently, I set 60 days in the test cluster.
Maybe 90 days on bigger cluster in the future.
I use it both for Mike and for storage
from logdevice.
root> logs show
▼ /
Version: 85899345931
Attributes:
- extra-copies: 0 (Overridden)
- synced-copies: 0 (Overridden)
- max-writes-in-flight: 1000 (Overridden)
- single-writer: False (Overridden)
- sequencer-affinity: None (Overridden)
- nodeset-size: None (Overridden)
- backlog: None (Overridden)
- delivery-latency: None (Overridden)
- scd-enabled: False (Overridden)
- local-scd-enabled: False (Overridden)
- write-token: None (Overridden)
- sticky-copysets: False (Overridden)
◎ /weiweitest5 (9000000..10000000)
Version: 85899345931
Attributes:
- replication-factor: 3 (Overridden)
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- nodeset-size: None
- backlog: 365d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /zxc_test_2000 (2001..3000)
Version: 85899345931
Attributes:
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- replicate-across: {'node': 2} (Overridden)
- nodeset-size: None
- backlog: 3d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /zxc_test (1000..2000)
Version: 85899345931
Attributes:
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- replicate-across: {'node': 2} (Overridden)
- nodeset-size: None
- backlog: 3d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /zxc (25001..30000)
Version: 85899345931
Attributes:
- replication-factor: 2 (Overridden)
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- nodeset-size: 2 (Overridden)
- backlog: 60d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /zxc_logs (11..20)
Version: 85899345931
Attributes:
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- replicate-across: {'node': 2} (Overridden)
- nodeset-size: None
- backlog: 3d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /zxc_test_3000 (3001..4000)
Version: 85899345931
Attributes:
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- replicate-across: {'node': 2} (Overridden)
- nodeset-size: None
- backlog: 3d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /zxc1 (35001..40000)
Version: 85899345931
Attributes:
- replication-factor: 2 (Overridden)
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- nodeset-size: 2 (Overridden)
- backlog: 30d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /jls (21..30)
Version: 85899345931
Attributes:
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- replicate-across: {'node': 2} (Overridden)
- nodeset-size: None
- backlog: 3d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /jlsapp1 (31..40)
Version: 85899345931
Attributes:
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- replicate-across: {'node': 2} (Overridden)
- nodeset-size: 2 (Overridden)
- backlog: 3d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /xxxxx (20001..25000)
Version: 85899345931
Attributes:
- replication-factor: 2 (Overridden)
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- nodeset-size: 2 (Overridden)
- backlog: 30d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /jlstestapp1 (15001..20000)
Version: 85899345931
Attributes:
- replication-factor: 2 (Overridden)
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- nodeset-size: 2 (Overridden)
- backlog: 30d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /app (1..10)
Version: 85899345931
Attributes:
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- replicate-across: {'node': 2} (Overridden)
- nodeset-size: None
- backlog: 3d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
◎ /jlstestapp (10001..15000)
Version: 85899345931
Attributes:
- replication-factor: 2 (Overridden)
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- nodeset-size: 2 (Overridden)
- backlog: 30d (Overridden)
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
▼ /jls/
Version: 85899345931
Attributes:
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- nodeset-size: None
- backlog: None
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
▼ /jls/log1/
Version: 85899345931
Attributes:
- extra-copies: 0
- synced-copies: 0
- max-writes-in-flight: 1000
- single-writer: False
- sequencer-affinity: None
- nodeset-size: None
- backlog: None
- delivery-latency: None
- scd-enabled: False
- local-scd-enabled: False
- write-token: None
- sticky-copysets: False
from logdevice.
hmmmm, 90 days is more than most of our usecases at facebook. Maybe increasing the partition duration to 1 hour would keep you under 2000 partitions? assuming that you wouldn't hit the partition size limit? @al13n321 is better suited to help you with this calculation, let's wait for him.
from logdevice.
Related Issues (20)
- Inconsistent column type for node_id in Sequencer and Nodes table HOT 3
- Settings abort-on-failed-check, abort-on-failed-catch and external-loglevel don't correctly subscribe to settings update HOT 1
- Tests failed in master due to some folly singletons being used without calling folly init
- Make DSCP reflection configurable in server settings HOT 1
- ModuleNotFoundError: No Module named 'libfb' HOT 1
- LogDevice build failing on Ubuntu bionic HOT 4
- logdevice/build_tools/fedora.deps file doesn't exist HOT 2
- With latest code, linking integration_test fail with SIGSEGV
- The log range specified must be positive 62-bit with start smaller or equal than end of range, supplied range is 0..200000 HOT 3
- Can not build on Ubuntu 19.10 HOT 3
- Will sequencer become a bottleneck for large size record with high throughput HOT 4
- Replication policy cross multiple cities with one special deferred? HOT 4
- Can not build on Ubuntu 18.04.4 HOT 2
- Conditional add question HOT 2
- Run quickstart through docker image failed HOT 1
- I want to integrate Kubernetes with VirtualLog. HOT 1
- Docker build fbthrift compilation thrift/lib/py3/test/gen-py3/testing/clients.cpp failed
- Is this project no longer maintained? HOT 12
- [email protected]
- LogDevice cluster start error --- NOSEQUENCER and sequencer:[invalid NodeID]
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logdevice.