Lose records ? about logdevice

[zhengxiaochuan-3]

root> SELECT count()  FROM partitions where node_id=0                                                                                                                                                                                    

+---------+
| count() |
+---------+
| 2052    |
+---------+
1 rows in set (221.00 msec)
root>                                                                                                                                                                                                                                    
root> SELECT count()  FROM partitions where node_id=1                                                                                                                                                                                    

+---------+
| count() |
+---------+
| 2188    |
+---------+
1 rows in set (233.00 msec)
root> SELECT count()  FROM partitions where node_id=2                                                                                                                                                                                    

+---------+
| count() |
+---------+
| 2050    |
+---------+
1 rows in set (228.00 msec)
root> SELECT count()  FROM partitions where node_id=3                                                                                                                                                                                    

+---------+
| count() |
+---------+
| 2187    |
+---------+
1 rows in set (10.00 msec)
root>  

[MohamedBassem]

The error logs on logdeviced shouldn't cause issues. It means that you have a lot of partitions (You can read more about what's that in the new docs we've landed https://logdevice.io/docs/LogsDB.html). Typically, a partition is 15 minutes (by default) worth of data, when you exceed a certain configurable soft limit (by default 2000), the partition durations increase to decrease the number of partitions. This warning, tells you about this expansion. You can read more about this setting (rocksdb-partition-count-soft-limit) in https://logdevice.io/docs/Settings.html#logsdb and adjust if you need to. So that's regarding the logdeviced error you've seeing and I think it might not be related. Btw, what is the retention of the logs you're writing to logdevice (according to my calculations it's higher than 25 days?).

The second part about failing to read logs. So given that partitions are being created, it means that logs are being written to those partitions. Your cluster might be write available but it could have lost read availability and this could happen for multiple reasons. So I'd need some more debugging informations from you:

1- To confirm that logs are being written, can you please run: select * from sequencers where log_id=<put your id here>? If you run this multiple times, and the field next_lsn is changing, it means that you're successfully writing records. The other field I'd be interested in, is last_released which is what is the last LSN the readers are allowed to read. If that's far from the log tail, it means you're facing read availability issues.
2- Stuck recoveries (https://logdevice.io/docs/Recovery.html) can block read availability. So to check if there's any pending recoveries, run: select * from recoveries.
3- What's your replication factor? Can you tolerate the loss of some nodes? Can you run select * from cluster_state to check if there are any unhealthy nodes? Can you also run select * from shard_authoritative_status to find if there's any unavailable nodes or data loss?
4- Client logs can be helpful as well, by default the loglevel is ERROR in ldcat and ldtail, but if you can lower it down to INFO here (https://github.com/facebookincubator/LogDevice/blob/4a9eb30ec8c706b972d8d1616a7cf69eafe0cfc3/logdevice/examples/cat.cpp#L48) and provide me with the log that would be helpful. I'll file an issue to make this easily configurable via cmdline flags.

Once you provide me with this debug information, we can investigate it more :)

[zhengxiaochuan-3]

@MohamedBassem I am really sorry, it's the "writer" bug, this "writer" include a Third-party log library which has a bug，and I run it in the background(didn't find crash in the first time).
Sorry for bothered you.

The debug tricks you mentioned are very useful to me. I will try to learn more about the cluster SQL info. Is there a way to count how many records on a logid?

In the future when the test is done, I will create a larger cluster :
20 servers
12TB * 12 disks on one server
2*10Gb network cards on one server

This servers, are they suitable for logdevice ?

The doc says that "RocksDB works best if you have less than 2000 partitions in a shard", and "rocksdb-partition-size-limit: Default is 6G","rocksdb-partition-duration:15 minutes"

I should adjust a configuration that is appropriate for our application. Any suggestions?

[MohamedBassem]

@zhengxiaochuan-3 no problem. Glad that you found where the problem is.

I will try to learn more about the cluster SQL

Here's the documentation of all the tables that we have: https://logdevice.io/docs/LDQuery.html. There's a lot of them, but you can use them as a reference.

Is there a way to count how many records on a logid?

There might be a way to do it with ldquery, but I think your best option is to count by reading them.

This servers, are they suitable for logdevice ?

Yeah definitely.

I should adjust a configuration that is appropriate for our application

What's the retention do you have in mind? How big are your records?
As for the configuration, I'm not an expert in rocksdb, @al13n321 can you advise on that?

[zhengxiaochuan-3]

What's the retention do you have in mind? How big is your records?

• My record is about 1KB~128KB
• About 3000~5000 client write to LogDevice concurrently.
• One client may write 50~100 LogIDs (each LogID has a bufferWriter) to LogDevice concurrently.
• One bufferWrite may write 20~100 records per second

[MohamedBassem]

What about the retention of the logs? just collecting data for Mike

You can find that in the backlog field when calling logs show.

[zhengxiaochuan-3]

Currently, I set 60 days in the test cluster.
Maybe 90 days on bigger cluster in the future.
I use it both for Mike and for storage

[zhengxiaochuan-3]

root> logs show                                                                                                                                                                                                                          
▼ /
  Version: 85899345931
  Attributes: 
  - extra-copies: 0    (Overridden)
  - synced-copies: 0    (Overridden)
  - max-writes-in-flight: 1000    (Overridden)
  - single-writer: False    (Overridden)
  - sequencer-affinity: None    (Overridden)
  - nodeset-size: None    (Overridden)
  - backlog: None    (Overridden)
  - delivery-latency: None    (Overridden)
  - scd-enabled: False    (Overridden)
  - local-scd-enabled: False    (Overridden)
  - write-token: None    (Overridden)
  - sticky-copysets: False    (Overridden)
    ◎ /weiweitest5 (9000000..10000000)
      Version: 85899345931
      Attributes:
      - replication-factor: 3    (Overridden)
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - nodeset-size: None
      - backlog: 365d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /zxc_test_2000 (2001..3000)
      Version: 85899345931
      Attributes:
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - replicate-across: {'node': 2}    (Overridden)
      - nodeset-size: None
      - backlog: 3d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /zxc_test (1000..2000)
      Version: 85899345931
      Attributes:
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - replicate-across: {'node': 2}    (Overridden)
      - nodeset-size: None
      - backlog: 3d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /zxc (25001..30000)
      Version: 85899345931
      Attributes:
      - replication-factor: 2    (Overridden)
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - nodeset-size: 2    (Overridden)
      - backlog: 60d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /zxc_logs (11..20)
      Version: 85899345931
      Attributes:
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - replicate-across: {'node': 2}    (Overridden)
      - nodeset-size: None
      - backlog: 3d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /zxc_test_3000 (3001..4000)
      Version: 85899345931
      Attributes:
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - replicate-across: {'node': 2}    (Overridden)
      - nodeset-size: None
      - backlog: 3d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /zxc1 (35001..40000)
      Version: 85899345931
      Attributes:
      - replication-factor: 2    (Overridden)
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - nodeset-size: 2    (Overridden)
      - backlog: 30d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /jls (21..30)
      Version: 85899345931
      Attributes:
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - replicate-across: {'node': 2}    (Overridden)
      - nodeset-size: None
      - backlog: 3d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /jlsapp1 (31..40)
      Version: 85899345931
      Attributes:
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - replicate-across: {'node': 2}    (Overridden)
      - nodeset-size: 2    (Overridden)
      - backlog: 3d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /xxxxx (20001..25000)
      Version: 85899345931
      Attributes:
      - replication-factor: 2    (Overridden)
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - nodeset-size: 2    (Overridden)
      - backlog: 30d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /jlstestapp1 (15001..20000)
      Version: 85899345931
      Attributes:
      - replication-factor: 2    (Overridden)
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - nodeset-size: 2    (Overridden)
      - backlog: 30d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /app (1..10)
      Version: 85899345931
      Attributes:
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - replicate-across: {'node': 2}    (Overridden)
      - nodeset-size: None
      - backlog: 3d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ◎ /jlstestapp (10001..15000)
      Version: 85899345931
      Attributes:
      - replication-factor: 2    (Overridden)
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - nodeset-size: 2    (Overridden)
      - backlog: 30d    (Overridden)
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False

    ▼ /jls/
      Version: 85899345931
      Attributes: 
      - extra-copies: 0
      - synced-copies: 0
      - max-writes-in-flight: 1000
      - single-writer: False
      - sequencer-affinity: None
      - nodeset-size: None
      - backlog: None
      - delivery-latency: None
      - scd-enabled: False
      - local-scd-enabled: False
      - write-token: None
      - sticky-copysets: False
        ▼ /jls/log1/
          Version: 85899345931
          Attributes: 
          - extra-copies: 0
          - synced-copies: 0
          - max-writes-in-flight: 1000
          - single-writer: False
          - sequencer-affinity: None
          - nodeset-size: None
          - backlog: None
          - delivery-latency: None
          - scd-enabled: False
          - local-scd-enabled: False
          - write-token: None
          - sticky-copysets: False

[MohamedBassem]

hmmmm, 90 days is more than most of our usecases at facebook. Maybe increasing the partition duration to 1 hour would keep you under 2000 partitions? assuming that you wouldn't hit the partition size limit? @al13n321 is better suited to help you with this calculation, let's wait for him.