"limit" is out of control ? about dmclock

[Yan-waller]

@ivancich could you help to take a look ?

[ivancich]

You're correct that the issue likely involves the calculation of rho and delta. The nature of the algorithm means the limit can never be perfectly enforced. Rho and delta are calculated by the client to give a snapshot summary of the recent service it has received from other servers. So when there's no recent history a client can get lots of service distributed over the servers, and then there will be a compensation afterward. That can lead to cycles of over- and under-service. Perhaps the algorithm can be modified to dampen such cycles over time.

You seem to be getting more dramatic cycling than I get. Here's a run with your same configuration file:

==== Client Data ====
     client:       0       1       2      97      98      99  total
        t_0:   54.50   53.00   53.50   53.50   51.00   53.50 5333.50
        t_1:   34.50   34.50   36.50   33.50   35.50   36.50 3658.50
        t_2:   30.00   32.00   30.00   31.50   31.50   28.00 3016.50
        t_3:   30.00   30.00   29.50   30.00   29.00   30.50 2990.50
        t_4:   31.00   30.00   30.00   30.50   30.50   29.50 2993.00
        t_5:   31.50   30.00   30.00   32.50   29.50   30.50 3005.50
        t_6:   29.00   30.00   30.00   28.00   30.00   30.00 3013.50
        t_7:   30.00   30.00   30.00   29.00   30.00   29.00 2987.00
        t_8:   30.00   30.00   30.00   30.00   30.00   31.00 3013.00
        t_9:   30.00   30.00   30.00   30.00   30.00   30.00 2998.50
       t_10:   30.00   30.00   30.00   30.50   30.00   29.50 3000.00
       t_11:   30.00   30.00   30.00   31.00   30.00   30.50 2999.00
       t_12:   30.00   30.00   30.00   29.50   30.00   29.50 2992.00
       t_13:   30.00   30.00   30.00   29.00   30.00   30.00 2999.00
       t_14:   29.50   29.50   32.00   29.50   30.00   37.00 3074.50
       t_15:   20.00   21.00   18.50   22.00   23.00   15.00 1919.00
       t_16:    0.00    0.00    0.00    0.00    0.00    0.00    7.00
       t_17:    0.00    0.00    0.00    0.00    0.00    0.00    0.00

Since you have server_random_selection set to false, each client sends its requests to its 10 servers in a very orderly manner -- a, b, c, d, e, f, g, h, i, j, a, b, c, d, e, f, g, h, i, j, .... When I take your configuration and set server_random_selection to true, the service becomes much less consistent, but every client seems to be over-served.

==== Client Data ====
     client:       0       1       2      97      98      99  total
        t_0:   57.50   50.50   56.50   57.50   55.50   53.50 5668.00
        t_1:   41.00   52.00   44.00   46.50   45.00   45.50 4653.00
        t_2:   36.50   35.50   43.00   41.00   41.00   44.50 3944.00
        t_3:   42.50   27.50   40.50   38.50   39.50   39.00 3873.00
        t_4:   37.50   47.00   37.50   38.50   33.50   42.50 3808.50
        t_5:   40.50   39.00   44.50   37.00   40.50   34.00 3851.50
        t_6:   32.00   39.50   28.50   40.50   38.50   43.00 3786.50
        t_7:   33.00   38.50   34.50   36.00   47.50   44.50 3791.00
        t_8:   39.50   36.50   40.50   35.50   34.50   38.50 3828.00
        t_9:   40.50   37.00   41.00   34.50   41.50   35.00 3816.00
       t_10:   47.00   40.50   37.00   36.00   38.00   34.00 3804.50
       t_11:   32.00   39.00   20.00   36.00   25.50   29.50 3190.00
       t_12:   14.50    9.00   13.50   11.50   13.00   10.50 1144.00
       t_13:    2.50    4.00   11.50    4.50    6.00    3.00  441.50
       t_14:    1.00    2.50    4.00    3.00    0.50    1.50  197.00
       t_15:    1.50    2.00    2.00    2.00    0.00    1.50   93.50
       t_16:    1.00    0.00    0.50    1.50    0.00    0.00   42.00
       t_17:    0.00    0.00    1.00    0.00    0.00    0.00   25.00
       t_18:    0.00    0.00    0.00    0.00    0.00    0.00   14.50
       t_19:    0.00    0.00    0.00    0.00    0.00    0.00    9.00
       t_20:    0.00    0.00    0.00    0.00    0.00    0.00    8.00
       t_21:    0.00    0.00    0.00    0.00    0.00    0.00    5.00
       t_22:    0.00    0.00    0.00    0.00    0.00    0.00    2.50
       t_23:    0.00    0.00    0.00    0.00    0.00    0.00    2.00
       t_24:    0.00    0.00    0.00    0.00    0.00    0.00    2.00
       t_25:    0.00    0.00    0.00    0.00    0.00    0.00    0.00

So given the total column, it seems as though on average each client is receiving around 38 reqs/sec, which is about 27% more than we'd ideally expect. I will review the calculation of rho and delta and see if I can detect an issue.

Thank you.

[Yan-waller]

Hi Eric, how's it going.
Would you mind telling me the plan of dmclock in ceph, and when does the Qos feature can be used?

[ivancich]

The first version that will be integrated will be focused on modifying the balance of operations between client ops, osd subops, recover, snaptrim, and scrub. From there we'd like to provide even more flexibility.