Log rotation function is failing about sonic-mgmt

[yxieca]

@kaushiksrivatsav The test case is passing on Microsoft testbeds. Can you upload syslog? I noticed that your image version says 'dirty', meaning that you have local changes when building the image. Can you test with a clean image and upload logs for further analysis?

[wen587]

Done some check on this test failure and found that test_logrotate_small_size is passing on 202305 branch.
However it keep failing in 202311 and master branch. The failure applies to all platform.

1. 202305

StartTimeUTC	OSVersion	TestCase	Result
2024-03-21T09:01:28Z	20230531.22	test_logrotate_small_size	success
2024-03-14T16:10:24Z	20230531.20	test_logrotate_small_size	success
2024-03-14T09:01:41Z	20230531.21	test_logrotate_small_size	success
2024-03-02T09:01:19Z	20230531.19	test_logrotate_small_size	success
2024-02-29T02:58:00Z	20230531.18	test_logrotate_small_size	success
2024-02-07T12:01:40Z	20230531.17	test_logrotate_small_size	success
2024-02-02T12:02:01Z	20230531.16	test_logrotate_small_size	success
2024-02-01T12:01:42Z	20230531.15	test_logrotate_small_size	success

2. 202311

StartTimeUTC	OSVersion	TestCase	Result
2024-03-21T12:01:34Z	20231110.06	test_logrotate_small_size	failure
2024-02-28T04:02:13Z	20231105.05	test_logrotate_small_size	failure
2024-02-07T04:02:37Z	20231105.04	test_logrotate_small_size	failure
2024-01-25T04:02:05Z	20231105.01	test_logrotate_small_size	failure

3. master

StartTimeUTC	OSVersion	TestCase	Result
2024-03-17T10:01:33Z	master.499617-a33330941	test_logrotate_small_size	failure
2024-03-05T04:01:59Z	master.487454-9fef78c4f	test_logrotate_small_size	failure
2024-02-27T08:03:21Z	master.479015-de2ee9f96	test_logrotate_small_size	failure
2024-02-11T10:01:36Z	master.475016-34bdfc8b3	test_logrotate_small_size	failure
2024-02-09T10:02:17Z	master.473616-5a2058941	test_logrotate_small_size	failure
2024-02-04T10:01:21Z	master.469905-665184ee4	test_logrotate_small_size	failure
2024-02-01T04:01:30Z	master.467438-77384494b	test_logrotate_small_size	failure
2024-01-28T10:01:20Z	master.464481-235cf3e66	test_logrotate_small_size	failure
2024-01-26T02:01:43Z	master.462943-1fb9732f4	test_logrotate_small_size	failure
2024-01-25T04:01:55Z	master.462237-c693e75f0	test_logrotate_small_size	failure
2024-01-22T01:07:20Z	master.459945-da0f4ace7	test_logrotate_small_size	failure

[wen587]

Hi, I did a initial triage and found that start service of logrotate-config doesn't take effect in branch of 202311 and master. It result to the rotate size was left as that set in test_logrotate_normal_size which is 16M.

Please see below print and put eyes on Active time for example:

• 202311:

admin@str-7260cx3-acs-2:~$ show ver
 
SONiC Software Version: SONiC.20231110.06
 
admin@str-7260cx3-acs-2:~$ sudo systemctl status logrotate-config
● logrotate-config.service - Update logrotate configuration
     Loaded: loaded (/lib/systemd/system/logrotate-config.service; enabled-runtime; vendor preset: enabled)
     Active: active (exited) since Fri 2024-03-22 04:56:08 UTC; 2min 29s ago
    Process: 88802 ExecStart=/usr/bin/logrotate-config.sh (code=exited, status=0/SUCCESS)
   Main PID: 88802 (code=exited, status=0/SUCCESS)
 
Mar 22 04:56:08 str-7260cx3-acs-2 systemd[1]: Starting Update logrotate configuration...
Mar 22 04:56:08 str-7260cx3-acs-2 systemd[1]: Finished Update logrotate configuration.

admin@str-7260cx3-acs-2:~$ sudo service logrotate-config start

admin@str-7260cx3-acs-2:~$ sudo systemctl status logrotate-config
● logrotate-config.service - Update logrotate configuration
     Loaded: loaded (/lib/systemd/system/logrotate-config.service; enabled-runtime; vendor preset: enabled)
     Active: active (exited) since Fri 2024-03-22 04:56:08 UTC; 2min 45s ago    <==== Active didn't refresh after start.
    Process: 88802 ExecStart=/usr/bin/logrotate-config.sh (code=exited, status=0/SUCCESS)
   Main PID: 88802 (code=exited, status=0/SUCCESS)
 
Mar 22 04:56:08 str-7260cx3-acs-2 systemd[1]: Starting Update logrotate configuration...
Mar 22 04:56:08 str-7260cx3-acs-2 systemd[1]: Finished Update logrotate configuration.

• 202305

admin@str3-7260cx3-acs-14:~$ show ver
 
SONiC Software Version: SONiC.20230531.20
 
admin@str3-7260cx3-acs-14:/var/log$ sudo systemctl status logrotate-config
● logrotate-config.service - Update logrotate configuration
     Loaded: loaded (/lib/systemd/system/logrotate-config.service; enabled-runtime; vendor preset: enabled)
     Active: inactive (dead) since Fri 2024-03-22 04:46:49 UTC; 15min ago
    Process: 35279 ExecStart=/usr/bin/logrotate-config.sh (code=exited, status=0/SUCCESS)
   Main PID: 35279 (code=exited, status=0/SUCCESS)
 
Mar 22 04:46:48 str3-7260cx3-acs-14 systemd[1]: Starting Update logrotate configuration...
Mar 22 04:46:49 str3-7260cx3-acs-14 systemd[1]: logrotate-config.service: Succeeded.
Mar 22 04:46:49 str3-7260cx3-acs-14 systemd[1]: Finished Update logrotate configuration.

admin@str3-7260cx3-acs-14:/var/log$ sudo service logrotate-config start

admin@str3-7260cx3-acs-14:/var/log$ sudo systemctl status logrotate-config
● logrotate-config.service - Update logrotate configuration
     Loaded: loaded (/lib/systemd/system/logrotate-config.service; enabled-runtime; vendor preset: enabled)
     Active: inactive (dead) since Fri 2024-03-22 05:02:28 UTC; 1s ago      <==== refresed after start
    Process: 42116 ExecStart=/usr/bin/logrotate-config.sh (code=exited, status=0/SUCCESS)
   Main PID: 42116 (code=exited, status=0/SUCCESS)
 
Mar 22 05:02:28 str3-7260cx3-acs-14 systemd[1]: Starting Update logrotate configuration...
Mar 22 05:02:28 str3-7260cx3-acs-14 systemd[1]: logrotate-config.service: Succeeded.
Mar 22 05:02:28 str3-7260cx3-acs-14 systemd[1]: Finished Update logrotate configuration.

[wen587]

Suspect feature issue. @qiluo-msft , please help assign to stakeholder to fix.

[qiluo-msft]

@liat-grozovik @echuawu @roy-sror Could you help check?

[congh-nvidia]

Hi @qiluo-msft , this issue has been triaged by @dgsudharsan, and I have created a PR for the fix, please review: #12140. Thank.