project-chip / certification-tool Goto Github PK

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.

Home Page: https://csa-iot.org/

License: Apache License 2.0

Shell 100.00%

build-with-matter certification connectivity-standards-alliance csa-iot internet-of-things iot matter testing-tools

certification-tool's Introduction

CSA Certification Tool

About

With the goal of simplifying development, testing, and certification of devices, the Connectivity Standards Alliance has developed a standardized set of tools as well a test harness that is presently used for Matter certifications.

To learn more about the open-source Matter SDK, please read more here

Project Overview

Goals

The test tooling, harness, and scripts are developed with the following goals and principles in mind:

Proven: The test harness and tools are built with and on top of existing technologies where possible.

Robust: The test harness and tools should be reliable and dependable.

Low Cost: The test harness and tools should run on low cost and accessible devices.

Flexible: The test harness and tools are flexible enough to accommodate deployment in different environments.

Easy to Use: The test harness and tools provide a smooth, cohesive, integrated experience.

Open: The Project’s design and technical processes are open and transparent to the general public, including non-members wherever possible.

Instructions

Detailed instructions for how to use this set of tools with Matter are located here

Related Repositories

Please be aware of these related repositories that all have to be used in concert to build, develop with, and use the certification tools

Minimum Hardware Requirements

SD card 64 GB or mo
RaspberryPi 4 or newer with at least 4 GB RAM

License

The test harness and tools are released under the Apache 2.0 license.

certification-tool's People

Contributors

Stargazers

Watchers

Forkers

ccruzagralopes antonio-amjr rquidute fabiowmm mikaelhm whenwords survensa shirelein semi-bin-blain

certification-tool's Issues

[TC-DGGEN-2.1] Semi auto - Test case is failing after test step 10a due to unexpected error

From chip-certification-tool created by Rajashreekalmane: CHIP-Specifications/chip-certification-tool#948

Describe the bug

[TC-DGGEN-2.1] Semi auto - Test case is failing after test step 10a due to unexpected error

CHIP:TOO TotalOperationalHours: 0
Test Step Completed [PASSED]: Step 10a: TH reads the TotalOperationalHours attribute from the DUT. Store the value in operational_hours1.
Executing Test Step: Wait for 2 hours and 5 minutes.
Test Step Completed [PASSED]: Wait for 2 hours and 5 minutes.
Executing Test Step: Step 10a: TH reads the TotalOperationalHours attribute from the DUT. Verify that operational_hours2 is greater than operational_hours1.
Test Step Error: Error occurred during execution of test case TC-DGGEN-2.1. no close frame received or sent
Test Step Completed [ERROR]: Step 10a: TH reads the TotalOperationalHours attribute from the DUT. Verify that operational_hours2 is greater than operational_hours1.
Test Case Completed[ERROR]: TC-DGGEN-2.1 (Semi-automated)
Unpairing chip_tool from device

PFA screenshot below

Steps to reproduce the behavior

Environment
Version: TH Fall2023
Sha: 58a51bef

Log files

DGGEN.txt

PICS file

General Diagnostics Cluster Test Plan.txt

Remove TestRunConfig

From chip-certification-tool created by mikaelhm: CHIP-Specifications/chip-certification-tool#816

Once the frontend support the new project format, we can remove the TestRunConfig completely and require TestRunExecutions to be created with selected_tests directly.

Full PIXIT support in Test Harness

From chip-certification-tool created by mikaelhm: CHIP-Specifications/chip-certification-tool#490

This is an Umbrella issue for discussing full PIXIT support in Test Harness

Hierarchical naming scheme (PIXIT.Cluster.Item, PIXIT.MCORE.Item)
Handling of passing different types that are not necessarily just strings, or that could lose in translation between Python and the final stringification
Handling the actual PIXIT as provided by the test plans rather than the ad-hoc config params in YAML format, which were not designed with PIXIT values/schemes in mind at the start
Format validations (e.g. sizing/typing) using constructed metadata to avoid erroneous PIXIT values

Known high level tasks:

PIXIT XML document Format finalized
Refactor TestPlan to use PIXIT for all variables
Refactor YAML test cases to support real PIXIT naming scheme and values
Refactor chip-tool code-den to handle real PIXIT naming (Need to file issue for chip-tool)
PIXIT XML document for Matter with all PIXIT from test-plan available.
Load/import PIXIT XML in Test Harness
Define PIXIT behavior in Test Harness

[TSTAT-2-1] [TSTAT-2-2] Setback is not selected in PICS but TH continues to test it

From chip-certification-tool created by pankore: CHIP-Specifications/chip-certification-tool#870

Description:

In PICS and zap, we unchecked the Setback related attributes, but TH continues to test for it, causing failure

Logs:

TC-TSTAT-2-1_2023_06_15_16_39_48.log

Additional Info:

v2.8.1 test harness
PICS tool v1.6.6

[Bug] TC-CADMIN-1.24 Test-steps are skipped in UI.

From chip-certification-tool created by Saravana-kr22: CHIP-Specifications/chip-certification-tool#960

Describe the bug

While executing the TC-CADMIN-1.24 in UI test-step 6 is skiped and step is not executed in UI . The test case ended with the below message:

INFO | 2023-09-07 06:38:30.852489 | Stopping chip-tool container
INFO | 2023-09-07 06:38:31.395283 | Test Suite Completed [PENDING]: FirstChipToolSuite
INFO | 2023-09-07 06:38:31.404772 | Test Run Completed [PENDING]

Steps to reproduce the behavior

From the Test-Harness user interface loaded the PICS xml file to select the test cases.
Select TC-cadmin-1.24 test case to executed.
Advertise the Dut using the command ./chip-all-clusters-app
Click the Start button on Test-Harness user interface.

Expected behavior

All the test step should be executed as the Pics used.

Log files

UI-log: cadmin-1-24.log
UI-report - cadmin-1-24.zip

PICS file

cadmin-pics.zip

Screenshots

Environment

TH Version:TH Fall2023
OS: Ubuntu
Browser: Chrome

[Bug][BR-4] bridge-app can't commission using controller(chip-tool).

From chip-certification-tool created by joonhaengHeo: CHIP-Specifications/chip-certification-tool#932

Describe the bug

bridge-app can't commission using controller(chip-tool).

I think bridge-app is only supported thread.
Is this test only available on Thread-supported Devices?

[1691542360.361952][33293:33295] CHIP:DMG: AttributeReportIB =
[1691542360.362020][33293:33295] CHIP:DMG: {
[1691542360.362101][33293:33295] CHIP:DMG: AttributeDataIB =
[1691542360.362197][33293:33295] CHIP:DMG: {
[1691542360.362261][33293:33295] CHIP:DMG: DataVersion = 0xbb46d872,
[1691542360.362286][33293:33295] CHIP:DMG: AttributePathIB =
[1691542360.362388][33293:33295] CHIP:DMG: {
[1691542360.362448][33293:33295] CHIP:DMG: Endpoint = 0x0,
[1691542360.362540][33293:33295] CHIP:DMG: Cluster = 0x31,
[1691542360.362665][33293:33295] CHIP:DMG: Attribute = 0x0000_FFFC,
[1691542360.362821][33293:33295] CHIP:DMG: }
[1691542360.363007][33293:33295] CHIP:DMG:
[1691542360.363098][33293:33295] CHIP:DMG: Data = 2,
[1691542360.363204][33293:33295] CHIP:DMG: },
[1691542360.363325][33293:33295] CHIP:DMG:
[1691542360.363375][33293:33295] CHIP:DMG: },
[1691542360.363480][33293:33295] CHIP:DMG:
[1691542360.363524][33293:33295] CHIP:DMG: ],

Steps to reproduce the behavior

Run bridge-app
Commission using ble-wifi.

Expected behavior

No response
[BR-4]Commissioner.txt
[BR-4]TH_log.txt

No response

PICS file

No response

Screenshots

No response

Environment

TH Version : TH Fall2023 - beta3

Additional Information

No response

[Bug] TC-CGEN-2.4 #4 returns status = 0x87 (CONSTRAINT_ERROR)

From chip-certification-tool created by AlvinHsiao: CHIP-Specifications/chip-certification-tool#968

Describe the bug

#4 returns status = 0x87 (CONSTRAINT_ERROR) instead of the expected error codes on receiving set-regulatory-config command.

Steps to reproduce the behavior

Start the commissioning process of TH1 on DUT
Execute the step 4

Expected behavior

The test case should return the expected error codes on receiving set-regulatory-config command.

Log files

DUT log.txt
TH log.txt

PICS file

ExtendedColorLight_PICS_TE2_V21.zip

Screenshots

No response

Environment

TH version - TH Fall2023 - beta3

Additional Information

It might be caused by adding the length check in src/app/clusters/general-commissioning-server/general-commissioning-server.cpp

[Bug] Incorrectly configured docker-compose.yaml

From chip-certification-tool created by hicklin: CHIP-Specifications/chip-certification-tool#975

Describe the bug

After setting up an new RPi with ssh access, cloning this repository on the RPi and install all the required docker tools, I ran the start.sh and got the following error.

ERROR: The Compose file './docker-compose.yml' is invalid because:
networks.default value Additional properties are not allowed ('name', 'enable_ipv6' were unexpected)

It seems that either the readme is incorrect, there is something wrong with the start.sh script or there is a misconfiguration in the docker-compose.yaml.

Steps to reproduce the behavior

Follow the readme until running the start.sh script.
Install the docker tools.
1. curl -fsSL https://get.docker.com -o get-docker.sh and run the script.
2. sudo apt install docker-compose.
run the start.sh script.

Expected behavior

The readme gives a code snippet of what is to be expected that after running the start.sh script.

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

Feature suggestion - Client test cases launch All-Clusters on the Test Harness

From chip-certification-tool created by EF-JANC: CHIP-Specifications/chip-certification-tool#472

For DUT as Client test cases, when running the Manual_tests, would it be possible to start the All-Clusters-app as TH in the Test Harness so this can be logged directly with the test case, instead of us having to run 3 separate logging windows (TH + SSH To All-Clusters + SSH to DUT)

[Bug] Typo or missing functionality

From chip-certification-tool created by hicklin: CHIP-Specifications/chip-certification-tool#974

Describe the bug

The readme suggests that all one needs to do to get the TH running in to execute the start.sh file. The readme says that this should install all the TH dependencies. This is not quite correct as the script fails due to missing docker tools such as docker-compose.

The readme doesn't say that these tools should be installed beforehand. If this is expected, maybe an extra point could be added to note this like the points for setting up ssh.

Steps to reproduce the behavior

Follow the read me and run the start.sh script in a new RPi.
Note that you get an error due to missing dependencies.

Expected behavior

No response

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

[SMOKECO] Fails in TH waits for a report from DUT with a timeout of 300 seconds

From chip-certification-tool created by KishokG: CHIP-Specifications/chip-certification-tool#951

Describe the bug

UI test runner fails at all "TH waits for a report from DUT with a timeout of 300 seconds" by throwing below error

Test Failure: The test expects no error but the "FAILURE" error occured. Test Failure: The test expects a value but it does not exists in the response."

Test case affected :

TC-SMOKECO-2.2
TC-SMOKECO-2.3
TC-SMOKECO-2.4
TC-SMOKECO-2.5
TC-SMOKECO-2.6

Steps to reproduce the behavior

From the Test-Harness user interface loaded the Smoke CO Alarm Cluster Test Plan (1).xml and General Diagnostics Cluster Test Plan.xml PICS file to select the test cases.
Selected the SMOKECO Automated and Semi Automated test case to executed.
Executed the below mentioned command to put DUT into a commissionable state, ./chip-all-clusters-app --enable-key 00112233445566778899aabbccddeeff
Click the Start button on Test-Harness user interface.

Expected behavior

TC should execute successfully without error

Log files

SMOKE_2023_08_29_18_44_15.log

PICS file

SMOKE_PICS.zip

Screenshots

Environment

Version: TH Fall2023
Sha: 58a51bef

Additional Information

No response

Fabric ID is not updated during automated test runs ( TC-GRPKEY-2.1 , TC-SC-5.1, TC-ACL-2.7, TC-ACL-2.8)

From chip-certification-tool created by Shweta-UL: CHIP-Specifications/chip-certification-tool#972

Describe the bug

Fabric ID is not updated correctly for the listed test cases. However, during the manual test run we could notice the correct response from DUT.

Steps to reproduce the behavior

No response

Expected behavior

No response

Log files

TC-ACL-2.8_Manual_Last_Step_Pass.txt
ACL_2_8_2023_09_15_15_37_29_Fail.log
[ACL_2_7_2023_09_15_15_20_31_Fail.log](https://github.com/CHIP-Specifications/chip-certification-tool/files/12662502/
TC_GRPKEY_2-1_2023_09_18_10_42_08.log
ACL_2_7_2023_09_15_15_20_31_Fail.log)
TC_SC_5-1_2023_09_18_12_56_06.log
TC-GRPKEY_2_1_Manual_Run_Log_Pass.txt

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

[Bug] Not able to perform decommissioning in simulated test case

From chip-certification-tool created by Ashwinigrl: CHIP-Specifications/chip-certification-tool#950

Describe the bug

Last step of all simulated test case there is a step prompting to decommission the DUT, But when trying to decommission using chip-tool command it end with timeout. Timeout during decommission happens only if it is already decommissioned.

I believe that runner is automatically decommissioning the DUT and prompting for decommissioning if in that case the prompt has to be removed if not the runner needs attention for the same.

Steps to reproduce the behavior

Open the UI
Upload the PICS
Ensure the Simulated test case is selected
Click on start
UI Prompts for commissioning
Provide the pairing code in DUT with payload
Provide the command as per the prompts
After completion of test case execution, Provide the Unpairing code in DUT with payload

Expected behavior

DUT should Decommissioned successfully

Log files

Ex log for TCOCC-2.4:
Simulated backend log.txt
TC-OCC-2.4(DUT).txt

Environment

TH Version: TH Fall2023
Sha: 58a51bef

Additional Information

Facing this issue for all the simulated test cases

[TC-TSTAT-2.1] [TC-TSTAT-2.2] incorrectly assumes default limits

From chip-certification-tool created by st-mediola: CHIP-Specifications/chip-certification-tool#824

Summary Title:

TC-TSTAT-2.1 & TC-TSTAT-2.2 uses several default values & limits that don't apply if a manufacturer specified attribute overrides them. Most importantly when checking whether temperatures at, above & below limits work as expected. In our case the lower heating setpoint limit causes failures.

Description:

The lower limit for (Un)OccupiedHeatingSetpoint should only be 700 if neither AbsMinHeatSetpointLimit nor MinHeatSetpointLimit is available. Both are optional so the logic to correctly calculate the effective limit would be firstDefined(MinHeatSetpointLimit, AbsMinHeatSetpointLimit, 700) (assuming firstDefined would return the the first parameter that has a defined value). It's sufficient to assume that e.g. 600 is below limit in case AbsMinHeatSetpointLimit is not defined but MinHeatSetpointLimit is because this implies that 700 <= MinHeatSetpointLimit.
The lower limit for MinHeatSetpointLimit itself should only be 700 if AbsMinHeatSetpointLimit is not present.
The same issue exist for the other heating attributes, e.g. the upper limit for OccupiedHeatingSetpoint should only be 3000 if neither AbsMaxHeatSetpointLimit nor MaxHeatSetpointLimit are defined
It's also equivalent for cooling, the lower limit of 1600 is changed by AbsMinCoolSetpointLimit and MinCoolSetpointLimit, the upper limit of 3200 is changed by AbsMaxCoolSetpointLimit and MaxCoolSetpointLimit
Steps like "Writes (sets back) default value of MinHeatSetpointLimit" cannot assume that the value used (700) is valid in case the manufacturer provides stricter Abs.. limits.
The default temperature of 20° used in several steps like "Sets OccupiedHeatingSetpoint to default value" steps is technically not required to be supported, a hypothetical freezer or sauna thermostat would fail here as well. Potential fix here and in other steps that reset a default could be to read the initial value and set it instead of setting a fixed value.
The supported temperature range is also not required to allow +-3°C which would fail the raise/lower command steps, even with adjusted "reset to default value" from above. In theory AbsMinHeatSetpointLimit==AbsMaxHeatSetpointLimit would be within spec as far as I can tell.

The hardcoded values cause failures depending on the DUT config, e.g. in our case a simple thermostat with heating only and

AbsMinHeatSetpointLimit=500 fails TC-TSTAT-2.2 in step "Writes OccupiedCoolingSetpoint to value below the MinCoolSetpointLimit because the script writes 600 and expects this to be below the limit. It should in this case write a value < 500 to come to the expected CONSTRAINT_ERROR source
- A potential fix here would be to split the step into && !TSTAT.S.A0003 and && TSTAT.S.A0003 and use either the default or the value of AbsMinHeatSetpointLimit
- logs&report: https://gist.github.com/st-mediola/41bc8c12f0c5b401ce4b68e9f117d822 (Found failure: status value mismatch: expected 135 but got 0)
AbsMinHeatSetpointLimit=500 && MinHeatSetpointLimit=500 fails TC-TSTAT-2.1 in step "Reads attribute from DUT: MinHeatSetpointLimit" because the constraints in this case are minValue: 700 & maxValue: 3000 which are incorrect once there are manufacturer specified Abs... limits. source

Overall, all the fixed values in both TCs look like they need to be reviewed and split into multiple mutually exclusive steps or made into manual steps due to combinatorial explosion.

Steps to reproduce:

Create DUT with Abs limits other than the default limits

Additional Info:

Relevant PICS are

TSTAT.S.A0003 # AbsMinHeatSetpointLimit (TSTAT.S.F00: O) or else default 7°
TSTAT.S.A0004 # AbsMaxHeatSetpointLimit (TSTAT.S.F00: O) or else default 30°
TSTAT.S.A0005 # AbsMinCoolSetpointLimit (TSTAT.S.F01: O) or else default 16°
TSTAT.S.A0006 # AbsMaxCoolSetpointLimit (TSTAT.S.F01: O) or else default 32°
TSTAT.S.A0015 # MinHeatSetpointLimit (TSTAT.S.F00: O) or else default AbsMinHeatSetpointLimit
TSTAT.S.A0016 # MaxHeatSetpointLimit (TSTAT.S.F00: O) or else default AbsMaxHeatSetpointLimit
TSTAT.S.A0017 # MinCoolSetpointLimit (TSTAT.S.F01: O) or else default AbsMinCoolSetpointLimit
TSTAT.S.A0018 # MaxCoolSetpointLimit (TSTAT.S.F01: O) or else default AbsMaxCoolSetpointLimit

[Bug] Typo in readme file

From chip-certification-tool created by hicklin: CHIP-Specifications/chip-certification-tool#973

Describe the bug

The readme file indicates that the start.sh script is in the root folder. This is not correct as it is in the scripts folder.

Steps to reproduce the behavior

Read the readme
look for the file

Expected behavior

The file is where the readme says it is.

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

[Suggestion] Tool unit testcase for stress testing

From chip-certification-tool-frontend created by karthick-grl: CHIP-Specifications/chip-certification-tool-frontend#133

Initial testcases proposed,

Testcase to populate continues console logs (may be in millions) top check if UI can handle it properly
Need a script that populates database with 200+ project entries and 500+ test execution entries so we could check if UI could populate everything in pagination correctly

[TC-SC-5.2] Test case is failing due to TH sends AddGroup command

From chip-certification-tool created by mideayanghui: CHIP-Specifications/chip-certification-tool#881

Describe the bug

[TC-SC-5.2] Test case is failing , just as the Test Harness's screenshot as below

Note
TC-SC-5.2 is passing while verifying manually.

Additional Info
TestHarness:
Version: v2.9-beta1
Sha: b0593a99

Test Harness UI execution log
TC-SC-5.2-1E1dQtKS_TH_log__2023_07_07_15_58_37.log
TC-SC-5.2-1E1dQtKS_TH_log_DUT_2023-07-07_15-52-49.log

Test Harness manual execution log
Uploading TC-SC-5.2-1E1dQtKS_TH_2023-07-04_15-13-06.log…
TC-SC-5.2-1E1dQtKS_DUT_2023-07-04_15-13-08.log

Steps to reproduce the behavior

No response

Expected behavior

No response

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

[Bug] Chip-tool fails to register with Avahi, reports "Incorrect State"

From chip-certification-tool created by jrhees-cae: CHIP-Specifications/chip-certification-tool#979

Describe the bug

On occasion, chip-tool will fail with "Incorrect State". This is especially prevalent on UI-Semi-Automated tests where both OTBR/TH and manual execution of chip-tool are occurring, such as TC-ACL-2.7

When failing, chip-tool will error out as follows:

ubuntu@ubuntu:~$ docker exec -it th-chip-tool /bin/bash
root@ubuntu:~# ./chip-tool  pairing open-commissioning-window 0x3000b760b7d068c9 1 400 2000 3841
[1694821417.478311][27:27] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_kvs
[1694821417.483926][27:27] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_factory.ini
[1694821417.484275][27:27] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_config.ini
[1694821417.484436][27:27] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_counters.ini
[1694821417.484885][27:27] CHIP:DL: writing settings to file (/tmp/chip_counters.ini-GZxz3p)
[1694821417.485635][27:27] CHIP:DL: renamed tmp file to file (/tmp/chip_counters.ini)
[1694821417.485711][27:27] CHIP:DL: NVS set: chip-counters/reboot-count = 2 (0x2)
[1694821417.486672][27:27] CHIP:DL: Got Ethernet interface: eth0
[1694821417.487321][27:27] CHIP:DL: Found the primary Ethernet interface:eth0
[1694821417.488004][27:27] CHIP:DL: Got WiFi interface: wlan0
[1694821417.488074][27:27] CHIP:DL: Failed to reset WiFi statistic counts
[1694821417.488131][27:27] CHIP:IN: UDP::Init bind&listen port=0
[1694821417.488240][27:27] CHIP:IN: UDP::Init bound to port=40410
[1694821417.488263][27:27] CHIP:IN: BLEBase::Init - setting/overriding transport
[1694821417.488280][27:27] CHIP:IN: TransportMgr initialized
[1694821417.488311][27:27] CHIP:FP: Initializing FabricTable from persistent storage
[1694821417.488483][27:27] CHIP:TS: Last Known Good Time: 2023-09-02T01:00:09
[1694821417.489930][27:27] CHIP:FP: Fabric index 0x1 was retrieved from storage. Compressed FabricId 0xDAD59C801F3BE943, FabricId 0x0000000000000001, NodeId 0x000000000001B669, VendorId 0xFFF1
[1694821417.492171][27:27] CHIP:ZCL: Using ZAP configuration...
[1694821417.496285][27:27] CHIP:DL: Avahi re-register required
[1694821417.496404][27:27] CHIP:DIS: Delaying proxy of operational discovery: missing delegate
[1694821417.496458][27:27] CHIP:CTL: System State Initialized...
[1694821417.496601][27:27] CHIP:CTL: Setting attestation nonce to random value
[1694821417.496641][27:27] CHIP:CTL: Setting CSR nonce to random value
[1694821417.496705][27:27] CHIP:IN: UDP::Init bind&listen port=5550
[1694821417.496827][27:27] CHIP:IN: UDP::Init bound to port=5550
[1694821417.496849][27:27] CHIP:IN: TransportMgr initialized
[1694821417.497096][27:29] CHIP:DL: CHIP task running
[1694821417.497281][27:29] CHIP:DL: HandlePlatformSpecificBLEEvent 32787
[1694821417.497617][27:29] CHIP:CTL: Setting attestation nonce to random value
[1694821417.497795][27:29] CHIP:CTL: Setting CSR nonce to random value
[1694821417.498672][27:29] CHIP:CTL: Generating NOC
[1694821417.499643][27:29] CHIP:FP: Validating NOC chain
[1694821417.501411][27:29] CHIP:FP: NOC chain validation successful
[1694821417.501569][27:29] CHIP:FP: Updated fabric at index: 0x1, Node ID: 0x000000000001B669
[1694821417.501605][27:29] CHIP:TS: Last Known Good Time: 2023-09-02T01:00:09
[1694821417.501626][27:29] CHIP:TS: New proposed Last Known Good Time: 2021-01-01T00:00:00
[1694821417.501647][27:29] CHIP:TS: Retaining current Last Known Good Time
[1694821417.506367][27:29] CHIP:FP: Metadata for Fabric 0x1 persisted to storage.
[1694821417.508356][27:29] CHIP:TS: Committing Last Known Good Time to storage: 2023-09-02T01:00:09
[1694821417.510249][27:29] CHIP:CTL: Joined the fabric at index 1. Fabric ID is 0x0000000000000001 (Compressed Fabric ID: DAD59C801F3BE943)
[1694821417.510305][27:29] CHIP:IN: UDP::Init bind&listen port=5550
[1694821417.510416][27:29] CHIP:IN: UDP::Init bound to port=5550
[1694821417.510439][27:29] CHIP:IN: TransportMgr initialized
[1694821417.538121][27:29] CHIP:CSM: FindOrEstablishSession: PeerId = [1:3000B760B7D068C9]
[1694821417.538176][27:29] CHIP:CSM: FindOrEstablishSession: No existing OperationalSessionSetup instance found
[1694821417.538216][27:29] CHIP:DIS: OperationalSessionSetup[1:3000B760B7D068C9]: State change 1 --> 2
[1694821417.538249][27:29] CHIP:DIS: OperationalSessionSetup[1:3000B760B7D068C9]: State change 2 --> 1
[1694821417.538294][27:29] CHIP:CTL: Failed to open pairing window on the device. Status src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state
[1694821417.538319][27:29] CHIP:-: src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state at ../../examples/chip-tool/commands/pairing/OpenCommissioningWindowCommand.cpp:50
[1694821417.538340][27:29] CHIP:-: src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state at ../../examples/chip-tool/commands/pairing/OpenCommissioningWindowCommand.cpp:57
[1694821417.538674][27:27] CHIP:CTL: Shutting down the commissioner
[1694821417.538905][27:27] CHIP:CTL: Shutting down the controller
[1694821417.539007][27:27] CHIP:IN: Expiring all sessions for fabric 0x1!!
[1694821417.539055][27:27] CHIP:FP: Forgetting fabric 0x1
[1694821417.539111][27:27] CHIP:TS: Pending Last Known Good Time: 2023-09-02T01:00:09
[1694821417.539357][27:27] CHIP:TS: Previous Last Known Good Time: 2023-09-02T01:00:09
[1694821417.539416][27:27] CHIP:TS: Reverted Last Known Good Time to previous value
[1694821417.539500][27:27] CHIP:CTL: Shutting down the commissioner
[1694821417.539576][27:27] CHIP:CTL: Shutting down the controller
[1694821417.539594][27:27] CHIP:CTL: Shutting down the System State, this will teardown the CHIP Stack
[1694821417.539783][27:27] CHIP:DMG: All ReadHandler-s are clean, clear GlobalDirtySet
[1694821417.539897][27:27] CHIP:FP: Shutting down FabricTable
[1694821417.539935][27:27] CHIP:TS: Pending Last Known Good Time: 2023-09-02T01:00:09
[1694821417.540055][27:27] CHIP:TS: Previous Last Known Good Time: 2023-09-02T01:00:09
[1694821417.540079][27:27] CHIP:TS: Reverted Last Known Good Time to previous value
[1694821417.540555][27:27] CHIP:DL: writing settings to file (/tmp/chip_counters.ini-hUu3yx)
[1694821417.541267][27:27] CHIP:DL: renamed tmp file to file (/tmp/chip_counters.ini)
[1694821417.541381][27:27] CHIP:DL: NVS set: chip-counters/total-operational-hours = 0 (0x0)
[1694821417.541434][27:27] CHIP:DL: Inet Layer shutdown
[1694821417.541477][27:27] CHIP:DL: BLE shutdown
[1694821417.541520][27:27] CHIP:DL: System Layer shutdown
[1694821417.541937][27:27] CHIP:TOO: Run command failure: src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state
root@ubuntu:~# ./chip-tool  pairing open-commissioning-window 0x3000b760b7d068c9 1 400 2000 3841
[1694821660.211356][30:30] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_kvs
[1694821660.217164][30:30] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_factory.ini
[1694821660.217463][30:30] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_config.ini
[1694821660.217604][30:30] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_counters.ini
[1694821660.218177][30:30] CHIP:DL: writing settings to file (/tmp/chip_counters.ini-LDIR63)
[1694821660.218906][30:30] CHIP:DL: renamed tmp file to file (/tmp/chip_counters.ini)
[1694821660.218982][30:30] CHIP:DL: NVS set: chip-counters/reboot-count = 3 (0x3)
[1694821660.219934][30:30] CHIP:DL: Got Ethernet interface: eth0
[1694821660.220556][30:30] CHIP:DL: Found the primary Ethernet interface:eth0
[1694821660.221255][30:30] CHIP:DL: Got WiFi interface: wlan0
[1694821660.221325][30:30] CHIP:DL: Failed to reset WiFi statistic counts
[1694821660.221382][30:30] CHIP:IN: UDP::Init bind&listen port=0
[1694821660.221491][30:30] CHIP:IN: UDP::Init bound to port=45042
[1694821660.221515][30:30] CHIP:IN: BLEBase::Init - setting/overriding transport
[1694821660.221534][30:30] CHIP:IN: TransportMgr initialized
[1694821660.221567][30:30] CHIP:FP: Initializing FabricTable from persistent storage
[1694821660.221734][30:30] CHIP:TS: Last Known Good Time: 2023-09-02T01:00:09
[1694821660.223233][30:30] CHIP:FP: Fabric index 0x1 was retrieved from storage. Compressed FabricId 0xDAD59C801F3BE943, FabricId 0x0000000000000001, NodeId 0x000000000001B669, VendorId 0xFFF1
[1694821660.225505][30:30] CHIP:ZCL: Using ZAP configuration...
[1694821660.229506][30:30] CHIP:DL: Avahi re-register required
[1694821660.229623][30:30] CHIP:DIS: Delaying proxy of operational discovery: missing delegate
[1694821660.229681][30:30] CHIP:CTL: System State Initialized...
[1694821660.229812][30:30] CHIP:CTL: Setting attestation nonce to random value
[1694821660.229850][30:30] CHIP:CTL: Setting CSR nonce to random value
[1694821660.229909][30:30] CHIP:IN: UDP::Init bind&listen port=5550
[1694821660.230027][30:30] CHIP:IN: UDP::Init bound to port=5550
[1694821660.230048][30:30] CHIP:IN: TransportMgr initialized
[1694821660.230436][30:32] CHIP:DL: CHIP task running
[1694821660.230642][30:32] CHIP:DL: HandlePlatformSpecificBLEEvent 32787
[1694821660.230987][30:32] CHIP:CTL: Setting attestation nonce to random value
[1694821660.231167][30:32] CHIP:CTL: Setting CSR nonce to random value
[1694821660.232125][30:32] CHIP:CTL: Generating NOC
[1694821660.233022][30:32] CHIP:FP: Validating NOC chain
[1694821660.234675][30:32] CHIP:FP: NOC chain validation successful
[1694821660.234834][30:32] CHIP:FP: Updated fabric at index: 0x1, Node ID: 0x000000000001B669
[1694821660.234908][30:32] CHIP:TS: Last Known Good Time: 2023-09-02T01:00:09
[1694821660.234956][30:32] CHIP:TS: New proposed Last Known Good Time: 2021-01-01T00:00:00
[1694821660.235027][30:32] CHIP:TS: Retaining current Last Known Good Time
[1694821660.237459][30:32] CHIP:FP: Metadata for Fabric 0x1 persisted to storage.
[1694821660.239587][30:32] CHIP:TS: Committing Last Known Good Time to storage: 2023-09-02T01:00:09
[1694821660.241774][30:32] CHIP:CTL: Joined the fabric at index 1. Fabric ID is 0x0000000000000001 (Compressed Fabric ID: DAD59C801F3BE943)
[1694821660.241830][30:32] CHIP:IN: UDP::Init bind&listen port=5550
[1694821660.241938][30:32] CHIP:IN: UDP::Init bound to port=5550
[1694821660.241958][30:32] CHIP:IN: TransportMgr initialized
[1694821660.268534][30:32] CHIP:CSM: FindOrEstablishSession: PeerId = [1:3000B760B7D068C9]
[1694821660.268587][30:32] CHIP:CSM: FindOrEstablishSession: No existing OperationalSessionSetup instance found
[1694821660.268624][30:32] CHIP:DIS: OperationalSessionSetup[1:3000B760B7D068C9]: State change 1 --> 2
[1694821660.268654][30:32] CHIP:DIS: OperationalSessionSetup[1:3000B760B7D068C9]: State change 2 --> 1
[1694821660.268695][30:32] CHIP:CTL: Failed to open pairing window on the device. Status src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state
[1694821660.268723][30:32] CHIP:-: src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state at ../../examples/chip-tool/commands/pairing/OpenCommissioningWindowCommand.cpp:50
[1694821660.268747][30:32] CHIP:-: src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state at ../../examples/chip-tool/commands/pairing/OpenCommissioningWindowCommand.cpp:57
[1694821660.269042][30:30] CHIP:CTL: Shutting down the commissioner
[1694821660.269138][30:30] CHIP:CTL: Shutting down the controller
[1694821660.269182][30:30] CHIP:IN: Expiring all sessions for fabric 0x1!!
[1694821660.269206][30:30] CHIP:FP: Forgetting fabric 0x1
[1694821660.269535][30:30] CHIP:TS: Pending Last Known Good Time: 2023-09-02T01:00:09
[1694821660.269697][30:30] CHIP:TS: Previous Last Known Good Time: 2023-09-02T01:00:09
[1694821660.269722][30:30] CHIP:TS: Reverted Last Known Good Time to previous value
[1694821660.269767][30:30] CHIP:CTL: Shutting down the commissioner
[1694821660.269859][30:30] CHIP:CTL: Shutting down the controller
[1694821660.269888][30:30] CHIP:CTL: Shutting down the System State, this will teardown the CHIP Stack
[1694821660.270057][30:30] CHIP:DMG: All ReadHandler-s are clean, clear GlobalDirtySet
[1694821660.270139][30:30] CHIP:FP: Shutting down FabricTable
[1694821660.270169][30:30] CHIP:TS: Pending Last Known Good Time: 2023-09-02T01:00:09
[1694821660.270289][30:30] CHIP:TS: Previous Last Known Good Time: 2023-09-02T01:00:09
[1694821660.270311][30:30] CHIP:TS: Reverted Last Known Good Time to previous value
[1694821660.270644][30:30] CHIP:DL: writing settings to file (/tmp/chip_counters.ini-xIVZYz)
[1694821660.271367][30:30] CHIP:DL: renamed tmp file to file (/tmp/chip_counters.ini)
[1694821660.271444][30:30] CHIP:DL: NVS set: chip-counters/total-operational-hours = 0 (0x0)
[1694821660.271467][30:30] CHIP:DL: Inet Layer shutdown
[1694821660.271484][30:30] CHIP:DL: BLE shutdown
[1694821660.271502][30:30] CHIP:DL: System Layer shutdown
[1694821660.272158][30:30] CHIP:TOO: Run command failure: src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state
root@ubuntu:~# ./chip-tool  pairing open-commissioning-window 0x3000b760b7d068c9 1 400 2000 3841
[1694821965.309539][33:33] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_kvs
[1694821965.315161][33:33] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_factory.ini
[1694821965.315462][33:33] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_config.ini
[1694821965.315598][33:33] CHIP:DL: ChipLinuxStorage::Init: Using KVS config file: /tmp/chip_counters.ini
[1694821965.316114][33:33] CHIP:DL: writing settings to file (/tmp/chip_counters.ini-pIQdX4)
[1694821965.316899][33:33] CHIP:DL: renamed tmp file to file (/tmp/chip_counters.ini)
[1694821965.317021][33:33] CHIP:DL: NVS set: chip-counters/reboot-count = 4 (0x4)
[1694821965.317962][33:33] CHIP:DL: Got Ethernet interface: eth0
[1694821965.318616][33:33] CHIP:DL: Found the primary Ethernet interface:eth0
[1694821965.319319][33:33] CHIP:DL: Got WiFi interface: wlan0
[1694821965.319392][33:33] CHIP:DL: Failed to reset WiFi statistic counts
[1694821965.319452][33:33] CHIP:IN: UDP::Init bind&listen port=0
[1694821965.319560][33:33] CHIP:IN: UDP::Init bound to port=53530
[1694821965.319633][33:33] CHIP:IN: BLEBase::Init - setting/overriding transport
[1694821965.319655][33:33] CHIP:IN: TransportMgr initialized
[1694821965.319709][33:33] CHIP:FP: Initializing FabricTable from persistent storage
[1694821965.319874][33:33] CHIP:TS: Last Known Good Time: 2023-09-02T01:00:09
[1694821965.321428][33:33] CHIP:FP: Fabric index 0x1 was retrieved from storage. Compressed FabricId 0xDAD59C801F3BE943, FabricId 0x0000000000000001, NodeId 0x000000000001B669, VendorId 0xFFF1
[1694821965.323707][33:33] CHIP:ZCL: Using ZAP configuration...
[1694821965.327318][33:33] CHIP:DL: Avahi re-register required
[1694821965.327432][33:33] CHIP:DIS: Delaying proxy of operational discovery: missing delegate
[1694821965.327492][33:33] CHIP:CTL: System State Initialized...
[1694821965.327622][33:33] CHIP:CTL: Setting attestation nonce to random value
[1694821965.327658][33:33] CHIP:CTL: Setting CSR nonce to random value
[1694821965.327789][33:33] CHIP:IN: UDP::Init bind&listen port=5550
[1694821965.327907][33:33] CHIP:IN: UDP::Init bound to port=5550
[1694821965.327926][33:33] CHIP:IN: TransportMgr initialized
[1694821965.328187][33:35] CHIP:DL: CHIP task running
[1694821965.328385][33:35] CHIP:DL: HandlePlatformSpecificBLEEvent 32787
[1694821965.328709][33:35] CHIP:CTL: Setting attestation nonce to random value
[1694821965.328886][33:35] CHIP:CTL: Setting CSR nonce to random value
[1694821965.329830][33:35] CHIP:CTL: Generating NOC
[1694821965.330761][33:35] CHIP:FP: Validating NOC chain
[1694821965.332551][33:35] CHIP:FP: NOC chain validation successful
[1694821965.332749][33:35] CHIP:FP: Updated fabric at index: 0x1, Node ID: 0x000000000001B669
[1694821965.332782][33:35] CHIP:TS: Last Known Good Time: 2023-09-02T01:00:09
[1694821965.332803][33:35] CHIP:TS: New proposed Last Known Good Time: 2021-01-01T00:00:00
[1694821965.332822][33:35] CHIP:TS: Retaining current Last Known Good Time
[1694821965.335097][33:35] CHIP:FP: Metadata for Fabric 0x1 persisted to storage.
[1694821965.337030][33:35] CHIP:TS: Committing Last Known Good Time to storage: 2023-09-02T01:00:09
[1694821965.338882][33:35] CHIP:CTL: Joined the fabric at index 1. Fabric ID is 0x0000000000000001 (Compressed Fabric ID: DAD59C801F3BE943)
[1694821965.338938][33:35] CHIP:IN: UDP::Init bind&listen port=5550
[1694821965.339048][33:35] CHIP:IN: UDP::Init bound to port=5550
[1694821965.339071][33:35] CHIP:IN: TransportMgr initialized
[1694821965.363320][33:35] CHIP:CSM: FindOrEstablishSession: PeerId = [1:3000B760B7D068C9]
[1694821965.363371][33:35] CHIP:CSM: FindOrEstablishSession: No existing OperationalSessionSetup instance found
[1694821965.363404][33:35] CHIP:DIS: OperationalSessionSetup[1:3000B760B7D068C9]: State change 1 --> 2
[1694821965.363442][33:35] CHIP:DIS: OperationalSessionSetup[1:3000B760B7D068C9]: State change 2 --> 1
[1694821965.363482][33:35] CHIP:CTL: Failed to open pairing window on the device. Status src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state
[1694821965.363508][33:35] CHIP:-: src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state at ../../examples/chip-tool/commands/pairing/OpenCommissioningWindowCommand.cpp:50
[1694821965.363531][33:35] CHIP:-: src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state at ../../examples/chip-tool/commands/pairing/OpenCommissioningWindowCommand.cpp:57
[1694821965.363899][33:33] CHIP:CTL: Shutting down the commissioner
[1694821965.364001][33:33] CHIP:CTL: Shutting down the controller
[1694821965.364046][33:33] CHIP:IN: Expiring all sessions for fabric 0x1!!
[1694821965.364065][33:33] CHIP:FP: Forgetting fabric 0x1
[1694821965.364094][33:33] CHIP:TS: Pending Last Known Good Time: 2023-09-02T01:00:09
[1694821965.364238][33:33] CHIP:TS: Previous Last Known Good Time: 2023-09-02T01:00:09
[1694821965.364261][33:33] CHIP:TS: Reverted Last Known Good Time to previous value
[1694821965.364301][33:33] CHIP:CTL: Shutting down the commissioner
[1694821965.364361][33:33] CHIP:CTL: Shutting down the controller
[1694821965.364414][33:33] CHIP:CTL: Shutting down the System State, this will teardown the CHIP Stack
[1694821965.364600][33:33] CHIP:DMG: All ReadHandler-s are clean, clear GlobalDirtySet
[1694821965.364719][33:33] CHIP:FP: Shutting down FabricTable
[1694821965.364786][33:33] CHIP:TS: Pending Last Known Good Time: 2023-09-02T01:00:09
[1694821965.364920][33:33] CHIP:TS: Previous Last Known Good Time: 2023-09-02T01:00:09
[1694821965.364976][33:33] CHIP:TS: Reverted Last Known Good Time to previous value
[1694821965.365416][33:33] CHIP:DL: writing settings to file (/tmp/chip_counters.ini-P1TTPK)
[1694821965.366142][33:33] CHIP:DL: renamed tmp file to file (/tmp/chip_counters.ini)
[1694821965.366220][33:33] CHIP:DL: NVS set: chip-counters/total-operational-hours = 0 (0x0)
[1694821965.366242][33:33] CHIP:DL: Inet Layer shutdown
[1694821965.366262][33:33] CHIP:DL: BLE shutdown
[1694821965.366281][33:33] CHIP:DL: System Layer shutdown
[1694821965.366649][33:33] CHIP:TOO: Run command failure: src/lib/dnssd/Discovery_ImplPlatform.cpp:689: CHIP Error 0x00000003: Incorrect state

Steps to reproduce the behavior

No response

Expected behavior

No response

Log files

See log files here:
https://csamembers.slack.com/archives/C05P2DKG0J0/p1695643214736949?thread_ts=1695229728.149429&cid=C05P2DKG0J0

and here:
https://csamembers.slack.com/archives/C05P2DKG0J0/p1695643291494439?thread_ts=1695229728.149429&cid=C05P2DKG0J0

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

[1.2 TE1] [TC-G-1.1]Test Harness fails to auto pair with a wifi-ble accessory.

From chip-certification-tool created by mideayanghui: CHIP-Specifications/chip-certification-tool#865

Description:

Click the Start to test, then fail to pair with accessory

Steps to reproduce:

Click the Start to test, then fail to pair with accessory

Logs:

Test Harness logs :

[TC-G-1.1]_[LQGd6XEO]_TH_log.log

DUT logs:

[TC-G-1.1]_[LQGd6XEO]_DUT_log.log

Additional Info:

Spec version: 2023 Fall 0.7
PICS: v1.8
PICS
Groups Cluster Test Plan XML.zip

https://github.com/CHIP-Specifications/chip-certification-tool.git
commit 68548cb, I have updated TH using cmd.

[Bug] TC-TCTL-1.1 When TCTL.S.F02(A_STEP) & TCTL.S.F00(TN) are true, step3 is skipped

From chip-certification-tool created by jeffzhong1: CHIP-Specifications/chip-certification-tool#984

Describe the bug

TCTL.S.F02(A_STEP) & TCTL.S.F00(TN) are true.
Step 3d should be performed according to the label description, but all step 3 is skipped

YAML
The description of the -label is inconsistent with PICS: TCTL.S.F02 && !TCTL.S.F00

label:
"Step 3d: TH reads from the DUT the FeatureMap attribute. bit 2: SHALL
be 1 if and only if TCTL.S.F02(A_STEP) & TCTL.S.F00(TN)"
command: "readAttribute"
attribute: "FeatureMap"
PICS: TCTL.S.F02 && !TCTL.S.F00

Steps to reproduce the behavior

No response

Expected behavior

No response

Log files

Test Step Not Applicable: Step 3d: TH reads from the DUT the FeatureMap attribute. bit 2: SHALL be 1 if and only if TCTL.S.F02(A_STEP) & TCTL.S.F00(TN) - Test step skipped: Step 3d: TH reads from the DUT the FeatureMap attribute. bit 2: SHALL be 1 if and only if TCTL.S.F02(A_STEP) & TCTL.S.F00(TN). TCTL.S.F02 && !TCTL.S.F00 == False

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

[TC-CC-6-x] Failing when we set OnOffTransitionTime to non-zero

From chip-certification-tool created by pankore: CHIP-Specifications/chip-certification-tool#871

TC-CC-6.1, TC-CC-6.2 usually pass when we set OnOffTransitionTime to zero.
However, some devices might set this attribute to non-zero, which will cause it to fail at the last part.

Last part:

Set off command, but DUT delays due to non-zero OnOffTransitionTime
TH reads OnOff value, sees that it is still on => fail
After OnOffTransitionTime pass, DUT set to off

How should we handle this in this scenario?

PICS Tool validation issue for TSTAT.S.F00 and/or TSTAT.S.F01

From chip-certification-tool created by paulux15: CHIP-Specifications/chip-certification-tool#875

Summary Title: PICS Tool validation issue for TSTAT.S.F00 and/or TSTAT.S.F01

There is a problem with the validation of PICS TSTAT.S, especially with the 2 items: TSTAT.S.F00 and TSTAT.S.F01
You cannot define you Thermostat as a Heating or a Cooler device only, as specified in the Specs.

Description:

If you select TSTAT.S.F00 (=true) and unselect TSTAT.S.F01 (=false), you get 2 errors after validation!

Normally you should be able to choose either F00 (Heating only), F01 (Cooling only) or both (Heating and Cooling).
I have the impression that there is a dependency issue somewhere.
With PICS Tool v1.6.4, 'Validate All' gives you 2 errors (F00 and F01), but you get the list of TCs to run.
While with the latest PICS Tool v1.6.8, you get these 2 errors without the display of the list of TCs.

Steps to reproduce:

Validate your Thermostat Cluster Test Plan.xml with TSTAT.S.F00 (=true) and TSTAT.S.F01 (=false), or the opposite.
You systematically will get 2 errors. TSTAT.S.F05 (Auto mode) can be unset (=false).

Logs:

Additional Info:

The versions of Tools used:  'XML_PICS Version Master - Ver -17' and PICS Tool v1.6.8 and v1.6.4

[Bug] TC-TIMESYNC-2.3 fails if TIMESYNC.S.F03 is not supported

From chip-certification-tool created by pankore: CHIP-Specifications/chip-certification-tool#967

Describe the bug

If declared not supporting TIMESYNC.S.F03 and disable TrustedTimeSource (TIMESYNC.S.A0003), the automated script will fail.
PICS is not indicated in the test plan, and automated script does not include PICS.

Step requiring PICS 1, 3, 4, 5

Test-plan:
https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/cluster/time_synchronization.adoc#tc-timesync-2-3-settrustedtimesource-command-with-dut_server

Steps to reproduce the behavior

No response

Expected behavior

No response

Log files

Test Harness log:
TC-TIMESYNC-2.3.log

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

Manual test case enhancement (Frontend)

From chip-certification-tool created by krypton36: CHIP-Specifications/chip-certification-tool#869

Manual test case enhancement:

Update the frontend to support chip-tool commands.

Description:

- Add REST Api to backend that hooks into chip-tool.py and allows for execution of chip-tool command.
- Use `send_websocket_command` in chip-tool.py to forward the command execution in the chip-tool binary
- Add Frontend interface for the user to be able to execute the chip-tool command and calling the REST API.

Additional Notes:

Phase2 Branch where the API is available.

Example Command

from app.chip_tool.chip_tool import ChipTool
import asyncio
import json

async def run_chip_tool_cmd():
    chip_tool = ChipTool()
    await chip_tool.start_container()
    chip_tool_command = "onoff read on-off 1234 1"
    response = await chip_tool.send_websocket_command(chip_tool_command)
    json_payload = json.loads(response)

asyncio.run(run_chip_tool_cmd())

[Bug] Link goes to a protected google document

Describe the bug

In this file: https://github.com/project-chip/certification-tool/blob/main/docs/Matter_User_Guide/Matter_User_Guide.md

There is a link to this document: https://docs.google.com/document/d/1bYDjmGrj6dPnu0KHoVRaDF7YhsFzGslw/edit#heading=h.wkkktfl30jjn

[Bug] UI Error During Multiple Test Case Validation: Code 1011 - Keepalive Ping Timeout and Missing Close Frame in TV-APP

From chip-certification-tool created by Rajashreekalmane: CHIP-Specifications/chip-certification-tool#986

Describe the bug

If we run the UI test runner by selecting one or two test cases, there are no issues. However, when selecting more than five test cases or two test cases with a larger number of test steps, the test runner ends with the following error: 'Sent 1011 (unexpected error) keepalive ping timeout; no close frame received.

Steps to reproduce the behavior

From the Test-Harness user interface loaded the PICS xml file to select the test cases.
Selected all the Concentration measurement test case to executed.
Executed the below mentioned command to put DUT into a commissionable state, ./chip-tv-app
Click the Start button on Test-Harness user interface.

Log files

automated_2023_10_05_11_24_49.log

PICS file

Media Cluster Test Plan.txt

Screenshots

Environment

Version : TH Fall2023
Sha : 3f84bff5
DUT : tv-app

Additional Information

No response

[Bug] TH hangs after closing a prompt from a simulated test

From chip-certification-tool created by ccruzagralopes: CHIP-Specifications/chip-certification-tool#983

Describe the bug

The test execution in TH is hanging when the user closes a prompt from a test step in a simulated test case.

Steps to reproduce the behavior

Run a simulated test case in TH (e.g.: TC-LVL-2.3)
Complete pairing as instructed during test suite setup
Close a prompt from a test step

Expected behavior

I expected the test case to finish with an error, which is what happens for manual test cases and chip-tool test cases

Log files

No response

PICS file

PICS.zip

Screenshots

Environment

TH Fall2023

Additional Information

No response

[ACL-2.7, 2.8, 2.10] TH2 command failing even after successful commissioning

From chip-certification-tool created by pankore: CHIP-Specifications/chip-certification-tool#872

TC-ACL-2.7
TC-ACL-2.8
TC-ACL-2.10
Above test cases require TH2 commissioning.

Description:

TH1 open commissioning window - ok
TH2 commission DUT - ok, but we don't know what nodeID to assign, we set it to 2 which is shown in the prompt
Subsequent auto commands ran by TH2 do not know that we assign nodeID to 2, causing failure for TH2 commands
Is there any way to configure the nodeID?

Logs: TC-ACL-2-8_2023_06_16_14_37_29.log

Additional Info:

v2.8.1 test harness

[Bug] Error while parsing YAML Files

From chip-certification-tool created by rquidute: CHIP-Specifications/chip-certification-tool#959

Describe the bug

The following test cases are not being displayed in TH because they are falling when parsing the test case.

Test_TC_SC_4_2
Test_TC_SC_4_3
Test_TC_SC_4_4
Test_TC_SC_4_6

In all YAML script files, the disabled property has and extra . at the end of the line, ex: disabled: true.

Steps to reproduce the behavior

1- Launch TH
2-In logs for each test case the following line is printed in backend log:
ERROR | 2023-09-06 18:50:23.055 | test_collections.yaml_tests.sdk_yaml_tests:_parse_all_sdk_yaml:82 | Error while parsing YAML File: /app/backend/test_collections/yaml_tests/yaml/sdk/Test_TC_SC_4_2.yaml

Expected behavior

TH launches with no parsing errors.

Log files

ERROR    | 2023-09-06 21:12:06.363 | test_collections.yaml_tests.models.yaml_test_parser:parse_yaml_test:57 | 1 validation error for YamlTest
tests -> 0 -> disabled
  value could not be parsed to a boolean (type=type_error.bool)
ERROR    | 2023-09-06 21:12:06.366 | test_collections.yaml_tests.sdk_yaml_tests:_parse_all_sdk_yaml:82 | Error while parsing YAML File: /app/backend/test_collections/yaml_tests/yaml/sdk/Test_TC_SC_4_6.yaml
ERROR    | 2023-09-06 21:12:26.433 | test_collections.yaml_tests.models.yaml_test_parser:parse_yaml_test:57 | 1 validation error for YamlTest
tests -> 0 -> disabled
  value could not be parsed to a boolean (type=type_error.bool)
ERROR    | 2023-09-06 21:12:26.435 | test_collections.yaml_tests.sdk_yaml_tests:_parse_all_sdk_yaml:82 | Error while parsing YAML File: /app/backend/test_collections/yaml_tests/yaml/sdk/Test_TC_SC_4_4.yaml
ERROR    | 2023-09-06 21:12:31.285 | test_collections.yaml_tests.models.yaml_test_parser:parse_yaml_test:57 | 1 validation error for YamlTest
tests -> 0 -> disabled
  value could not be parsed to a boolean (type=type_error.bool)
ERROR    | 2023-09-06 21:12:31.287 | test_collections.yaml_tests.sdk_yaml_tests:_parse_all_sdk_yaml:82 | Error while parsing YAML File: /app/backend/test_collections/yaml_tests/yaml/sdk/Test_TC_SC_4_3.yaml
ERROR    | 2023-09-06 21:12:35.889 | test_collections.yaml_tests.models.yaml_test_parser:parse_yaml_test:57 | 1 validation error for YamlTest
tests -> 0 -> disabled
  value could not be parsed to a boolean (type=type_error.bool)
ERROR    | 2023-09-06 21:12:35.891 | test_collections.yaml_tests.sdk_yaml_tests:_parse_all_sdk_yaml:82 | Error while parsing YAML File: /app/backend/test_collections/yaml_tests/yaml/sdk/Test_TC_SC_4_2.yaml

PICS file

N/A

Screenshots

N/A

Environment

Test Engine version is TH Fall2023
Test Engine SHA is 22538eb7
Test Engine SDK SHA is 5ccb774

Additional Information

No response

[Enhancement] Add a copy button to the manual test pop-up window to simplify manual tests

From chip-certification-tool created by mkardous-silabs: CHIP-Specifications/chip-certification-tool#425

Problem

As of now, to copy a command, the pop-up window needs to be on the edge of the screen for the text selection to register.
This makes it annoying to copy and paste commands durant the manual tests

Proposed Solution

Add a copy button in the pop-up window to accelerate test running for manual commands
This would be a great post-SVE1 feature.

[ TC-AUDIOOUTPUT-1.8] Test case is failing due to featuremap attribute value mismatch

From chip-certification-tool created by Rajashreekalmane: CHIP-Specifications/chip-certification-tool#988

Describe the bug

As per test-plan step 2, Test case is failing due to FEATURE-MAP attribute value mismatch.

Steps to reproduce the behavior

From the Test-Harness user interface loaded the Media.xml PICS file to select the test cases.
Selected the TC-AUDIOOUTPUT-1.8 Automated test case to executed.
Executed the below mentioned command to put DUT into a commissionable state, ./chip-tv-app
Click the Start button on Test-Harness user interface.

Expected behavior

TC should execute successfully without error

Log files

UI_Test_Run_2023_10_06_17_33_51(1).log

PICS file

Media Cluster Test Plan.txt

Screenshots

Environment

Version: TH Fall2023
Sha: 3f84bff5

Additional Information

No response

[Bug] The submit button is not appearing

Describe the bug

The submit button is not appearing in the first step of the Test Case: TC-DD-1.1

Steps to reproduce the behavior

Run the TH
Select python_tests suite
Select TC-DD-1.1
Run

Expected behavior

The button should appear

Log files

No response

PICS file

Color Control Cluster .xml
On-Off Cluster .xml

Screenshots

Environment

TH environment built from the public repository

Additional Information

No response

[Matter1.2 TE1-2.9]Test Harness fails to pairing accessory in interactive mode

From chip-certification-tool created by mideayanghui: CHIP-Specifications/chip-certification-tool#866

Description:

Test Harness fails to pairing accessory in interactive mode

Steps to reproduce:

step 1 : ./chip-tool interactive
step 2: . pairing ble-wifi 1 smart-US xxxx 31397237 820 --paa-trust-store-path /var/paa-root-certs
Finally, fail to commission. Failed in verifying 'Attestation Information' command received from the device: err 101.

Logs:

 Chip-tool  logs:

test_chip_interactive.log

 DUT logs:

COM10_2023-06-08_16-09-30-DUT.log

Additional Info:

https://github.com/CHIP-Specifications/chip-certification-tool.git commit id: dac897864388

If we use ./chip-tool ble-wifi 1 smart-US xxxx 31397237 820 --paa-trust-store-path /var/paa-root-certs, it can success to commssion.
Chip-tool logs:
test_chip.log

DUT logs:
COM10_2023-06-08_16-05-39-DUT.log

[Feature] Merge submodules into single tool repo

Feature description

Merge backend, frontend, cli into this repo.

Use Cases

These submodules provide little value, and only add overhead, we should merge these together.

Test Cases

No response

Additional Information

No response

[Bug] TC-CADMIN-1.6 consistently failed during clean up.

From chip-certification-tool created by yinyihu-silabs: CHIP-Specifications/chip-certification-tool#995

Describe the bug

While running TC-CADMIN-1.6, after Step 12, during unpairing chip_tool_from device, test ran into error

INFO       | 2023-10-19 17:37:34.887718 | Executing Test Step: Step 12: TH_CR2 starts a commissioning process on DUT_CE
INFO       | 2023-10-19 17:38:05.169099 | Test Step Completed [PASSED]: Step 12: TH_CR2 starts a commissioning process on DUT_CE
INFO       | 2023-10-19 17:38:05.179263 | Executing Test Step: DUT_CE is commissioned by TH_CR2
INFO       | 2023-10-19 17:38:05.184443 | Test Step Completed [PASSED]: DUT_CE is commissioned by TH_CR2
INFO       | 2023-10-19 17:38:05.191108 | Test Case Completed[PENDING]: TC-CADMIN-1.6 (Semi-automated)
INFO       | 2023-10-19 17:38:05.198615 | Unpairing chip_tool from device
ERROR      | 2023-10-19 17:38:15.587297 | Test Suite Error: Error occurred during cleanup of test suite FirstChipToolSuite. Connecting to ws://172.18.0.1:9002 failed.
INFO       | 2023-10-19 17:38:15.602690 | Test Suite Completed [ERROR]: FirstChipToolSuite
INFO       | 2023-10-19 17:38:15.616839 | Test Run Completed [ERROR]

Other cadmin tests such as 1.10, 1.16, 1.23 do not experience the same issue

Steps to reproduce the behavior

No response

Expected behavior

No response

Log files

cadmin-1-6_2023_10_19_13_32_32.log

PICS file

No response

Screenshots

Environment

Additional Information

No response

[TC-SM-1.1] Step 5 states incorrect cluster requirements

From chip-certification-tool created by CKnoerzer: CHIP-Specifications/chip-certification-tool#930

Describe the bug

In [TC-SM-1.1] Step 5 you will find in Column G:

On TH(chip-tool), Validate all the mandatory clusters(Basic Information(40), Access Control(31), Group Key Management(63) General Commissioning(48), Administrator Commissioning(60), Node Operational Credentials(62), Localization Configuration(43), Time Format Localization(44), Unit Localization(45), General Diagnostics(51), and ICD Management(70)) for the Root Node Device Type exist on the Root Node endpoint(endpoint 0)

But Localization Configuration (43), Time Format Localization(44), Unit Localization(45) and ICD Management(70) are not mandatory for all devices. They are only mandatory under some conditions.

| ID | ClusterName | Client/Server | Quality | Conformance
| 0x0028 | Basic Information | Server | I | M
| 0x001F | Access Control | Server | I | M
| 0x002E | Power Source Configuration | Server | I | O, D
| 0x0038 | Time Synchronization | Server | I | P, O
| 0x003F | Group Key Management | Server | I | M
| 0x0030 | General Commissioning | Server | I | M
| 0x0031 | Network Commissioning | Server | | !CustomNetworkConfig
| 0x003C | Administrator Commissioning | Server | I | M
| 0x003E | Node Operational Credentials | Server | I | M
| 0x002B | Localization Configuration | Server | I | LanguageLocale
| 0x002C | Time Format Localization | Server | I | TimeLocale
| 0x002D | Unit Localization | Server | I | UnitLocale
| 0x0033 | General Diagnostics | Server | I | M
| 0x0032 | Diagnostic Logs | Server | I | O
| 0x0034 | Software Diagnostics | Server | I | O
| 0x0037 | Ethernet Network Diagnostics | Server | I | [Ethernet]
| 0x0036 | Wi-Fi Network Diagnostics | Server | I | [Wi-Fi]
| 0x0035 | Thread Network Diagnostics | Server | I | [Thread]
| 0x0046 | ICD Management | Server | I | SIT | LIT
| 0x0038 | Time Synchronization | Server | I | O

Steps to reproduce the behavior

No response

Expected behavior

No response

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

Received "Failed to perform commissioning step 20" error

From chip-certification-tool created by njames-11: CHIP-Specifications/chip-certification-tool#925

Describe the bug

Setup details:

Rpi4 with THv2.8.1
DUT is a Matter over Thread device

The issue occurs when running any auto test cases using Test Harness.
Please note that:
a. If using "chip-tool" command, then commissioning passes.
b. If DUT is a Matter over Wifi device, then commissioning passes

Configuration on TH:

{
"config": {
"network": {
"wifi": {
"ssid": "testharness",
"password": "wifi-password"
},
"thread": {
"rcp_serial_path": "/dev/ttyACM0",
"rcp_baudrate": 115200,
"on_mesh_prefix": "fd11:22::/64",
"network_interface": "wlan0",
"dataset": {
"channel": "15",
"panid": "0x1234",
"extpanid": "1111111122222222",
"networkkey": "00112233445566778899aabbccddeeff",
"networkname": "Matter-Thread"
},
"otbr_docker_image": null
}
},
"dut_config": {
"discriminator": "2580",
"setup_code": "16016",
"pairing_mode": "ble-thread",
"chip_tool_timeout": null
},
"test_parameters": {
"endpoint": 0
}
}

Steps to reproduce the behavior

Start TH
Load PICS
Set configuration as above
Select TC-ACL-1.1
Power cycle on DUT
Start test

Expected behavior

No response

Log files

PFA for:

TH log
DUT log
Rpi4 manual pairing log
DUT manual paring log

UI_Test_Run_2023_08_05_17_23_02.log
UI_Test_Run_2023_08_05_17_23_02-DUT.log
Manual-Commissioning-Rpi4.log
Manual-Commissioning-DUT.log

PICS file

No response

Screenshots

No response

Environment

TH v1.8.1
Patch applied

Additional Information

No response

[Bug] Test case gets failed by throwing an error

From chip-certification-tool created by Saravana-kr22: CHIP-Specifications/chip-certification-tool#961

Describe the bug

While running the TC-CADMIN-1.21 and 1.22 test case in Step 4: TH_CR1 reads the window status to verify the DUT_CE window is closed test case is failed with the error:

Test Step Error: Error occurred during execution of test case TC-CADMIN-1.21. no close frame received or sent

Steps to reproduce the behavior

From the Test-Harness user interface loaded the PICS xml file to select the test cases.
Select TC-CADMIN-1.21 or TC-CADMIN-1.22 test case to executed.
Advertise the Dut using the command ./chip-all-clusters-app
Click the Start button on Test-Harness user interface.

Expected behavior

Test case should executed without any error.

Log files

UI-log: cadmin-log.zip
UI-reports: cadmin-report.zip

PICS file

cadmin-pics.zip

Screenshots

Environment

TH Version:TH Fall2023
OS: Ubuntu
Browser: Chrome

Additional Information

No response

[1.2 TE1][TC-MOD-1.2][FAILED]Step 3: TH reads the OnMode attribute from the DUT. Step 4: TH reads the StartUpMode attribute from the DUT.

From chip-certification-tool created by luojr9: CHIP-Specifications/chip-certification-tool#867

Summary Title:

 [1.2 TE1][TC-MOD-1.2][FAILED]Step 3: TH reads the OnMode attribute from the DUT. 
Step 4: TH reads the StartUpMode attribute from the DUT.

Description:

We are testing Dishwasher. But TH had no test case DishwasherModeSelect. So we made WiFi module support Model Select. Should module add attribute OnMode  and StartUpMode?

Steps to reproduce:

1. pair DUT.
2. TH run test case TC-MOD-1.2
3. Pi run ./chip-tool modeselect read on-mode 1 1
4. Pi run ./chip-tool modeselect read start-up-mode 1 1

Logs:

[TC-MOD-1.2]_[jLUniBv1]_TH_log.log
[TC-MOD-1.2]_[jLUniBv1]_DUT_log.log
[TC-MOD-1.2]_[jLUniBv1]_CHIPTool_log.txt

Additional Info:

TC-MOD-1.2 result upload fail. Show error log: Matter Master TCID List is required

[Bug] manual verification test steps will not appear if a PICs file is not loaded previously

From chip-certification-tool created by gvargas-csa: CHIP-Specifications/chip-certification-tool#939

Describe the bug

I have been using the version: TH Fall 2023 - beta3 and I realized that if a PICs file is not loaded previously, the dialogs in the frontend to help the user with what manual verification test steps (commands) they have to send will not appear. But if you use a PICs file it doesn't matter if it doesn't match the tests you run, the dialogs appear again.

In my case I have been using a test plan for the DoorLock and the test that I ran it was TC-FAN-1.1 (Semi-Automated)

Steps to reproduce the behavior

Create a new project in the Test Harness Frontend
Setup the project with your values
When is created the project, Go to edit the project and verify it does not have a PICs File loaded.
Now go to test-run in the project
Create a New Test Run
Add a new operator name
Please select a test case TC-FAN-1.1 (Semi-Automated) there is under the SDK YAML Tests -> FirstChipToolSuite
Run in your terminal or raspberry pi the ./chip-all-cluster-app this depends on your setup.
Now go to start

Expected behavior

The manual verification tests steps must be appears in a dialog for help to the user to know what kind of commands should be use to send to the chip-tool to continues with the test.

Log files

No response

PICS file

Door lock Test Plan.xml.zip

Screenshots

TH.Fall.2023.-.No.PICs.mov

TH.Fall.2023.-.PICs.mov

Environment

TH Version: TH Fall 2023 - Beta3
OS: macOS 13.4.1 (c)
Browser: Safari 16.5.2

Additional Information

https://csamembers.slack.com/archives/C03MA7WR7Q8/p1691639376979739

Add SDK SHA also to TH UI

From chip-certification-tool created by raju-apple: CHIP-Specifications/chip-certification-tool#724

Request to add the SDK SHA also in the TH UI

TC-CGEN-2.4 PASESession timed out at step 3

From chip-certification-tool created by PSONALl: CHIP-Specifications/chip-certification-tool#883

Describe the bug

While running python inside docker test case PASESession timed out at step 3 for the test case CGEN-2.4
This test case works well with chip-tool

Steps to reproduce the behavior

start the test harness, put the DUt in commissioning mode
Start CGEN-2.4 python inside docker test

Expected behavior

The test case should pass and DUT return the expected error codes on receiving set-regulatory-config command

Log files

DUT logs esp32-cgen-2.4.txt
TH logs th-cgen-2.4.txt

PICS file

NO PICS required

Screenshots

No response

Environment

TH version - 2.9 beta1

Additional Information

No response

Require Project id in `TestRunExecutionCreate` schema, once UI support project management

From chip-certification-tool created by mikaelhm: CHIP-Specifications/chip-certification-tool#818

null

[TC-WNCV-4.2] The test case is failing due to an OperationalStatus attribute value mismatch when executed along with other Window Covering Test cases

From chip-certification-tool created by Rajashreekalmane: CHIP-Specifications/chip-certification-tool#962

Describe the bug

[TC-WNCV-4.2] The test case is failing due to an OperationalStatus attribute value mismatch when executed along with other Window Covering Test cases but It passes when executed individually

PFA Screenshot of individual Run

PFA Screenshot when TC executed along with other Window Covering Test cases

Log files

PFA log of individual Run

UI_Test_Run_2023_09_07_16_08_40(1).log

PFA log when TC executed along with other Window Covering Test cases
UI_Test_Run_2023_09_07_16_06_07.log

PICS file

Window Covering Cluster Test Plan.txt

Additional Info:
Version: TH Fall2023
Sha: 58a51bef

Investigate the possibility of switching SDK simulated apps between test steps in the same test case

From chip-certification-tool created by fabiowmm: CHIP-Specifications/chip-certification-tool#684

Some test cases specify application nodes with different capabilities at different test steps. Is it possible to have the TH backend to launch different simulated apps at different steps of the test? Can we recommission in the middle of a test case execution? Can we keep the same commissioning state between apps? Do we need SDK changes?

[TC-CC-3.2] - Test Case failing due to Cutrrent Hue attribute value mismatch

From chip-certification-tool created by Rajashreekalmane: CHIP-Specifications/chip-certification-tool#649

Description
[TC-CC-3.2] - Test Case failing due to Cutrrent Hue attribute value mismatch at step 2d

Additional Info :
TestHarness:
Version: v2.7Beta2
Sha: 1aff1e6

PFA log below
CC.log

[Bug] Attempting to upload a manual log file on TC-DGTHREAD-2.1 results in a "Failed to connect' error and partial log

From chip-certification-tool created by jrhees-cae: CHIP-Specifications/chip-certification-tool#970

Describe the bug

When attempting to upload a manual log file on TC-DGTHREAD-2.1 we receive a "Failed to connect' error and only a portion of the log is included.

Steps to reproduce the behavior

Run TC-DGTHREAD-2.1
At end of test, upload attached log file

Log file can be found here: https://csamembers.slack.com/files/U037HSS0V2P/F05SSE1DTF0/tc-dgthread-2.1.txt

Expected behavior

No response

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

[Bug] Test case gets failed by throwing an error

From chip-certification-tool created by KishokG: CHIP-Specifications/chip-certification-tool#957

Describe the bug

While executing more number of test cases combinedly, the test case gets fail after completion of some test case by throwing below error

-Test Step Error: Error occurred during execution of test case TC-OZCONC-1.1. sent 1011 (unexpected error) keepalive ping timeout; no close frame received

Steps to reproduce the behavior

From the Test-Harness user interface loaded the PICS xml file to select the test cases.
Selected all the Concentration measurement test case to executed.
Executed the below mentioned command to put DUT into a commissionable state, ./chip-all-clusters-app
Click the Start button on Test-Harness user interface.

Expected behavior

TC should execute successfully without error

Log files

Execution Logs: CONCENTRATION_MEASUREMENT.log
Back-end Log: Backend log.txt

PICS file

Concentration measurement PICS.zip

Screenshots

Environment

Version: TH Fall2023
Sha: 58a51bef

Additional Information

No response

[1.1] Moving pairing / commissioning outside of test suite run

From chip-certification-tool created by raju-apple: CHIP-Specifications/chip-certification-tool#584

Will also unblock the following 2 issues :

Sorting TCIDs
Removing fabric after each test run

[1.1] Have a way to externally inject a python script

From chip-certification-tool created by raju-apple: CHIP-Specifications/chip-certification-tool#587

For some semi-automated test cases where we have a user-prompt for a reboot test case with the text "Reboot your DUT"
We should allow the vendor / ATL to write a reboot.py script and that script should be able to provide the "Y" or whatever the TH is waiting for once it's done to continue further . So this in short can convert a few of the semi-automated tests into fully automated ones.