microsoft / test-netstack

This tool tests the native, synthetic, and hardware network data path

Test-NetStack: A network integration testing tool

Synopsis

Test-NetStack is a PowerShell-based testing tool that performs ICMP, TCP, and RDMA traffic testing of networks. Test-NetStack can help identify potential network (fabric and host) misconfiguration or operational instability.

Test-NetStack can validate the various network data paths on Windows, testing native, synthetic, and hardware offloaded (RDMA) data paths for issues with:

• Connectivity
• Packet fragmention
• Low throughput
• Congestion

Test Details

Test-NetStack first performs connectivity mapping across a cluster, specific nodes, or IP targets then tests:

- Stage1: ICMP Connectivity, Reliability, and PMTUD
- Stage2: TCP Stress 1:1
- Stage3: RDMA Connectivity
- Stage4: RDMA Stress 1:1
- Stage5: RDMA Stress N:1
- Stage6: RDMA Stress N:N

For more information, run:

help Test-NetStack

Install the Tool

Install-Module Test-NetStack

Run the Tool

$NetStackResults = Test-NetStack

$NetStackResults = Test-Netstack -Nodes 'Node1', 'Node2', 'NodeN'

$NetStackResults = Test-Netstack -IPTarget '192.168.1.1', '192.168.1.2', '192.168.1.3', '192.168.1.4'

Reviewing Results

$NetStackResults.Stage1 | ft *

$NetStackResults.Stage2 | ft *

$NetStackResults.Stage3 | ft *

$NetStackResults.Stage4 | ft *

$NetStackResults.Stage5 | ft *

$NetStackResults.Stage6 | ft *

Testable vs Disqualified Networks

Test-NetStack will identify networks that can and cannot be tested. To review the Test-NetStack networks that will be tested, use:

$NetStack.TestableNetworks

To review the Test-NetStack networks that cannot be tested.

$NetStack.DisqualifiedNetworks

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

test-netstack's Issues

MTU Check Recommendations

When MTU mismatches occur we should recommend matching. In the below case, Node1 (Nic1 and 2) both had 9014 mtu size while node 2 had 1514. This wasn't identified as an issue.

Rename Module and files to reflect Test-NetStack module name

Path vs Stage Status

StageStatus should have it's own property. Subproperties should define each stage status and be a rollup of all the paths tested in the stage;

Use the "PathStatus" property for individual paths inside the stage

Disqualify network if all nodes don't have a member of the network

Consider this situation where ip addresses were improperly distributed to nodes. 1 Node has 2 IPs in the same network, while another node doesn't have any ips in that network. This network should be disqualified.

Use AdapterAdvancedProperties to identify RDMA Protocol

The current mechanism to specify the RDMA Protocol (RoCE vs iWARP) does not scale to include additional vendors where they may include multi-protocol options. E.g. Intel and Marvell both support RoCE and iWARP.

A more appropriate mechanism would be to use Get-NetAdapterAdvancedProperty on the specific adapter and look for the NetworkDirectTechnology type. In the future this will be required to be non-zero. Until then, we should just default to the first option in the list for that adapter (e.g. if it specifies RoCE or RoCEv2, then choose RoCEv2 or if iWARP and RoCE, default to iWARP.

https://docs.microsoft.com/en-us/windows-hardware/drivers/network/inf-requirements-for-ndkpi

Add additional RDMA counters to ascertain health

To include latency, connection errors, etc. Everything we discussed :-)

Randomize buffer size to reflect SMB/live migration

In Stage 2, use CTS Traffic's randomize buffer size option to reflect traffic generated by SMB and live migration

Move prerequisites to prerequisites.psm1

Prerequisites function in prerequisites.psm1 should be run at the beginning of each run to validate requirements like

• Firewall openings
• NDKPerf Version

Should be run by the selected stages. E.g. if stage 1 and 2 are selected unique requirements for stage 3 - n shouldn't be tested.

Establish interface viability check in ndk perf n:1 stage

CTSTraffic path is manually specified in tcp.psm1

Needs to rather do something like this:

$ModuleBase = (Get-Module Test-NetStack -listAvailable | Select -First 1).ModuleBase

Join-Path $ModuleBase -ChildPath tools\cts...

Improve test speed with runspaces

Practical example

https://learn-powershell.net/2016/04/22/speedy-ping-using-powershell/

Outsource recommendations

Similar to how localication is implemented in separate PSD1 files, separating the recommendations into a separate file makes it easier to read, debug, etc.

APIPA Should be removed

In this section, we retrieve all adapters reported by Get-NetIPAddress which includes tunnel adapters that don't appear in Get-NetAdapter (causing an error).

Test-NetStack/helpers/internal.psm1

Line 142 in a5d7939

$AdapterIP = Get-NetIPAddress -IPAddress $IP -AddressFamily IPv4 -SuffixOrigin Dhcp, Manual -AddressState Preferred, Invalid, Duplicate |

Simple fix (Any reason not to do this - APIPA is not production supported):

$AdapterIP = Get-NetIPAddress -IPAddress $IP -AddressFamily IPv4 -SuffixOrigin Dhcp, Manual -AddressState Preferred, Invalid, Duplicate | Select InterfaceAlias, InterfaceIndex, IPAddress, PrefixLength, AddressState

        **# Remove APIPA
        $AdapterIP = $AdapterIP | Where IPAddress -NotLike '169.254.*'**

$NetAdapter = Get-NetAdapter -InterfaceIndex $AdapterIP.InterfaceIndex

Logfile should be written as progress is made

For diagnostic purposes, we should write the logfile out as we go, so we can see progress and understand where (when/if) we get stuck running Test-NetStack.

E2E Integration

Assert-RDMA.ps1 does not appear to be needed

Is this file needed? Otherwise we should remove it.

PAHostVNICs should be limited to Stage1 Only

PA Host vNICs don't have VMMQ enabled so will never fill up/saturate a TCP Pipe or use RDMA.

Net-Net, they should be excluded from at least Stage 2 - 6

On-screen output returns in the wrong order

Output appears out of order, 71.11 is started, but then 71.12 reports as completed before starting 71.12

Add -ErrorAction SilentlyContinue

Specifically on setup files (setup and localsetup.ps1); may be other places where this is necessary

Stage# vs TestName

Do we have any dependencies on the stage# notation in NetStackResults? It seems rather than call an arbitrary stage number, we should provide a name. Take this for example, it's unclear what this was without understanding the tool. Perhaps we should call this NDKPing, or Another TCPStressN1, etc.

Enable connectivity view only

In the recent example where a node didn't have an IP address in a specific network, we had to try multiple times to test the connectivitymap. to do this, we had to run a full stage and get the output. It would be better to offer a switch like -ConnectivityMapOnly.

Run prerequisite tests prior to beginning

One obvious check is to validate that ICMP is working

If not, then:

MTU check should start from the NIC value

Since each tests run for each NIC, we should be able to ping -l -f where the frame size is equivalent to the *JumboPacket value from Get-NetAdapterAdvancedProperty. If a failure occurs, iterate down until the mtusize is identified.

Failure scenario is where different mtu values exist for each adapter in a specified connectivity map

Need interface name/index to easily identify sender info in stage results

Need to add interface name or index to easily identify the sender adapters.

Remove Copy-File from Stage2

No need to do this, prerequisites will ensure that all files exist (and same version) on all machines being tested.

Build does not pull in latest CTS-Traffic from its repo

Stage should be in the on-screen output

If some cases, it's difficult to identify what stage we're in because the output is covered. Would be better to just output this on screen as we go.

RDMA Stage 3 doesn't report failure

RDMA is disabled on all adapters. This section doesn't run at all but reports pass. The result is that it continues going onto Stage 4 despite it not having any enabled adapters.

Suggestion: If no adapters report RDMA Enabled, create an object with the interface and path status of fail for consistency with other stages.

Determine if Hyper-V is needed and only use commands if needed

Please see image below. If Hyper-V is not installed on the system, we should not call these commands.

Remove all local paths from Test-NetStack

Use something like this

$ModuleBase = (Get-Module Test-NetStack -ListAvailable).ModuleBase
Join-Path $ModuleBase -ChildPath...

Decide where to put tests

Currently the test file is in the main folder, however there is a folder with an empty file under Tests > global...ps1

We should just pick one final location and eliminate the other. IMHO, we should keep the Tests > testfile.ps1 structure and move the tests out of the root of the module to remove additional perceived customer complexity.

mtu and connectivity tests do not need to test adapters on the same host

If the connectivity map contains more than one adapter on the same host, these adapters should not test each other. This unnecessarily adds time to the tests, but more importantly confuses the output.

Here you can see two adapters on AzStackHCI01 both with 9k MTUSize. Since they're on the same host, the traffic never leaves the host and is not testing/validating anything overly useful (you'll always be able to connect from the same host and MTUSize is not important on the same system because it doesn't need to be segmented to fit on the fabric).

Use throw to identify failing scenarios rather than Exit

Using 'Exit' command will close a PowerShell Window if a customer uses the Shell (instead of vscode or ISE). This would be the most common case. Instead we should use 'throw' to send terminating errors or Write-Error and Write-Warning to identify non-terminating scenarios.

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/write-error?view=powershell-7

This was observed in the validation of $MachineList

EnableFirewallRules should enable iWARP FW rules

Will be needed for iWARP solutions

Feedback - Recommendation for NDK Ping

“verify that RDMA is enabled with Get-NetAdapterRDMA, that "NetworkDirect Functionality" is enabled from Get-NetAdapterAdvancedProperty -Name <> And that "NetworkDirect Technology" is the same value on each node and adapter combination being tested. If this system uses Data Center Bridging, consider running Validate-DCB prior to running this tool (Test-NetStack)”

ACTION REQUIRED: Microsoft needs this private repository to complete compliance info

There are open compliance tasks that need to be reviewed for your Test-NetStack repo.

Action required: 4 compliance tasks

To bring this repository to the standard required for 2021, we require administrators of this and all Microsoft GitHub repositories to complete a small set of tasks within the next 60 days. This is critical work to ensure the compliance and security of your microsoft GitHub organization.

Please take a few minutes to complete the tasks at: https://repos.opensource.microsoft.com/orgs/microsoft/repos/Test-NetStack/compliance

• The GitHub AE (GitHub inside Microsoft) migration survey has not been completed for this private repository
• No Service Tree mapping has been set for this repo. If this team does not use Service Tree, they can also opt-out of providing Service Tree data in the Compliance tab.
• No repository maintainers are set. The Open Source Maintainers are the decision-makers and actionable owners of the repository, irrespective of administrator permission grants on GitHub.
• Classification of the repository as production/non-production is missing in the Compliance tab.

You can close this work item once you have completed the compliance tasks, or it will automatically close within a day of taking action.

If you no longer need this repository, it might be quickest to delete the repo, too.

GitHub inside Microsoft program information

More information about GitHub inside Microsoft and the new GitHub AE product can be found at https://aka.ms/gim.

FYI: current admins at Microsoft include @dcuomo, @CoreNetBuilder, @manjushetkar

Capture packets sent per second data for CTS Traffic

Get packets sent per second counter during Stage 2, verify at least 600,000 packets per second sent by CTS Traffic

Unused code

Do we use this code at all? It doesn't appear we do. It appears to only appear (sort of) in 2 places that appear to be dead code paths ($thisInterfaceDetails.VLAN isn't used anywhere right?).

https://github.com/microsoft/Test-NetStack/blob/main/helpers/internal.psm1

Should bail if there are no testable networks

No testable networks but it continues

Failures should not return in NetStackResults when there are no failures

Add External Modules to PSD1 manifest

If a module cannot be downloaded via the PowerShell Gallery (powershellgallery.com) than it should be listed in the ExternalModules portion of the manifest

Stage2 CTSTraffic cmd does not push enough traffic to saturate link

@smoon193 - Could you please investigate this? Similar to @mgodfre3 issue. On my system, stage2 does not report any information.

Stage 7: N:N RDMA with UDP Congestion

If Stage 6 (N:N RDMA Stress) passes, add UDP N:N at the same time in a new stage. This adds unbounded non-flow controlled UDP traffic to the fabric, which could cause PFC on the RDMA traffic class.

Required Modules

• Add Pester 4.9.0
• Remove Test-RDMA as required (prevents module import)

Hyper-V Should not be required of the system running the tests.

It's reasonable to assume that this tool will be used for non-Hyper-V clusters. However, currently this is not possible. We should detect whether or not the SUTs have Hyper-V installed, and perform only the necessary tests

Fix ParameterSets

Parameter Sets currently only specify two options but much more exist.

Add destination machine to results

It becomes fairly difficult to track down destination IP addresses with multiple nodes in a cluster. This is compounded by the fact that storage adapter likely will not be registered in DNS.

Add ContinueOnFailure Switch and Stop execution if one of the lower phases fail

Move connectivity mapper to a separate function and condense into a single path

Once it's moved to a separate function, converge the IPTarget and Nodes code paths so the results look the same.

Several tests doesn't run if VLAN is 0 (stages 3 - 6)