microsoft / sdndiagnostics

SdnDiagnostics is a PowerShell module that is designed to simplify the diagnostic troubleshooting and data collection process when troubleshooting issues related to Microsoft Software Defined Network.

Project

SdnDiagnostics is a PowerShell module that is designed to simplify the diagnostic troubleshooting and data collection process when troubleshooting issues related to Microsoft Software Defined Network.

Please refer to the wiki on how to install and use SdnDiagnostics in your environment.

Package Statistics

Package Gallery	Total Downloads	Current Version
PowerShell Gallery
NuGet

GitHub Action Status

Actions	Current Status
Build Validation
Script Analyzer

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

To get started on contributing to this module, refer to the contributing guide on this project.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

sdndiagnostics's Issues

Enable NCRest and NCAdmin Credential Parameters

Need ability for operator of the module to define different credentials for the REST API endpoints vs credentials to access the SDN server/mux/gateway resources.

Ensure health states of MUXes

As part of the SDN VIP debugging, need to ensure that:
- MUX is online
- MUX Services are Running
- MUX provisioningState and configurationState are success
- MUX is peered w/ ToR Switches

Get-SdnVfpPortRule fails to return object

Describe the bug

"REMOVED","2023-09-06 17-34-58","Get-VfpPortRule","Warning","Unable to add Meterinfo to object. Failing back to use NoteProperty."
"REMOVED","2023-09-06 17-34-58","Get-VfpPortRule","Warning","Unable to add Counteroffloaded to object. Failing back to use NoteProperty."
"REMOVED","2023-09-06 17-34-58","Get-VfpPortRule","Warning","Unable to add Meterinfo to object. Failing back to use NoteProperty."
"REMOVED","2023-09-06 17-34-58","Get-VfpPortRule","Warning","Unable to add Counteroffloaded to object. Failing back to use NoteProperty."
"REMOVED","2023-09-06 17-34-58","Get-SdnVfpPortRule","Error","System.Management.Automation.RuntimeException: The property 'RuleData' cannot be found on this object. Verify that the property exists and can be set.
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)
   at System.Management.Automation.PSScriptCmdlet.RunClause(Action`1 clause, Object dollarUnderbar, Object inputToProcess)
   at System.Management.Automation.PSScriptCmdlet.DoEndProcessing()
   at System.Management.Automation.CommandProcessorBase.Complete()
at Get-VfpPortRule, C:\Program Files\WindowsPowerShell\Modules\SdnDiagnostics\SdnDiagnostics\modules\SdnDiag.Server\SdnDiag.Server.psm1: line 651
at Get-SdnVfpPortRule, C:\Program Files\WindowsPowerShell\Modules\SdnDiagnostics\SdnDiagnostics\modules\SdnDiag.Server\SdnDiag.Server.psm1: line 1938
at Show-SdnVfpPortConfig, C:\Program Files\WindowsPowerShell\Modules\SdnDiagnostics\SdnDiagnostics\modules\SdnDiag.Server\SdnDiag.Server.psm1: line 2490
at <ScriptBlock>, <No file>: line 1"
"REMOVED","2023-09-06 17-34-59","Get-VfpPortRule","Warning","Unable to add MSSdelta to object. Failing back to use NoteProperty."

Environment Details (please complete the following information):

• Windows Server OS: Windows Server 2019
• SdnDiagnostics Version: 2309

Additional context
Add any other context about the problem here.

Speed up data collection as part of Start-SdnDataCollection

Feature Request:
Need to speed up the data collection process to more effectively leverage the other hosts processing power. Data collection should be initiated remotely, then go into a monitoring state once it has started on all nodes to monitor and then pick up the logs for each remote job that completes.

install-sdndiagnostics failed when remote computer has multiple versions installed

Describe the bug
install-sdndiagnostics failed when remote computer has multiple versions installed

To Reproduce
Steps to reproduce the behavior:

1. Install multiple sdndiagnostics module on remote computer
2. call install-sdndiagnostic again
3. Failure received as below:

[WAC] Getting current installed version of SdnDiagnostics on remote nodes
Trace-Output : [WAC] System.Management.Automation.RuntimeException: Cannot convert value "System.Object[]" to type "System.Version". Error: "Input string was not in a correct format." ---> System.Management.Automation.PSInvalidCastException: Cannot convert value "System.Object[]" to type "System.Version". Error:
"Input string was not in a correct format." ---> System.FormatException: Input string was not in a correct format.
   at System.Version.VersionResult.SetFailure(ParseFailureKind failure, String argument)
   at System.Version.TryParseComponent(String component, String componentName, VersionResult& result, Int32& parsedComponent)
   at System.Version.TryParseVersion(String version, VersionResult& result)
   at System.Version.Parse(String input)
   --- End of inner exception stack trace ---
   at System.Management.Automation.LanguagePrimitives.ConvertViaParseMethod.ConvertWithoutCulture(Object valueToConvert, Type resultType, Boolean recursion, PSObject originalValueToConvert, IFormatProvider formatProvider, TypeTable backupTable)
   at CallSite.Target(Closure , CallSite , Object )
   at System.Management.Automation.Interpreter.DynamicInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   --- End of inner exception stack trace ---
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at Install-SdnDiagnostics, C:\Program Files\WindowsPowerShell\Modules\sdndiagnostics\2.2207.351.134440\modules\Utilities\public\Install-SdnDiagnostics.ps1: line 91
at <ScriptBlock>, <No file>: line 1
at <ScriptBlock>, <No file>: line 1
At C:\Program Files\WindowsPowerShell\Modules\sdndiagnostics\2.2207.351.134440\modules\Utilities\public\Install-SdnDiagnostics.ps1:134 char:59
+ ... {1}" -f $_.Exception, $_.ScriptStackTrace | Trace-Output -Level:Error
+                                                 ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Trace-Output

5. See error

Expected behavior
No errors. Install-SdnDiagnostics success

• SdnDiagnostics Version:
  2.2207.351.134440

Typo in wiki installation step

The cmdlet in wiki has extra "s". The correct cmdlet appears to be "Install-SdnDiagnostic"

Get-SdnInfrastructureInfo get exception HTTP 404 due to API version info missing

Describe the bug
Get-SdnInfrastructureInfo get exception due to API version info missing

To Reproduce
Steps to reproduce the behavior:

• Run Get-SdnInfrastructureInfo with no NC Rest Credential provided cause 401 error and API Version global variable overwritten to null
• Run Get-SdnInfrastructureInfo with correct NC Rest Credential. The command failed to retrieve Gateway, MUX and Host info with error 401

PS D:\Temp\SdnDiagnostics> Get-SdnInfrastructureInfo -NetworkController sdnexpnc01 -NcRestCredential (Get-Credential)

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential
Trace-Output : System.Net.WebException: The remote server returned an error: (404) Not Found.
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(WebRequest request)
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
AbsoluteUri:https://sdnexpnc.corp.contoso.com/networking//loadBalancerMuxes
at Get-SdnResource, D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnResource.ps1: line 61
at Get-SdnLoadBalancerMux, D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnLoadBalancerMux.ps1: line 31
at Get-SdnInfrastructureInfo, D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnInfrastructureInfo.ps1: line 66
at <ScriptBlock>, <No file>: line 1
At D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnResource.ps1:77 char:113
+ ... .Address.AbsoluteUri, $_.ScriptStackTrace | Trace-Output -Level:Error
+                                                 ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Trace-Output

Trace-Output : System.Net.WebException: The remote server returned an error: (404) Not Found.
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(WebRequest request)
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
AbsoluteUri:https://sdnexpnc.corp.contoso.com/networking//gateways
at Get-SdnResource, D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnResource.ps1: line 61
at Get-SdnGateway, D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnGateway.ps1: line 31
at Get-SdnInfrastructureInfo, D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnInfrastructureInfo.ps1: line 71
at <ScriptBlock>, <No file>: line 1
At D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnResource.ps1:77 char:113
+ ... .Address.AbsoluteUri, $_.ScriptStackTrace | Trace-Output -Level:Error
+                                                 ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Trace-Output

Trace-Output : System.Net.WebException: The remote server returned an error: (404) Not Found.
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(WebRequest request)
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
AbsoluteUri:https://sdnexpnc.corp.contoso.com/networking//servers
at Get-SdnResource, D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnResource.ps1: line 61
at Get-SdnServer, D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnServer.ps1: line 31
at Get-SdnInfrastructureInfo, D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnInfrastructureInfo.ps1: line 76
at <ScriptBlock>, <No file>: line 1
At D:\Temp\SdnDiagnostics\modules\NetworkController\public\Get-SdnResource.ps1:77 char:113
+ ... .Address.AbsoluteUri, $_.ScriptStackTrace | Trace-Output -Level:Error
+                                                 ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Trace-Output

Expected behavior
The 2nd call with correct credential should success

[Validation] X509Certificate credential object referenced by server resource have correct certificate found on NCs

Is your feature request related to a problem? Please describe.
Validation scenario. The Server resource have credential object referenced by its connection. Example:

{
  "resourceRef": "/servers/xxxxxxx",
  "resourceId": "xxxxx",
  "resourceMetadata": {
    "client": "xxx",
    "resourceName": "xxx"
  },
  "etag": "W/\"xxxx\"",
  "instanceId": "xxx",
  "properties": {
    "provisioningState": "Succeeded",
    "connections": [
      {
        "managementAddresses": [
          "server1.contoso.com"
        ],
        "credential": {
          "resourceRef": "/credentials/xxxxx"
        },
        "credentialType": "usernamePassword"
      },
      {
        "managementAddresses": [
          "server1.contoso.com"
        ],
        "credential": {
          "resourceRef": "/credentials/xxxx"
        },
        "credentialType": "X509Certificate"
      }
    ],


{
  "resourceRef": "/credentials/xxxx",
  "resourceId": "xxx",
  "etag": "W/\"xxx\"",
  "instanceId": "xxx",
  "properties": {
    "provisioningState": "Succeeded",
    "type": "X509Certificate",
    "value": "thumbprint",
    "networks": []
  }
}

The X509Certificate credential resource have value pointed to certificate on NC by thumbprint. The cert need to be exist, otherwise will cause connection problem from NC To Hosts.

Describe the solution you'd like

• Read all credential resource of type certificate referenced by server.
• Check each of the credential resource to confirm if certificate match the thumbprint found or not.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Install-SdnDiagnostic failed with exception due to close busy session

Describe the bug
Install-SdnDiagnostic failed with exception due to close busy session

To Reproduce
Steps to reproduce the behavior:

PS D:\Temp\SdnDiagnostics> Install-SdnDiagnostic -ComputerName $infra.Host -Verbose
VERBOSE: TraceFile: C:\Windows\Tracing\SdnDataCollection\SdnDiagnostics_TraceOutput_20210727.csv
VERBOSE: SdnDiagnostics module found at D:\Temp\SdnDiagnostics\
VERBOSE: PSRemotingSession use default credential
VERBOSE: Created powershell session WinRM6 to SDNEXP1.Corp.contoso.com
Copying D:\Temp\SdnDiagnostics\ to SDNEXP1.Corp.contoso.com via WinRM
System.Management.Automation.PSInvalidOperationException: Cannot connect the PSSession because the session is not in the Disconnected state, or is not available for connection.
   at System.Management.Automation.RemoteRunspace.Connect()
   at Microsoft.PowerShell.Commands.RemovePSSessionCommand.ProcessRecord()
   at System.Management.Automation.CommandProcessor.ProcessRecord()
at Install-SdnDiagnostic, D:\Temp\SdnDiagnostics\src\modules\public\utilities.ps1: line 33
at <ScriptBlock>, <No file>: line 1

If we dump out the list of sessions we see there is a strange session created with availability of "Busy" not able to remove. It will disappear by itself soon

PS D:\Temp\SdnDiagnostics> Install-SdnDiagnostic -ComputerName $infra.Host -Verbose
VERBOSE: TraceFile: C:\Windows\Tracing\SdnDataCollection\SdnDiagnostics_TraceOutput_20210727.csv
VERBOSE: SdnDiagnostics module found at D:\Temp\SdnDiagnostics\
VERBOSE: Located existing powershell session WinRM6
Copying D:\Temp\SdnDiagnostics\ to SDNEXP1.Corp.contoso.com via WinRM

 Id Name            ComputerName    ComputerType    State         ConfigurationName     Availability
 -- ----            ------------    ------------    -----         -----------------     ------------
  8 WinRM1          SDNEXP1.Corp... RemoteMachine   Disconnected  Microsoft.PowerShell          Busy
  6 WinRM6          SDNEXP1.Corp... RemoteMachine   Opened        Microsoft.PowerShell     Available

Expected behavior
CmdLet finished successfully

Collect Audit Logs if Enabled

Describe the solution you'd like
If NC Audit Logging is enabled, determine source location and pull them as part of data collection.

Remote jobs are failing with no details as to why

Describe the bug
This is related to #110 where file copy job fails. This is because the scripts to dump the configuration/logs on the remote nodes failed.

"<COMPUTERNAME>","2/23/2022 3:43:38 PM","Wait-PSJob","Verbose","JobName: c06dba31-f64d-405c-8397-80c6cbaad11a PollingInterval: 1 seconds ExecutionTimeout: 600 seconds"
"<COMPUTERNAME>","2/23/2022 3:44:04 PM","Wait-PSJob","Warning","JobName: c06dba31-f64d-405c-8397-80c6cbaad11a Operation Failed. Total Elapsed Time: 25.6296083"
"<COMPUTERNAME>","2/23/2022 3:44:04 PM","Wait-PSJob","Warning","JobName c06dba31-f64d-405c-8397-80c6cbaad11a: Job for COMPUTERNAME.FQDN failed with State: Failed | Status: "
"<COMPUTERNAME>","2/23/2022 3:44:04 PM","Wait-PSJob","Warning","JobName c06dba31-f64d-405c-8397-80c6cbaad11a: Job for COMPUTERNAME.FQDN failed with State: Failed | Status: "
"<COMPUTERNAME>","2/23/2022 3:44:04 PM","Wait-PSJob","Error","JobName c06dba31-f64d-405c-8397-80c6cbaad11a: State: Failed StatusMessage: "

Expected behavior
If we see a failure in the remote job execution, we need to capture the appropriate details so we can understand why the job failed.

Gracefully handle if no Gateways or MUXes provisioned yet

Describe the bug
In newly deployed environments, there may not yet be gateways or muxes deployed yet. This causes issues when running Get-SdnGateway or Get-SdnLoadBalancerMUX where it generates warnings and errors.

"<REMOVED>","2/23/2022 3:07:36 PM","Get-SdnLoadBalancerMux","Warning"," is reporting provisioningState: "
"<REMOVED>","2/23/2022 3:07:36 PM","Get-SdnLoadBalancerMux","Error","System.Management.Automation.ParameterBindingValidationException: Cannot bind argument to parameter 'ResourceRef' because it is an empty string.
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at Get-SdnLoadBalancerMux, C:\Program Files\WindowsPowerShell\Modules\SDndiagnostics\2.2202.215.82917\modules\NetworkController\public\Get-SdnLoadBalancerMux.ps1: line 41
at Get-SdnInfrastructureInfo, C:\Program Files\WindowsPowerShell\Modules\SDndiagnostics\2.2202.215.82917\modules\NetworkController\public\Get-SdnInfrastructureInfo.ps1: line 89
at <ScriptBlock>, <No file>: line 1"

Expected behavior
We should gracefully indicate no resources provisioned if call to API endpoint is successful but no resources returned.

Enable L3 NAT Debugging

Public IP for NIC is not properly identified if the LB FE config is a static IP address

Describe the bug
Per https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-ncnbi/c2aac45c-6c08-4c94-8308-7a65d2ca5b6f, the public IP address can be referenced using the publicIpAddress field (references the publicIPAddresses resource) or by defining the privateIPAddress.

To Reproduce

1. Configure the privateIPAddress of loadBalancer frontend configuration to use an IP address from the public VIP logical network.

Expected behavior
Depending on the environment, we need to either return privateIPAddress or publicIPAddress.

RestName property for Get-NetworkController may be $null

Describe the bug
Network Controller can either use IP Address (RestIPAddress) or RestName, however not both. This causes Get-SdnInfrastructureInfo to fail to load in environments as the RestURI cannot be configured into cache for subsequent calls.

Using an IP address returns an authentication error

System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The 
remote certificate is invalid according to the validation procedure.

Environment Details (please complete the following information):

• Windows Server OS: Windows Server 2019
• SdnDiagnostics Version: 2.2201.198.12840

Configuration and ProvisioningState is only logging as Information event

Describe the bug

[SDN-DC01] Validating configuration and provisioning state of Gateways
[SDN-DC01] /Gateways/SDN-GW01.SDN.LAB is reporting configurationState status Failure:
- Unable to configure SLB rules

[Fix] Need to include Status of Failure rather than Error.

Enable support for RemoteLogLocation

Is your feature request related to a problem? Please describe.
In environments where log files are saved to a RemoteLogLocation rather than default log directory of C:\Windows\Tracing\SdnDiagnostics, when running Start-SdnDataCollection -IncludeLogs, the logs are not collected for the NC services.

Describe the solution you'd like
Logs should be picked up from the RemoteLogLocation.

[Debug] Ensure VFP Port Profiles configured for adapters using VFP Enabled Switches

By default if the vSwitch is using VFP, then an appropriate VFP Profile must be configured for routing to work. Should integrate a Debug cmdlet either as part of the fabric tests, or add new tests for adapters that if vSwitch has VFP enabled that we should raise up guidance to resolve.

See https://learn.microsoft.com/en-us/azure-stack/hci/manage/manage-default-network-access-policies-virtual-machines#vms-created-outside-of-windows-admin-center for additional details.

Get-SdnApiResource throws 404 exception for certain Api endpoints

Describe the bug
When running Get-SdnApiResource, errors are thrown indicating 404 not found.

Screenshots

[Debug] Add checks to ensure Gateway VLAN/ProfileId configured correctly for interfaces

Add new check for Gateways role to ensure that we have the expected VLAN/Isolation settings configured for the interfaces.

Add copy of the TraceOutput to the dated folder for Start-SdnDataCollection

Feature Request:
A lot of time, the dated folder is provided for analysis.

• C:\Windows\Tracing\SdnDataCollection
  • YYYYMMDD-######
  • SdnDiagnostics_TraceOutput_*.csv

Need to automatically take a copy of the trace output, and save to the dated folder so we have it for analysis if required.

Move-SdnServiceFabricReplica fails if only using a single NC node

Describe the bug
Attempting to run Move-SdnServiceFabricReplica fails when only running a single NC.

Move-SdnServiceFabricReplica -NetworkController $nc -ServiceName 'fabric:/NetworkController/GatewayManager'
No nodes satisfied all the constraints
    + CategoryInfo          : InvalidOperation: (Microsoft.Servi...usterConnection:ClusterConnection) [Move-ServiceFab
   ricPrimaryReplica], FabricException
    + FullyQualifiedErrorId : MoveReplicaErrorId,Microsoft.ServiceFabric.Powershell.MovePrimaryReplica
    + PSComputerName        : <COMPUTERNAME>

Trace-Output : [COMPUTERNAME] System.NullReferenceException: Unable to return results from service fabric
at Invoke-SdnServiceFabricCommand, C:\Program Files\WindowsPowerShell\Modules\SdnDiagnostics\2.2208.397.134009\modules\
NetworkController\public\Invoke-SdnServiceFabricCommand.ps1: line 83
at Move-SdnServiceFabricReplica, C:\Program Files\WindowsPowerShell\Modules\SdnDiagnostics\2.2208.397.134009\modules\Ne
tworkController\public\Move-SdnServiceFabricReplica.ps1: line 78
at <ScriptBlock>, <No file>: line 1
At C:\Program Files\WindowsPowerShell\Modules\SdnDiagnostics\2.2208.397.134009\modules\NetworkController\public\Invoke-
SdnServiceFabricCommand.ps1:94 char:59
+ ... {1}" -f $_.Exception, $_.ScriptStackTrace | Trace-Output -Level:Error
+                                                 ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Trace-Output

Expected behavior
This is expected from SF cluster perspective, since no node to move the service to. We should gracefully handle this exception.

Environment Details (please complete the following information):

• Windows Server OS:
• SdnDiagnostics Version:

Create debug framework for VIP troubleshooting

Install-SdnDiagnostics not installed to PS Profile path

Describe the bug
Install-SdnDiagnostics not installed to PS Profile path if the local module installed from arbitrary path other than

C:\Program Files\WindowsPowerShell\Modules

This happens for the self-hosted runner scenario where Module installed from build path.

To Reproduce
Issues found from Self Hosted runner Server 2019

Environment Details (please complete the following information):

• Windows Server OS: 2019
• SdnDiagnostics Version: 1.2109.70.132852

Get-SdnServiceFabricClusterManifest fails when calling back to itself

Describe the bug
Get-SdnServiceFabricClusterManifest fails when calling back to itself

To Reproduce
Steps to reproduce the behavior:

1. Connect to Network Controller and run Get-SdnServiceFabricClusterManifest which will fail

Expected behavior
Cluster Manifest should be returned back successfully.

Screenshots

Service Fabric parameters are case sensitive

Describe the bug
-ServiceTypeName and -ServiceName for Service Fabric cmdlets are case sensitive, resulting in an error returned that it was not able to return results.

To Reproduce
Steps to reproduce the behavior:

1. Run Get-SdnServiceFabricReplica -ServiceTypeName 'firewallservice'
2. Look at results, then trying with name of FirewallService

Expected behavior
This is just a problem with SF cmdlets themselves, so ideally SdnDiagnostics should handle this to either provide more precise feedback, or just make sure it's using the proper casing

Screenshots

Environment Details (please complete the following information):

• Windows Server OS: Windows Server 2022
• SdnDiagnostics Version: 5.2212.508.225728

Additional context
Add any other context about the problem here.

New-PSRemotingSession is not properly detecting IP address

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Get-SdnOvsdbAddressMapping -ComputerName 'xxx.xx.4.128'

"<removed>","2/11/2022 4:52:37 PM","New-PSRemotingSession","Warning","Unable to create powershell session toxxx.xx.4.128
	System.Management.Automation.Remoting.PSRemotingTransportException: Connecting to remote server xxx.xx.4.128 failed with the following error message : The WinRM client cannot process the request. Default authentication may be used with an IP address under the following conditions: the transport is HTTPS or the destination is in the TrustedHosts list, and explicit credentials are provided. Use winrm.cmd to configure TrustedHosts. Note that computers in the TrustedHosts list might not be authenticated. For more information on how to set TrustedHosts run the following command: winrm help config. For more information, see the about_Remote_Troubleshooting Help topic."

Expected behavior
It should detect IP address, and then add the IP address to the TrustedHosts. src/modules/Utilities/private/New-PSRemotingSession.ps1 is not properly detecting this.

Environment Details (please complete the following information):

• Windows Server OS: Windows Server 2019
• SdnDiagnostics Version: 2.2201.198.12840

Exception when call Get-NetworkInterfaceEncapOverheadSetting

PS D:\Temp\SdnDiagnostics> icm sdnexp1 -ScriptBlock{Get-NetworkInterfaceEncapOverheadSetting }
Network interface Microsoft Hyper-V Network Adapter does not support EncapOverhead.
System.Management.Automation.ArgumentTransformationMetadataException: Cannot convert the "System.String[]" value of type "System.String[]" to type "System.Int32". ---> System.Management.Automation.PSInvalidCastException: Cannot convert the "System.String[]" value of type "System.String[]" to type "System.Int32".
   at System.Management.Automation.LanguagePrimitives.ThrowInvalidCastException(Object valueToConvert, Type resultType)
   at System.Management.Automation.LanguagePrimitives.ConvertNoConversion(Object valueToConvert, Type resultType, Boolean recurse, PSObject originalValueToConvert, IFormatProvider formatProvider, TypeTable backupTable)
   at System.Management.Automation.LanguagePrimitives.ConversionData`1.Invoke(Object valueToConvert, Type resultType, Boolean recurse, PSObject originalValueToConvert, IFormatProvider formatProvider, TypeTable backupTable)
   at System.Management.Automation.LanguagePrimitives.ConvertTo(Object valueToConvert, Type resultType, Boolean recursion, IFormatProvider formatProvider, TypeTable backupTypeTable)
   at System.Management.Automation.ArgumentTypeConverterAttribute.Transform(EngineIntrinsics engineIntrinsics, Object inputData, Boolean bindingParameters, Boolean bindingScriptCmdlet)
   --- End of inner exception stack trace ---
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at Get-NetworkInterfaceEncapOverheadSetting, C:\Program Files\WindowsPowerShell\Modules\SdnDiagnostics\modules\public\role\server.ps1: line 940

Jumbo Packet property as below

PS D:\Temp\SdnDiagnostics\tests> icm sdnexp1 -ScriptBlock{ Get-NetAdapterAdvancedProperty -RegistryKeyword "*JumboPacket"}
>>

Name                      DisplayName                    DisplayValue                   RegistryKeyword RegistryValue   PSComputerName
----                      -----------                    ------------                   --------------- -------------   --------------
Ethernet 2                Jumbo Packet                   Disabled                       *JumboPacket    {1514}          sdnexp1
Mgmt                      Jumbo Packet                   Disabled                       *JumboPacket    {1514}          sdnexp1
Ethernet                  Jumbo Packet                   Disabled                       *JumboPacket    {1514}          sdnexp1

Install-SdnDiagnostics failed to install module on remote machine

Describe the bug
The module installed to path like C:\Program Files\WindowsPowerShell\Modules\1.2109.76.448\SdnDiagnostics.psm1 and failed to be auto loaded when run any command from the SdnDiagnostics module

To Reproduce
Steps to reproduce the behavior:

1. Install module from PS gallery on a jump box.
2. Run Install-SdnDiagnostics to install the module to remote infra nodes

Expected behavior
Expect to be able to run SdnDiagnostics cmdlet after module installed but not actually due to the incorrect module path copied.

Screenshots
Module on local machine:

Module on remote machine by Install-SdnDiagnostics

Environment Details (please complete the following information):

• Windows Server OS: Server 2019
• SdnDiagnostics Version: 1.2109.76.448

KI - Detect if the Host Root CA store include any non-Root Certificate

Is your feature request related to a problem? Please describe.
New KI - Detect if the Host Root CA store include any non-Root Certificate.

Describe the solution you'd like
Enum the certificates under Host's Root CA store. Report any certificate that Subject != Issuer.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Non Root Cert should not be exist in Root CA store. This will prevent NC talk to Host with error HTTP 401.

Get-SdnNetworkInterfaceOutboundPublicIPAddress is not returning the Public VIP related to load balancer

Describe the bug
When attempting to identify the public VIP associated with a networkInterface, Get-SdnNetworkInterfaceOutboundPublicIPAddress is able to identify the publicIpAddress when directly assigned to networkInterface, however returns $null when query a networkInterface that is part of a backend pool.

To Reproduce
Steps to reproduce the behavior:

1. Deploy load balancer w/ public IP using Windows Admin Center
2. Run Get-SdnNetworkInterfaceOutboundPublicIPAddress against the networkInterface associated with the backendpool

Expected behavior
Would expect the public VIP for the load balancer to be returned.

Environment Details (please complete the following information):

• Windows Server OS: Windows Server 2022
• SdnDiagnostics Version: N/A

Improve the output of VIP and DIP status

Get-SdnSlbStateInformation queries NC to get a consolidated Fabric and Tenant view of all VIP related resources. While it's useful, it's limited as it dumps out everything in a partial JSON format, making it so that you cannot use .NET to parse the actual information returned for a specific VIP.

Need to create new custom functions that will allow the results to be returned as a .NET object rather than the console output currently being returned.

Copy-FileToRemoteComputerSMB didn't throw error if test-path failed cause no failover to WinRM

Describe the bug
When copy file from remote machine via SMB, the Test-Path call will test if the path can be accessed or not.

        if (-NOT (Test-Path -Path $remotePath)) {
            "Unable to access {0}" -f $remotePath | Trace-Output -Level:Error
        }

Test-Path won't throw exception that can be handled by Copy-FileFromRemoteComputer to fallback to WinRM file copy. Logs failed to be copied finally.

Expected behavior
Test-Path to test each subpath, if all path can not be accessed. Throw exception. If any of the path can be accessed don't do that as could be folder level issue instead of machine level issue prevent SMB Copy

Another method might be use any cmdlet that can test SMB access and able to throw exception. For example: Get-Item

Screenshots
If applicable, add screenshots to help explain your problem.

Environment Details (please complete the following information):

• Windows Server OS: Server 2019
• SdnDiagnostics Version: 2.2202.215.82917

Additional context
Add any other context about the problem here.

Warnings generated for vfp port state during data collection

Describe the bug
When collecting data from Hyper-V host, we get warnings regarding

WARNING: [SDN-HOST02] Unable to process PORT STATE
WARNING: [SDN-HOST02] Unable to process   No Encapsulation Offloads Enabled
WARNING: [SDN-HOST02] Unable to process Command get-port-state succeeded!

Expected behavior
These are not valid VFP port properties, so we should skip these lines for processing

trace file locked and not able to write intermittently

Describe the bug
Trace file is intermittently locked and unable to be written to

"<NODE>","2022-11-17 11-22-34","Get-SdnDiagnosticLog","Error","System.IO.IOException: The process cannot access the file 'C:\Windows\Tracing\SdnDataCollection\SdnDiagnostics_TraceOutput_20221117.csv' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at System.Management.Automation.PathUtils.MasterStreamOpen(PSCmdlet cmdlet, String filePath, Encoding resolvedEncoding, Boolean defaultEncoding, Boolean Append, Boolean Force, Boolean NoClobber, FileStream& fileStream, StreamWriter& streamWriter, FileInfo& readOnlyFileInfo, Boolean isLiteralPath)
at Trace-Output, C:\Program Files\WindowsPowerShell\Modules\SdnDiagnostics\2.2210.469.134929\modules\Utilities\private\Trace-Output.ps1: line 66
at Get-SdnDiagnosticLog, C:\Program Files\WindowsPowerShell\Modules\SdnDiagnostics\2.2210.469.134929\modules\Common\public\Get-SdnDiagnosticLog.ps1: line 31"

To Reproduce
N/A

Expected behavior
This is related to mutex locking and need to enable mutexes to properly handle this.

Screenshots
N/A

Environment Details (please complete the following information):
N/A

Additional context
N/A

managementAddresses may contain both FQDN and IP Address

Describe the bug
In some configurations, the managementAddress property may contain both an IP address and FQDN of the resource. This causes issues later when trying to leverage the cache to perform operations.

Expected behavior
We should only return a single object or improve logic handling if multiple management addresses are found for a specific resource.

Screenshots

Environment Details (please complete the following information):

• Windows Server OS: Windows Server 2019
• SdnDiagnostics Version: 2.2201.198.12840

SLBM service name parameter is missing in test-sdnexpressbgp

Describe the bug
A clear and concise description of what the bug is.

TestSdnExpressBGP function tries to get service running information with Get-Service command.
The service name parameter is missing.

To Reproduce
Steps to reproduce the behavior:

1. Run TestSdnExpressBGP when SBLM service is running in mux
2. See error

Expected behavior
A clear and concise description of what you expected to happen.
SBLM service status should be checked

Screenshots
If applicable, add screenshots to help explain your problem.

Environment Details (please complete the following information):

• Windows Server OS:
• SdnDiagnostics Version: 3.2310.816.174130

Additional context
Add any other context about the problem here.

$mux = Get-Service -Name -erroraction silentlycontinue
if ($null -ne $mux) {
$muxstartup = $mux.starttype
$muxstatus = $mux.status

            if (($muxstatus -ne "Stopped") -or ($Muxstartup -ne "Disabled")) {
                if ($force) {
                    $RestoreMuxState = $true
                    **Set-Service -Name** -startup Disabled
                    **stop-Service -Name**
                }
                else {
                    throw "SLB Mux service is active.  Use -force to temporarily disable it during test."
                }
            }
        }
        else {
            $muxstate = $null
        }

• After putting 'slbmux', it worked.

[Debug] Add checks to ensure MUX VLAN/ProfileID configured correctly for interfaces

Add new test for LoadBalancerMuxes to validate that the VMNetworkAdapter VLAN/Isolation settings are configured correctly.

Error generated with Get-PACAMapping results in function terminating

Describe the bug

In the logs, and then cross-comparing w/ Get-SdnConfig, can see that Get-PACAMapping generated the terminating error, as this command is executed after Get-VM. Due to this, no further commands are executed as part of the data collection.

Expected behavior
We should gracefully handle the exception and not terminate the entire function.

Verify that the RESTNAME IP matches the IP address of primary replica

Cannot process argument transformation on parameter 'RotateNodeCerts

Describe the bug
certificate rotate fails with the following exception:

"<REMOVE PII>","2024-02-20 11-46-58","Trace-Exception","Error","Cannot process argument transformation on parameter 'RotateNodeCerts'. Cannot convert value """" to type ""System.Boolean"". Boolean parameters accept only Boolean values and numbers, such as $True, $False, 1 or 0."

Enable Load Balancer Debugging

Enable insights for NCHostAgent and SLBHostAgent TCP connections

Is your feature request related to a problem? Please describe.
Need to quickly confirm if we see the appropriate TCP connections from NCHostAgent and SLBHostAgent to NC.

Describe the solution you'd like
Add automated insights for checking fabric health to make sure we see appropriate TCP connections.

NCHostAgent -> [NC SDNAPI]:6640
SLBHostAgent -> [NC SDNSLBM]:8571

NC SDNVSM -> [NCHostAgent]:6640
NC SDNFW -> [NCHostAgent]:6640

multiple versions of SdnDiagnostics module exist cause installation failures when seeding to remote nodes

Describe the bug
When running Install-SdnDiagnostic, it fails with an exception indicating:

PS C:\> Install-SdnDiagnostic -ComputerName $environmentDetails.FabricNodes
Current SdnDiagnostics version is 1.2109.69.131442
System.Management.Automation.RuntimeException: Cannot convert the "System.Object[]" value of type "System.Object[]" to type "System.Version". ---> System.Management.Automation.PSInvalidCastException: Cannot convert the "Sys
tem.Object[]" value of type "System.Object[]" to type "System.Version".
   at System.Management.Automation.LanguagePrimitives.ThrowInvalidCastException(Object valueToConvert, Type resultType)
   at System.Management.Automation.LanguagePrimitives.ConvertNoConversion(Object valueToConvert, Type resultType, Boolean recurse, PSObject originalValueToConvert, IFormatProvider formatProvider, TypeTable backupTable)
   at CallSite.Target(Closure , CallSite , Object )
   at System.Management.Automation.Interpreter.DynamicInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   --- End of inner exception stack trace ---
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at Install-SdnDiagnostic, C:\Program Files\WindowsPowerShell\Modules\SdnDiagnostics\1.2109.69.131442\modules\Utilities\public\Install-SdnDiagnostic.ps1: line 60
at <ScriptBlock>, <No file>: line 1

Environment Details (please complete the following information):

• Windows Server OS:
• SdnDiagnostics Version: 1.2109.69.131442

Workaround:

Invoke-Command -ComputerName $environmentDetails.FabricNodes -ScriptBlock {Remove-Item -Path 'C:\Program Files\WindowsPowerShell\Modules\SdnDiagnostics' -Recurse -Force}

Test-SdnKINetworkInterfacePlacement incorrectly triggers warning

Describe the bug
In scenarios where the name within servers API does not match the hyper-V host name, function will throw non-accurate warnings:

[NODE] Validate placement of network interfaces between Network Controller and Hypervisor
WARNING: [NODE] 020100000108: Network Controller believes [NetBIOS] exists on [GUID] while hypervisor is reporting it exists on [NetBIOS]

Expected behavior
Need to gracefully be able to handle if the resourceName does not align to the netBIOS name of the server.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment Details (please complete the following information):

• Azure Stack HCI: 20348.859.amd64fre.fe_release_svc_prod2.220707-1832
• SdnDiagnostics Version: N/A

Additional context
N/A

Publish PS cmdlet documentation to learn.microsoft.com

Enable automatic publication of function documentation to learn.microsoft.com rather than the local SdnDiagnostics/Wiki for better consumption.

Install-SdnDiagnostics failure due to module not found

Current SdnDiagnostics version is 1.2108.24.7433
SdnDiagnostics module found on sdnexp1 but remote version 0.2108.20.84322 mismatch with local version 1.2108.24.7433. Will perform install.
System.Management.Automation.ParameterBindingValidationException: Cannot bind argument to parameter 'Path' because it is null.
at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at Install-SdnDiagnostic, D:\Temp\SdnDiagnostics\modules\public\utilities.ps1: line 66
at <ScriptBlock>, <No file>: line 1

[LogCollection] Function for Get-SdnLoadBalancerMuxes and Get-SdnGatways return management address based on Virtual Server

Get-SdnLoadBalancerMuxes and Get-SdnGateways currently use -ResourceIdOnly to return resource Id. This might be the correct management address for MAS or SDNExpress but may not always True.

Need to implement -ManagementAddressOnly to retrieve management address from virtual server associated by MUX or Gateway, this align with Get-SdnServers function.

Verify SdnDiagnostics module is installed on remote nodes

Is your feature request related to a problem? Please describe.
If operator has not seeded the module to all the SDN fabric nodes, larger scripts such as data collection will throw a lot of errors and ultimately fail.

Describe the solution you'd like
For some of the larger orchestrations, we should do a pre-validation to make sure the node(s) specified have the SdnDiagnostics module installed.

Describe alternatives you've considered
N/A

Additional context
N/A

Unable to import SdnDiagnostics directly after using Install-SdnDiagnostics

Describe the bug
After leveraging Install-SdnDiagnostics, when navigating to a node that module was installed to, Import-Module SdnDiagnostics fails as indicates not module exists, however calling the functions directly work.

In looking at the C:\ProgramFiles\WindowsPowerShell\Modules directory, can see that it's creating it as \Modules\SdnDiagnostics\SdnDiagnostics\{contents} which is why the failure is happening as we expect the psd1 and psm1 to be in the root of the first directory.

This appears to be a regression introduced in #180

Expected behavior
We should maintain the correct directory structure