Deadlock in netkvm with iscsi about kvm-guest-drivers-windows

[YanVugenfirer]

Thanks for the report.

By the way - what version did you use when you experienced this deadlock?

Thanks,
Yan.

[rob-scheepens]

Looks like we're hitting this as well, on an IIS/FTP server:

0: kd> knL f

Child-SP RetAddr Call Site

00 fffff802 3d2a3c08 fffff802 3bc371c2 nt!KeBugCheckEx
01 fffff802 3d2a3c10 fffff802 3be2de4d hal!HalBugCheckSystem+0x7e
02 fffff802 3d2a3c50 fffff802 3bc37fa1 nt!WheaReportHwError+0x22d
03 fffff802 3d2a3cb0 fffff802 3be51c20 hal!HalHandleNMI+0xfe
04 fffff802 3d2a3ce0 fffff802 3bdc77c2 nt!KiProcessNMI+0x150
05 fffff802 3d2a3d30 fffff802 3bdc7636 nt!KxNmiInterrupt+0x82
06 fffff802 3d2a3e70 fffff802 3bcfcd6b nt!KiNmiInterrupt+0x176
07 ffffd001 b1ec21a0 fffff800 a89d0443 nt!KxWaitForSpinLockAndAcquire+0x1f
08 (Inline Function) -------- -------- netkvm!CParaNdisTX::DoPendingTasks::__l8::<lambda_0d1768a7969216d060dd330cbb84655c>::operator()+0x6a
09 ffffd001 b1ec21d0 fffff800 a89d12a5 netkvm!CParaNdisTX::DoWithTXLock<<lambda_0d1768a7969216d060dd330cbb84655c> >+0x87
0a ffffd001 b1ec2210 fffff800 a89d1477 netkvm!CParaNdisTX::DoPendingTasks+0x75
0b ffffd001 b1ec22c0 fffff800 a89d1b7c netkvm!CParaNdisTX::NBLMappingDone+0x6b
0c ffffd001 b1ec22f0 fffff802 3bc06a37 netkvm!CNBL::RegisterMappedNB+0x6c
0d ffffd001 b1ec2320 fffff800 a7254e82 hal!HalBuildScatterGatherListV2+0x207
0e ffffd001 b1ec23c0 fffff800 a89d1d4a NDIS!NdisMAllocateNetBufferSGList+0x1a2

2: kd> knL f

Child-SP RetAddr Call Site

00 ffffd001 b4bb9cc0 fffff802 3bcfcd32 nt!KxWaitForSpinLockAndAcquire+0x1f
01 ffffd001 b4bb9cf0 fffff800 a89d1431 nt!KeAcquireSpinLockRaiseToDpc+0x32
02 (Inline Function) -------- -------- netkvm!CNdisSpinLock::Lock+0xa
03 (Inline Function) -------- -------- netkvm!CLockedContext::{ctor}+0xa
04 (Inline Function) -------- -------- netkvm!CParaNdisTX::DoWithTXLock+0xa
05 ffffd001 b4bb9d20 fffff800 a89d1b7c netkvm!CParaNdisTX::NBLMappingDone+0x25
06 ffffd001 b4bb9d50 fffff802 3bc06a37 netkvm!CNBL::RegisterMappedNB+0x6c
07 ffffd001 b4bb9d80 fffff800 a7254e82 hal!HalBuildScatterGatherListV2+0x207
08 ffffd001 b4bb9e20 fffff800 a89d1d4a NDIS!NdisMAllocateNetBufferSGList+0x1a2
09 ffffd001 b4bb9ee0 fffff800 a89d0722 netkvm!CNB::ScheduleBuildSGListForTx+0x2e
0a (Inline Function) -------- -------- netkvm!CNBL::StartMapping::__l6::<lambda_ea34f6eaf6f02c2b80267054dd87e843>::operator()+0x8
0b ffffd001 b4bb9f20 fffff800 a89d2247 netkvm!CNdisList<CNB,CRawAccess,CNonCountingObject>::ForEachDetached<<lambda_ea34f6eaf6f02c2b80267054dd87e843> >+0x3a
0c ffffd001 b4bb9f50 fffff800 a89d1e55 netkvm!CNBL::StartMapping+0x27
0d ffffd001 b4bb9f80 fffff800 a89d5b9a netkvm!CParaNdisTX::Send+0xfd
0e ffffd001 b4bba080 fffff800 a7252fb1 netkvm!ParaNdis6_SendNetBufferLists+0x46

This is with:

2: kd> !lmi netkvm
Loaded Module Info: [netkvm]
Module: netkvm
Base Address: fffff800a89c8000
Image Name: netkvm.sys
Machine Type: 34404 (X64)
Time Stamp: 562ea366 Mon Oct 26 18:04:22 2015
Size: 1e000
CheckSum: 288e1

[ladipro]

By the way - what version did you use when you experienced this deadlock?

As far as I can tell this issue exists in the current master branch.

CParaNdisTX::DoPendingTasks calls a potentially reentrant NDIS!NdisMSendNetBufferListsComplete under the m_CompletionLock, which then waits on CParaNdisTX::m_Lock. Other threads holding CParaNdisTX::m_Lock wait for m_CompletionLock ==> deadlock.

The pattern in CParaNdisTX::DoPendingTasks is definitely odd:

lock(m_Lock);
lock(m_CompletionLock);
unlock(m_Lock);
NdisMSendNetBufferListsComplete();
unlock(m_CompletionLock);

Any chance we could get rid of m_CompletionLock?

[rob-scheepens]

@ladipro: version was in my comment, it's:

2: kd> !lmi netkvm
Loaded Module Info: [netkvm]
Module: netkvm
Base Address: fffff800a89c8000
Image Name: netkvm.sys
Machine Type: 34404 (X64)
Time Stamp: 562ea366 Mon Oct 26 18:04:22 2015
Size: 1e000
CheckSum: 288e1

[ladipro]

@rob-scheepens: thanks, yes, I just wanted to point out that it's very unlikely that the bug has already been fixed. The problematic code is still there. My comment was targeted at @YanVugenfirer who is the netkvm expert.

[YanVugenfirer]

All locking mechanisms for TX are under review now. In a meanwhile I suggest that if no multi-queue support is needed to use stable build with single queue support (95).

[YanVugenfirer]

Please try the following build: https://dl.dropboxusercontent.com/u/97739133/virtio-win-prewhql-0.1-118.zip