Network in WIndows Server 2008R2 guest drops during heavy host IO about kvm-guest-drivers-windows

[YanVugenfirer]

Thank you for the report.

Did you experience such issues with the previous version of the driver as well or this is a regression?

[opoplawski]

Unfortunately I've been experiencing this for a while and with a number of recent revisions. At this point I'm not sure what triggered it.

[YanVugenfirer]

I think we should try to do two things:

1. Try to get logs from the failing drivers. Please download DebugView, run
   it with administrative privileges and enable kernel logging:
   https://technet.microsoft.com/en-us/sysinternals/bb896647.aspx
2. Try to check if this is a new regression. We didn't release official
   drivers for quite a long time so it's worth checking the following version:
   https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso

Best regards,
Yan.

On Sun, Nov 1, 2015 at 6:39 PM, Orion Poplawski [email protected]
wrote:

Unfortunately I've been experiencing this for a while and with a number of
recent revisions. At this point I'm not sure what triggered it.

—
Reply to this email directly or view it on GitHub
https://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues/43#issuecomment-152842535
.

---

Daynix Computing LTD
Yan Vugenfirer, CEO
Email: [email protected]
Phone (Israel): [phone]
Phone (USA): [phone]
Phone (UK): [phone]
Web: www.daynix.com

[YanVugenfirer]

Hi,

Unfortunately our QE couldn't reproduce the issue on RHEL 6.7 with network longevity tests with the versions you described.

Could you please elaborate more on the setup on the host (bridge, macvtap, something else?) and provide more info on what is running on the guest side?

Thank you.

[opoplawski]

The system lost network connectivity around 6:22 AM this morning. Log from DbgView is at http://sw.cora.nwra.com/tmp/PLUTO.LOG.gz. It's over a couple days so if you scroll back from the end you'll start to see problems around 6:22 AM.

I'll try the 0.1.102 version from your link now.

This is the libvirt/qemu xml for the guest:

<domain type='kvm'>
  <name>pluto</name>
  <uuid>00abd12e-e234-73bf-6532-3fd095de9ef0</uuid>
  <memory unit='KiB'>3121152</memory>
  <currentMemory unit='KiB'>3121152</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.4.0'>hvm</type>
    <boot dev='hd'/>
    <boot dev='cdrom'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/data/sw/windows/install/virtio-win-0.1.110.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/vg_data/pluto-disk0'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:2c:89:46'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='vga' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

After looking a bit more closely at the timing of everything, it may actually be the running of Windows Server Backup on the guest which started at 6am that triggers the issue. It is backing up to a network drive on the shared from the host. The high I/O processes on the host ended around 5am. The guest is a domain controller, but otherwise fairly idle.

[opoplawski]

Still see the problem with 0.1.102.

[lijinlijin]

Hi,

virtio-win QE failed to reproduce this issue with following steps:
1.boot two win2008R2 guests on rhel6.7 host with virtio-net-pci;
2.Add Windows Server Backup feature in both win2008R2 guests;
3.Schedule windows server backup in both guest at 6:00am via following cli(backup windows to a share drive on the host):

wbadmin enable backup -addtarget:\host_ip\backup\ -schedule:06:00 -user:xxx -password:xxx -quiet -vssfull

Actual result:
Check network status next day,network still alive and windows backup scheduled correctly.

Package info:
qemu-kvm-0.12.1.2-2.479.el6_7.2.x86_64
kernel-2.6.32-573.7.1.el6.x86_64
virtio-win-prewhql-0.1-110

Here are my questions:
1.I note that you said your windows backup started at 6:00am and host high i/o processes ended around 5:00am.
Could you tell me what's the high i/o processes on host is,seems it's not the windows backup according to scheduled time.

2.Is this issue 100% reproducible,I mean does the guest lost network everyday after the windows server backup?I let the guests run 48 hours,still not hit your issue.

Could you check my steps if any step is not suitable?Thanks.

[opoplawski]

I'm afraid that the problem is very intermittent and appears sensitive to the processes on the machine. For a while it was happening almost every night, prompting me to file this bug. But recently it's gone back to running fine most of the time. Some other variables may be:

• running amanda backups from the host machine overnight 1-1.4TB backed up
• use or not of cgroups with libvirt - it's been on or off at different times due to other issues.

So it may be very hard to reproduce. I'm happy to run any other diagnostics, debug code, etc. that might help track it down though.

[YanVugenfirer]

Thank you very much for trying to reproduce the steps that lead to the
issue. Let me think what else can be done. By the way, are you enabling
virtio multi-queue in your setup?

Best regards,
Yan.

On Fri, Nov 20, 2015 at 8:24 PM, Orion Poplawski [email protected]
wrote:

I'm afraid that the problem is very intermittent and appears sensitive to
the processes on the machine. For a while it was happening almost every
night, prompting me to file this bug. But recently it's gone back to
running fine most of the time. Some other variables may be:

• running amanda backups from the host machine overnight 1-1.4TB
  backed up
• use or not of cgroups with libvirt - it's been on or off at
  different times due to other issues.

So it may be very hard to reproduce. I'm happy to run any other
diagnostics, debug code, etc. that might help track it down though.

—
Reply to this email directly or view it on GitHub
https://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues/43#issuecomment-158482162
.

---

Daynix Computing LTD
Yan Vugenfirer, CEO
Email: [email protected]
Phone (Israel): [phone]
Phone (USA): [phone]
Phone (UK): [phone]
Web: www.daynix.com

[opoplawski]

And now we're back to two failures in two nights. No multi-queue.

[opoplawski]

We recently removed GFI Vipre from our Windows VMs and I'm back to not seeing this any more. Perhaps that was a trigger.

[YanVugenfirer]

Interesting. Will try to reproduce with GFI Vipre as well.

On Mon, Dec 21, 2015 at 7:05 PM, Orion Poplawski [email protected]
wrote:

We recently removed GFI Vipre from our Windows VMs and I'm back to not
seeing this any more. Perhaps that was a trigger.

—
Reply to this email directly or view it on GitHub
https://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues/43#issuecomment-166360669
.

---

Daynix Computing LTD
Yan Vugenfirer, CEO
Email: [email protected]
Phone (Israel): [phone]
Phone (USA): [phone]
Phone (UK): [phone]
Web: www.daynix.com

[rossica]

I believe I have a similar issue to this.
I have a windows 10 Pro 64-bit guest, on a Linux 4.3.3-3-ARCH SMP PREEMPT x86_64 host.
I am using the virtio-win-0.1.1 drivers for win8.1 x64 (NetKVM.sys displays file version 62.72.104.10200).
The guest is running in VirtualBox 5.0.14 with a virtio paravirtualized NIC, using the NAT connectivity type.

I have a srcds server instance running in the guest. After about 3 hours of clients connecting and pinging the server, it stops responding. Restarting srcds does not remedy this. Other networking to the guest still works; for example, I am still able to use Remote Desktop Connection to the guest (not the virtualbox vRDP). I must reboot the guest for connectivity to the srcds server to be restored.
The host itself has minimal network traffic during this time; all the traffic is in the guest.

To isolate the issue, I reconfigured the VM to use the emulated Intel NIC instead of virtio, and the issue disappeared.

A couple questions for next steps:

1. Is this similar enough to issue #43 to be investigated with the same, or should I file a new issue?
2. What logs/traces would be useful for your investigation?

[YanVugenfirer]

Hi,

Please open additional issue.

You can raise the debug level of NetKVM driver in "advanced" tab of the
device property in device manager to 4. And then save the logs DbgView -
https://technet.microsoft.com/en-us/sysinternals/debugview.aspx. Run
DbgView as administrator and set it to log kernel trace.

Best regards,
Yan.

On Sun, Jan 31, 2016 at 12:09 AM, rossica [email protected] wrote:

I believe I have a similar issue to this.
I have a windows 10 Pro 64-bit guest, on a Linux 4.3.3-3-ARCH SMP PREEMPT
x86_64 host.
I am using the virtio-win-0.1.1 drivers for win8.1 x64 (NetKVM.sys
displays file version 62.72.104.10200).
The guest is running in VirtualBox 5.0.14 with a virtio paravirtualized
NIC, using the NAT connectivity type.

I have a srcds server instance running in the guest. After about 3 hours
of clients connecting and pinging the server, it stops responding.
Restarting srcds does not remedy this. Other networking to the guest still
works; for example, I am still able to use Remote Desktop Connection to the
guest (not the virtualbox vRDP). I must reboot the guest for connectivity
to the srcds server to be restored.
The host itself has minimal network traffic during this time; all the
traffic is in the guest.

To isolate the issue, I reconfigured the VM to use the emulated Intel NIC
instead of virtio, and the issue disappeared.

A couple questions for next steps:

1. Is this similar enough to issue #43
   https://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues/43
   to be investigated with the same, or should I file a new issue?
2. What logs/traces would be useful for your investigation?

—
Reply to this email directly or view it on GitHub
https://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues/43#issuecomment-177317736
.

---

Daynix Computing LTD
Yan Vugenfirer, CEO
Email: [email protected]
Phone (Israel): [phone]
Phone (USA): [phone]
Phone (UK): [phone]
Web: www.daynix.com

[rossica]

@YanVugenfirer My apologies for not responding sooner; I narrowed my issue down to VirtualBox's NAT connectivity. When I change the VM to "bridge" connectivity and I use the virtio paravirtualized NIC, I have had no problems whatsoever.

[YanVugenfirer]

Thanks for the update!

On Sat, Mar 19, 2016 at 11:41 PM, rossica [email protected] wrote:

@YanVugenfirer https://github.com/YanVugenfirer My apologies for not
responding sooner; I narrowed my issue down to VirtualBox's NAT
connectivity. When I change the VM to "bridge" connectivity and I use the
virtio paravirtualized NIC, I have had no problems whatsoever.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues/43#issuecomment-198788428

---

Daynix Computing LTD
Yan Vugenfirer, CEO
Email: [email protected]
Phone (Israel): [phone]
Phone (USA): [phone]
Phone (UK): [phone]
Web: www.daynix.com