Network connections toggle after reboot with NetKVM version 60.63.103.nnnn on Windows Server 2008 R2 about kvm-guest-drivers-windows

[MvdLande]

The issue appears to be Windows Server 2008 R2 related (or x86_64?). When I test the drivers on a Windows 7 32-bit guest and on windows server 2003 R2 32-bit guest the network connectivity doesn't toggle after a reboot. I haven't tested any other 64-bit Windows OS. I'll install a 64-bit windows 7 guest and test again.

[YanVugenfirer]

Hello,

Thank you for reporting this issue.

This scenario was tested numerous times on RHEL and Fedora and also guest OS is rebooted several times during WHQL certification.

Can you please describe your host network configuration?

Best regards,
Yan.

[MvdLande]

Network Setup:
Network Adapter: NC552SFP Dual 10Gb Fiber (eth0 and eth1)
(but the same happens when using 1Gb e1000e ethernet adapters for bonding)
This adapter is bonded in mode 1 to two separate switches, then multiple vlans are added and connected to a corresponding network bridge

eth0+eth1-->bond0
bond0.110 is added to br110
br110 is used for guest networking

config files

/etc/modprobe.d/bonding.conf
alias netdev-bond0 bonding # KVM networking + clustering

/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
NM_CONTROLLED=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
TYPE=Ethernet
IPV6INIT=no
/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
NM_CONTROLLED=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
TYPE=Ethernet
IPV6INIT=no

/etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
BOOTPROTO=static
IPADDR=172.16.111.1 # VMhost1a
PREFIX=24
GATEWAY=172.16.111.254
DNS1=172.16.110.1
DNS2=172.16.110.2
DOMAIN=test.local
ONBOOT=yes
BONDING_OPTS="mode=1 miimon=100"
NM_CONTROLLED=no
TYPE=Bonding
IPV6INIT=no

/etc/sysconfig/network-scripts/ifcfg-bond0.110

vlan110 used for VM hosts

DEVICE=bond0.110
ONBOOT=yes
BOOTPROTO=none
BRIDGE=br110
VLAN=yes

/etc/sysconfig/network-scripts/ifcfg-br110

bridge used for KVM virtual networking

DEVICE=br110
ONBOOT=yes
BOOTPROTO=none
NM_CONTROLLED=no
TYPE=Bridge
DELAY=0
IPV6INIT=no

The KVM guest is connected to br110

This is a snippet of the VM network config
...
interface type='bridge'
mac address='52:54:00:f1:84:05'
source bridge='br110'
model type='virtio'
driver name='vhost'
interface
...

Best Regards,

Maurits van de Lande

[MvdLande]

On windows 7 x86_64 the issue also shows up.
build-27

1. Installed network driver, connectivity ok
2. reboot guest, connectivity ok
3. reboot guest, connectivity fails
4. reboot guest, connectivity ok
5. reboot guest, connectivity ok
6. reboot guest, connectivity fails

build-29

1. Installed network driver, connectivity ok
2. reboot guest, connectivity fails
3. reboot guest, connectivity ok
4. reboot guest, connectivity fails

[MvdLande]

Is there anything I can do to help to resolve this issue?

Best regards,

Maurits

[YanVugenfirer]

Hi Maurits,

Your help will be appreciated.

Please do the following:

1. Please change "Init.UsePublishEvents" parameter to disable. In order to change it go to Device Manager (run dev,g,t.msc) -> Double click on the NetKVM device -> Advanced Tab -> Change the setting and click OK. Driver will be reloaded with the new setting. Try to reproduce the issue.
2. Please send debug logs with debug level 2. Also in advance tab change "Logging.Level" to 2.
   Download DebugViewutility http://technet.microsoft.com/en-us/sysinternals/bb896647.aspx . Run it and: a. enable kernel capture from "Capture"->"Capture Kernel" menu. Also in the "Capture" menu check "Log boot" option. Reboot run DebugView and it should show you the buffered log. Save it and send it to me.

Thanks!
Yan.

[MvdLande]

It has been a while since I reported this problem.

Meanwhile I have updated Centos 6 to the latest centos-plus kernel and applied all available updates.
elrepo has build the latest kernel modules for the 10Gbe network cards.
Currently I'm testing on a HP dl385-g7 with a NC523SFP 10Gbe network adapter.
The NetKVM driver version = 60.63.103.2700

And..... I have not been able te reproduce the issue!

It looks like it's fixed!

Tomorrow I run the test on a different server.

[YanVugenfirer]

I could be nice to know what part was responsible for the failure - the host side is the guest driver. In any case nice to hear the good news.

Sent from my iPhone

On Aug 16, 2012, at 23:32, MvdLande [email protected] wrote:

It has been a while since I reported this problem.

Meanwhile I have updated Centos 6 to the latest centos-plus kernel and applied all available updates.
elrepo has build the latest kernel modules for the 10Gbe network cards.
Currently I'm testing on a HP dl385-g7 with a NC523SFP 10Gbe network adapter.
The NetKVM driver version = 60.63.103.2700

And..... I have not been able te reproduce the issue!

It looks like it's fixed!

Tomorrow I run the test on a different server.

—
Reply to this email directly or view it on GitHub.

[MvdLande]

I moved the VM to another host. This host is not yet updated to the latest el6 kernel and network drivers.
On this host I'm able to reproduce the "toggle" effect.
I still use The NetKVM driver version = 60.63.103.2700

It looks like te problem is related to the host system,not the guest driver.

I'll collect the debug logs and send them to you.

[MvdLande]

Please change "Init.UsePublishEvents" parameter to disable.

With "Init.UsePublishEvents" set to "disable" I cannot reproduce the "toggle effect"
http://www.linux-kvm.org/page/Internals_of_NDIS_driver_for_VirtIO_based_network_adapter

Even when I re-enable the setting the "toggle effect" does not happen.

I'll reinstall this VM and check if I can reproduce the issue.

[MvdLande]

I have re-installed the VM:

OS: Windows 7 Enterprise x64 with SP1
Guest drivers: 60.63.103.2700 see http://people.redhat.com/vrozenfe/
No updates applied and firewall turned off

At this time the issue does not show up.

I applied all windows updates available on 2012-08-17 and the rebooted the VM several times

The issue still does not show up.

I have downgraded the drivers to version 60.63.103.2600

The issue still does not show up.

I'll install the drivers from virtio-win-0.1-22 these are the ones I installed when I first encountered this issue

The issue still does not show up.

Installed the drivers from http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-0.1-30.iso

The issue still does not show up.

I do not know why I have seen the "toggle effect" a few times this morning.
It appears to be working know.

One of the mayor changes since I first encountered the "toggle effect" is that the core network has been upgraded to fix the following:

Meshing (CR_0000103714) —ARP replies are not always forwarded through the mesh, resulting in loss of
communication to affected end nodes.

(we are using HP switch meshing)

I'll install the latest drivers on a few servers and monitor it for a while. If the "toggle effect" does not show up I'll close this issue.

[MvdLande]

Alas, the "toggle effect" showed up on a windows server 2008 R2 SP1 Standard Edition.
I have installed the driver version 60.63.103.3000

With "Init.UsePublishEvents" set to "disable" the driver works fine. Network connectivity does not toggle after a reboot.

After enabling it the "toggle effect" is back.

I have got a debug log file I'll send it to your redhat email address.

[YanVugenfirer]

Thanks a lot for the effort !

Sent from my iPhone

On Aug 18, 2012, at 22:56, MvdLande [email protected] wrote:

Alas, the "toggle effect" showed up on a windows server 2008 R2 SP1 Standard Edition.
I have installed the driver version 60.63.103.3000

With "Init.UsePublishEvents" set to "disable" the driver works fine. Network connectivity does not toggle after a reboot.

After enabling it the "toggle effect" is back.

I have got a debug log file I'll send it to your redhat email address.

—
Reply to this email directly or view it on GitHub.

[YanVugenfirer]

Hi,

The actual bug that was responsible for the network availability toggle was finally caught. Fix will be added soon.

Best regards,
Yan.

[MvdLande]

The actual bug that was responsible for the network availability toggle was finally caught. Fix will be added soon

This is great news! Let me know when the new drivers are available, I’ll be happy to test them.

Best regards,

Maurits

Van: Yan Vugenfirer [mailto:[email protected]]
Verzonden: dinsdag 4 september 2012 14:04
Aan: YanVugenfirer/kvm-guest-drivers-windows
CC: Maurits van de Lande
Onderwerp: Re: [kvm-guest-drivers-windows] Network connections toggle after reboot with NetKVM version 60.63.103.nnnn on Windows Server 2008 R2 (#5)

Hi,

The actual bug that was responsible for the network availability toggle was finally caught. Fix will be added soon.

Best regards,
Yan.

—
Reply to this email directly or view it on GitHubhttps://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues/5#issuecomment-8260532.

[ohwebmaster]

Hi,

Is there any update to this? I suffer from exactly the same problem with 2008 R2 x64 guests.

[MvdLande]

I do not know if an update is available. There is a fix which works in my case.

Go to the network adapter properties -> advanced tab

Set Init.UsePublishEvents = disabled

I’m using virtio-win version 61.63.103.3000

[cid:[email protected]]

Best regards,

Maurits van de Lande

Van: ohwebmaster [mailto:[email protected]]
Verzonden: donderdag 11 oktober 2012 14:21
Aan: YanVugenfirer/kvm-guest-drivers-windows
CC: Maurits van de Lande
Onderwerp: Re: [kvm-guest-drivers-windows] Network connections toggle after reboot with NetKVM version 60.63.103.nnnn on Windows Server 2008 R2 (#5)

Hi,

Is there any update to this? I suffer from exactly the same problem with 2008 R2 x64 guests.

—
Reply to this email directly or view it on GitHubhttps://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues/5#issuecomment-9337774.

[YanVugenfirer]

The code is already inside. I can check then there will be binary update
for Fedora.

And yes - you can disable "UsePublishEvents " as a workaround.

Best regards,
Yan.

On Thu, Oct 11, 2012 at 2:27 PM, MvdLande [email protected] wrote:

I do not know if an update is available. There is a fix which works in my
case.

Go to the network adapter properties -> advanced tab

Set Init.UsePublishEvents = disabled

I’m using virtio-win version 61.63.103.3000

[cid:[email protected]]

Best regards,

Maurits van de Lande

Van: ohwebmaster [mailto:[email protected]]
Verzonden: donderdag 11 oktober 2012 14:21
Aan: YanVugenfirer/kvm-guest-drivers-windows
CC: Maurits van de Lande
Onderwerp: Re: [kvm-guest-drivers-windows] Network connections toggle
after reboot with NetKVM version 60.63.103.nnnn on Windows Server 2008 R2
(#5)

Hi,

Is there any update to this? I suffer from exactly the same problem with
2008 R2 x64 guests.

—
Reply to this email directly or view it on GitHub<
https://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues/5#issuecomment-9337774>.

—
Reply to this email directly or view it on GitHubhttps://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues/5#issuecomment-9337951.

---

Daynix Computing LTD
Yan Vugenfirer, Technology expert and consultant
Email: [email protected]
Phone: [phone]
Web: www.daynix.com

[ohwebmaster]

Thanks Yan. We are disabling 'UsePublishEvents' as a workaround until a binary update is released.

[dennisjac]

Apparently there is a new binary build for the drivers now:
http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-0.1-49.iso

Can anyone confirm that this build contains the fix for this issue?

[YanVugenfirer]

Yes, it should contain the fix.

Best regards,
Yan.

Sent from my iPad

On Jan 4, 2013, at 12:05 AM, Dennis Jacobfeuerborn [email protected] wrote:

Apparently there is a new binary build for the drivers now:
http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-0.1-49.iso

Can anyone confirm that this build contains the fix for this issue?

—
Reply to this email directly or view it on GitHub.

[MvdLande]

Hello Yan,

I have installed the above mentioned drivers a few weeks ago. All is working fine! Could it be that those drivers utilize the driver offload functions to KVM more than the previous drivers? It appears like the VM's perform better then before. (disk and net I/O)

Best regards,

Maurits

[YanVugenfirer]

Hi Maurits,

I the latest version we did several improvements:

1. Fixed published indices feature (should reduce CPU usage and increase
   performance with large packets) feature.
2. Done refactoring and simplified some of the code paths related to
   offloads.
3. Verified offloads with new MS HCK certification tool kit.

So it is expected to get some improvement in performance.

Best regards,
Yan.

On Thu, Jan 24, 2013 at 9:54 AM, MvdLande [email protected] wrote:

Hello Yan,

I have installed the above mentioned drivers a few weeks ago. All is
working fine! Could it be that those drivers utilize the driver offload
functions to KVM more than the previous drivers? It appears like the VM's
perform better then before. (disk and net I/O)

Best regards,

Maurits

—
Reply to this email directly or view it on GitHubhttps://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues/5#issuecomment-12640995.

---

Daynix Computing LTD
Yan Vugenfirer, CEO
Email: [email protected]
Phone: [phone]
Web: www.daynix.com

[necarnot]

Hi,

No one may care, but just to say I was hit by the very same issue (2008 R2 std 64b) with a virtio in version 60.63.xxx.xxx don't remember.
The workaround described above worked, but I decided to upgrade this driver by using virtio-win-0.1-74 and I restored the original driver setting (re-enabled it), and though, now, after reboots, the NIC keeps working, so :

THANK YOU :)

[YanVugenfirer]

Thanks for update.