We are experiencing the same problem on multiple hardware platforms:

Kernel 2.6.18-194.el5
   - bnx2 (2.0.8b)
   - e1000 (7.3.21-k4-NAPI)


BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0]
CPU 7:
Modules linked in: 8021q bridge autofs4 lockd sunrpc bonding ipv6 xfrm_nalgo crypto_api dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport st e1000e serio_raw hpilo shpchp bnx2(U) pcspkr sg dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod qla2xxx scsi_transport_fc cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 0, comm: swapper Tainted: G      2.6.18-194.el5 #1
RIP: 0010:[<ffffffff80065c20>]  [<ffffffff80065c20>] .text.lock.spinlock+0x26/0x30
RSP: 0018:ffff81031feb3cb0  EFLAGS: 00000286
RAX: ffff81031feabfd8 RBX: ffff8103125b16c0 RCX: ffff8103125b1000
RDX: ffff81031b016710 RSI: ffff8103125b1000 RDI: ffff8103125b1758
RBP: ffff81031feb3c30 R08: ffff8103176f2e80 R09: ffff810316545180
R10: ffff81031feb3db8 R11: 00000000000000c8 R12: ffffffff8005ec8e
R13: ffff8103125b1758 R14: ffffffff8007922b R15: ffff81031feb3c30
FS:  0000000000000000(0000) GS:ffff81031fe283c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 000000001efbb670 CR3: 0000000310e8a000 CR4: 00000000000006e0

Call Trace:
 <IRQ>  [<ffffffff884ce002>] :bonding:rlb_arp_recv+0xb2/0x146
 [<ffffffff800209ba>] netif_receive_skb+0x43e/0x49f
 [<ffffffff882abe50>] :bnx2:bnx2_poll+0x1245/0x14e3
 [<ffffffff80151248>] __next_cpu+0x19/0x28
 [<ffffffff8008ccb0>] find_busiest_group+0x20d/0x621
 [<ffffffff800c907c>] free_pages_bulk+0x1f0/0x268
 [<ffffffff8000c88a>] net_rx_action+0xac/0x1e0
 [<ffffffff80012409>] __do_softirq+0x89/0x133
 [<ffffffff8005f2fc>] call_softirq+0x1c/0x28
 [<ffffffff8006dba8>] do_softirq+0x2c/0x85
 [<ffffffff8006da30>] do_IRQ+0xec/0xf5
 [<ffffffff8005e615>] ret_from_intr+0x0/0xa
 <EOI>  [<ffffffff8019e040>] acpi_processor_idle_simple+0x17d/0x30e
 [<ffffffff8019df2f>] acpi_processor_idle_simple+0x6c/0x30e
 [<ffffffff8019dec3>] acpi_processor_idle_simple+0x0/0x30e
 [<ffffffff8019dec3>] acpi_processor_idle_simple+0x0/0x30e
 [<ffffffff800497be>] cpu_idle+0x95/0xb8
 [<ffffffff80078997>] start_secondary+0x498/0x4a7


BUG: soft lockup - CPU#3 stuck for 10s! [swapper:0]
CPU 3:
Modules linked in: 8021q bridge autofs4 lockd sunrpc bonding ipv6 xfrm_nalgo crypto_api dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport st sg hpilo shpchp e1000e pcspkr bnx2(U) serio_raw dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod qla2xxx scsi_transport_fc cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 0, comm: swapper Tainted: G      2.6.18-194.el5 #1
RIP: 0010:[<ffffffff80065c20>]  [<ffffffff80065c20>] .text.lock.spinlock+0x26/0x30
RSP: 0018:ffff81010afbfd60  EFLAGS: 00000286
RAX: ffff81010afb9fd8 RBX: ffff8103170826c0 RCX: ffff810317082000
RDX: ffff8103152f1710 RSI: ffff810317082000 RDI: ffff810317082758
RBP: ffff81010afbfce0 R08: 0000000000000000 R09: 0000000000000000
R10: ffff8103153d95c0 R11: 00000000000000c8 R12: ffffffff8005ec8e
R13: ffff810317082758 R14: ffffffff8007922b R15: ffff81010afbfce0
FS:  0000000000000000(0000) GS:ffff81031ff236c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000003516003080 CR3: 0000000000201000 CR4: 00000000000006e0

Call Trace:
 <IRQ>  [<ffffffff884c7002>] :bonding:rlb_arp_recv+0xb2/0x146
 [<ffffffff800209ba>] netif_receive_skb+0x43e/0x49f
 [<ffffffff882e33cd>] :e1000e:e1000_receive_skb+0x1b5/0x1d6
 [<ffffffff882e7b27>] :e1000e:e1000_clean_rx_irq+0x27a/0x321
 [<ffffffff882e5bc5>] :e1000e:e1000_clean+0x7c/0x29a
 [<ffffffff8000c88a>] net_rx_action+0xac/0x1e0
 [<ffffffff882e5a55>] :e1000e:e1000_intr_msi+0xd6/0xe0
 [<ffffffff80012409>] __do_softirq+0x89/0x133
 [<ffffffff8005f2fc>] call_softirq+0x1c/0x28
 [<ffffffff8006dba8>] do_softirq+0x2c/0x85
 [<ffffffff8006da30>] do_IRQ+0xec/0xf5
 [<ffffffff8005e615>] ret_from_intr+0x0/0xa
 <EOI>  [<ffffffff8019e040>] acpi_processor_idle_simple+0x17d/0x30e
 [<ffffffff8019df2f>] acpi_processor_idle_simple+0x6c/0x30e
 [<ffffffff8019dec3>] acpi_processor_idle_simple+0x0/0x30e
 [<ffffffff8019dec3>] acpi_processor_idle_simple+0x0/0x30e
 [<ffffffff800497be>] cpu_idle+0x95/0xb8
 [<ffffffff80078997>] start_secondary+0x498/0x4a7

I'm trying to reproduce this on a machine running 2.6.18-194.3.1.el5 and can't.  Yury, do you still have that problem if you upgrade to 2.6.18-194.3.1.el5?

I confirm to have this problem with 2.6.18-194.3.1.el5 on a Dell PowerEdge 1650 with two e1000 NICs.

lspci output:
00:00.0 Host bridge: Broadcom CNB20HE Host Bridge (rev 23)
00:00.1 Host bridge: Broadcom CNB20HE Host Bridge (rev 01)
00:00.2 Host bridge: Broadcom CNB20HE Host Bridge (rev 01)
00:00.3 Host bridge: Broadcom CNB20HE Host Bridge (rev 01)
00:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
00:0f.0 Host bridge: Broadcom CSB5 South Bridge (rev 93)
00:0f.1 IDE interface: Broadcom CSB5 IDE Controller (rev 93)
00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 05)
00:0f.3 ISA bridge: Broadcom CSB5 LPC bridge
01:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit Ethernet Controller (Copper) (rev 02)
01:04.0 Ethernet controller: Intel Corporation 82544EI Gigabit Ethernet Controller (Copper) (rev 02)
01:08.0 PCI bridge: Intel Corporation 80303 I/O Processor PCI-to-PCI Bridge (rev 01)
01:08.1 RAID bus controller: Dell PowerEdge Expandable RAID Controller 3/Di (rev 01)

Hotfix: change to bonding mode 5.

I'm experiencing the same problem on a Dell 2950 and 2.6.18-194.3.1.el5
with two Intel and two Broadcom (the embedded ones) adapters.
I have bonding and VLANs too.
Updated (via scratch install) from rh el 4.5 x86_64 to rh el 5.5 (+updates till today) and I'm experiencing these problems that before I had not.

Changing mode from 6 to 5 workarounds the problem for me too.
# lspci|grep thern
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
09:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
0a:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
0a:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)

I get the same stack trace (I'm going to attach screenshot) both with bonding done on top of a pairs of bnx2 and on top of a pair of e1000e ones.
So it is driver independent

Created attachment 423266 [details]
stack trace with bonding on a paier of intel cards

I had this issue when I switched to mode 6 (alb-balance).


BUG: soft lockup - CPU#1 stuck for 10s! [swapper:0]

Pid: 0, comm:              swapper
EIP: 0060:[<c061dc4c>] CPU: 1
EIP is at _spin_lock_bh+0xd/0x18
 EFLAGS: 00000286    Not tainted  (2.6.18-194.3.1.el5 #1)
EAX: c0749000 EBX: f69a3d50 ECX: f68a8d2c EDX: f69a3800
ESI: f69a3cfc EDI: f69a3d50 EBP: f68fd600 DS: 007b ES: 007b
CR0: 8005003b CR2: b7fc8000 CR3: 00742000 CR4: 000006d0
 [<f8fed0ea>] rlb_arp_recv+0x98/0x11d [bonding]
 [<c05c0aa8>] netif_receive_skb+0x3ac/0x401
 [<f8ba5df3>] tg3_poll+0x64f/0xc28 [tg3]
 [<c041eeda>] __activate_task+0x4a/0x59
 [<c04ee889>] rb_erase+0x176/0x22f
 [<c05c2995>] net_rx_action+0x9c/0x1a7
 [<c042a377>] __do_softirq+0x87/0x114
 [<c04073cf>] do_softirq+0x52/0x9c
 [<c044f158>] __do_IRQ+0x0/0xd6
 [<c04074ce>] do_IRQ+0xb5/0xc3
 [<c0405946>] common_interrupt+0x1a/0x20
 [<c0403bb0>] default_idle+0x0/0x59
 [<c0403be1>] default_idle+0x31/0x59
 [<c0403ca8>] cpu_idle+0x9f/0xb9
 =======================

I can also confirm similar behavior on Cisco UCS B250-M2 blade system using the ixgbe network driver.

In our situation we had physical eth0, eth1, eth2, eth3 which we were using mode 6 to create a bond0.  As soon as the bond0.x interface was brought up during boot the boot process is halted.  If done via command line, a stack trace similar to the above is produced.

Unfortunately the soft-lockups don't really tell me much as it only shows the receive patch not being able to take the lock.  I'll have to try and reproduce this myself.

The CPU is stuck at rlb_update_entry_from_arp() trying to get the lock
_lock_rx_hashtbl(bond) and for some reason, it's unable to get it, so the
watchdog fires showing that back trace. However, this is a consequence
and not a root cause because, in this case, another CPU is holding that
lock leaving others waiting for it long enough to trigger the watchdog.
Therefore, can you get few outputs of sysrq+t and sysrq+w while the
problem is happening?

It could point to us which CPU is holding the lock and why.

thanks,
fbl

I could reproduce this here, so no need to provide sysrq+t or sysrq+w.

Flavio, can you post any new information you have?

Created attachment 432903 [details]
suggested patch

Hi Andy,

The problem has been introduced by the following patch:
[net] bonding: allow arp_ip_targets on separate vlan from bond device

and not fixed by the later patch:
[net] fixup problems with vlans and bonding

The problem happens because in rlb_arp_recv(), the struct bonding *bond
pointer is a vlan's net_device struct instead, so it can either oops or
just hangs on a invalid spinlock. 

I can reproduce both situations following the instructions in the
ticket's summary.

The upstream fixes rlb_arp_recv() to look for the flag IFF_802_1Q_VLAN 
and if it is present, then find the underlying bonding device.

I have the patch backported and it works out on my tests.
Please review.
fbl

I don't think this can count as a regression.  The upstream patch below that adds the code described above was added in 2008 and were it not for the code that added support for arp_ip_targets on a separate VLAN this problem would have never been seen.  Here is the upstream patch we should consider:

commit 6146b1a4da98377e4abddc91ba5856bef8f23f1e
Author: Jay Vosburgh <fubar.com>
Date:   Tue Nov 4 17:51:15 2008 -0800

    bonding: Fix ALB mode to balance traffic on VLANs

This does not diminish the importance, but finding out that something is broken after a feature was added is not a regression.

*** Bug 615996 has been marked as a duplicate of this bug. ***

Created attachment 432929 [details]
bonding-fix-alb-mode-to-balance-traffic-on-vlans.patch

This patch should resolve the issue based on Flavio's analysis.  It is a backport of upstream commits:

6146b1a4da98377e4abddc91ba5856bef8f23f1e bonding: Fix ALB mode to balance traffic on VLANs
2690f8d62e98779c71625dba9a0fd525d8b2263d bonding: Remove debug printk

I have added this to my rhel5 gtest repo and will post to this bug when new test kernels are available.

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

I thought it was a regression because before vlan over bonding was working
and after adding a new feature, the same setup doesn't work anymore.

Regarding to the new patch, the next merges will be a bit more complicated
because the loop is not using vlan_dev_real_dev() as upstream does, so I have
added a macro to deal with it. Also, I intentionally removed pk_type->dev
initialization because it didn't make sense to my eyes for this kernel version,
but I could be overlooking something.

(In reply to comment #18)
> Also, I intentionally removed pk_type->dev
> initialization because it didn't make sense to my eyes for this kernel version,
> but I could be overlooking something.    

Removing it was a good idea.  It was removed today as it caused a panic in a multiple bond configuration.

(In reply to comment #19)
> Removing it was a good idea.  It was removed today as it caused a panic in a
> multiple bond configuration.    

Yeah, let me know when you have the final patch ready.

Created attachment 434836 [details]
bonding-fix-alb-mode-to-balance-traffic-on-vlans-updated.patch

Here is an updated patch.  Feedback is welcome.

My test kernels have been updated to include a patch for this bugzilla.

http://people.redhat.com/agospoda/#rhel5

Please test them and report back your results.

(In reply to comment #27)
> My test kernels have been updated to include a patch for this bugzilla.
> 
> http://people.redhat.com/agospoda/#rhel5
> 
> Please test them and report back your results.

We've been running this for a couple of weeks on one of our production boxes and haven't seen any issues whatsoever.

# uptime 
 20:39:44 up 13 days, 20:30,  1 user,  load average: 1.30, 1.52, 1.56
# uname -a
Linux foo.mozilla.net 2.6.18-212.el5.gtest.89 #1 SMP Mon Aug 16 14:01:15 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

Glad to hear that, Shyam!

I appreciate the feedback.

Andy,
 Finnaly, I got a change to put 2.6.18-212.el5.gtest.89 under a test on a IBM HS21 system with two nics in two switch configuration. Works smoothly for about half a day and perform pretty good (~ 1883 Mbits/sec on multi-node iperf test).
Thanks for the patch and test kernel package. It seems like this bug is fixed.

Andy,

 Could you please share kernel-headers package of your test kernel build? I would like to test GPFS filesystem with bonding on your test kernel and I need to build IBM gpfs modules and some other kernel-dependent staff. kernel-devel is not enough for me.

Glad to know they are working, Yury.  Thanks for that feedback.

I also added all of the headers rpms to my people page here:

http://people.redhat.com/agospoda/#rhel5

Please let me know if any other rpms from that build would be helpful and are not on my people page.

Thank you, Andy for making headers package available. I will report if any of my ongoing stress tests fail. Until now it works and perform perfectly.

in kernel-2.6.18-219.el5
You can download this test kernel from http://people.redhat.com/jwilson/el5

Detailed testing feedback is always welcomed.

It's ok also in my Dell 2950 with kernel 2.6.18-219.el5 and alb bonding and some vlans over bonding of two Broadcom adapters:
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
09:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)

At least in half an hour no error at all.
In dmesg:
Ethernet Channel Bonding Driver: v3.4.0 (October 7, 2008)
bonding: In ALB mode you might experience client disconnections upon reconnection of a link if the bonding module updelay parameter (200 msec) is incompatible with the forwarding delay time of the switch
bonding: MII link monitoring set to 100 ms
ADDRCONF(NETDEV_UP): bond0: link is not ready
bonding: bond0: Adding slave eth0.
bnx2: eth0: using MSI
bonding: bond0: enslaving eth0 as an active interface with a down link.
bonding: bond0: Adding slave eth1.
bnx2: eth1: using MSI
bonding: bond0: enslaving eth1 as an active interface with a down link.
bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex
bonding: bond0: link status up for interface eth0, enabling it in 0 ms.
bonding: bond0: link status definitely up for interface eth0.
bonding: bond0: making interface eth0 the new active one.
bonding: bond0: first active interface up!
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex
bonding: bond0: link status up for interface eth1, enabling it in 200 ms.
bonding: bond0: link status definitely up for interface eth1.
802.1Q VLAN Support v1.8 Ben Greear ...
All bugs added by David S. Miller ...
bond0: no IPv6 routers present
bond0.13: no IPv6 routers present
bond0.139: no IPv6 routers present
bond0.221: no IPv6 routers present
bond0.66: no IPv6 routers present
bond0.68: no IPv6 routers present
bond0.800: no IPv6 routers present

Gianluca

Excellent!  Thanks for the feedback, Gianluca.

(In reply to comment #40)
> in kernel-2.6.18-219.el5
> You can download this test kernel from http://people.redhat.com/jwilson/el5
> 
> Detailed testing feedback is always welcomed.

# uname -a
Linux foo.mozilla.net 2.6.18-219.el5 #1 SMP Thu Sep 9 17:10:23 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
# uptime
 07:41:25 up 9 days, 10:42,  1 user,  load average: 4.77, 4.59, 4.59

All good, same production server as in comment #31 and seems to be fine.

*** Bug 602197 has been marked as a duplicate of this bug. ***

This patch works for us. Can you please let us know when this patch will be available in redhat-release, as it fails on latest kernel also 2.6.18-194.11.4.el5

Thanks

(In reply to comment #46)
> This patch works for us. Can you please let us know when this patch will be
> available in redhat-release, as it fails on latest kernel also
> 2.6.18-194.11.4.el5
> 
> Thanks

Right now it will not be available until the official RHEL5.6 update ships.  As of today, I cannot say for sure when RHEL5.6 will ship.

If you need support in a RHEL5.5 update kernel, please go through our support portal:

https://access.redhat.com/home

and request that this patch is added.  You can reference this bugzilla if needed.

I DO have a case opened for this bug and latest answer has been (right today after asking for a date):
> Any information about an official and supported update containing the fix? 
At this moment the errata which will contain the fix is not yet confirmed.

My case is 
Case Number      : 00332049 
Case Open Date   : 2010-06-11 14:42:07

If you see the date, we are about 3 months and no official solution yet....
So the question is: what is the added value of active subscription?

Excuse me for being a little sarcastic...

Gianluca

(In reply to comment #48)
> I DO have a case opened for this bug and latest answer has been (right today
> after asking for a date):
> > Any information about an official and supported update containing the fix? 
> At this moment the errata which will contain the fix is not yet confirmed.
> 
> My case is 
> Case Number      : 00332049 
> Case Open Date   : 2010-06-11 14:42:07
> 
> If you see the date, we are about 3 months and no official solution yet....
> So the question is: what is the added value of active subscription?
> 
> Excuse me for being a little sarcastic...
> 
> Gianluca

I'm glad you opened a support ticket for this.  Opening tickets is really the only way for us to know that this fix is critical enough to paying customers that we need to add it to the currently shipping (in this case 2.6.18-194) kernel stream.  Thanks for doing that.

It can be a bit confusing, but we open a bug for each release that requires a patch.  This bug will address the problem on upcoming RHEL5.6 and bug 630540 will address the problem on already released RHEL5.5 since there was enough customer demand (not just noise in bugzilla) to fix it before RHEL5.6 shipped.  I made some noise over in bug 630540, so hopefully things will move forward with a fix in RHEL5.5.

(In reply to comment #49)
> I made some noise over in bug 630540, so hopefully things will move forward
> with a fix in RHEL5.5.

I saw it, thanks.
I (we) keep on waiting for a fix, then

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
An attempt to create a VLAN interface on a bond of two bnx2 adapters in two switch configurations resulted in a soft lockup after a few seconds. This was caused by an incorrect use of a bonding pointer. With this update, soft lockups no longer occurs and creating a VLAN interface works as expected.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0017.html