BNX2 Link Disabled When High Traffic

Posted by blogadmin on June 19, 2012

Recently I had strange issue with one of my xen sever running on ProLiant ML370 G5 hardware. As I started rsync from other machine to this box which is in same LAN, the port goes in to disabled state. The server works fine in normal network load. The interface starts to flap after starting rsync.I changed cables, ports, tried different switch, swapped eth0 to eth1 but no joy. After googling I found solution to this problem is to increase Ring RX buffer.


My /var/log/messges shown following lines.

Jun  4 10:20:30 xen02 kernel: [1287626.110422] bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Jun  4 10:20:30 xen02 kernel: [1287626.110671] br0: port 1(eth1) entering learning state
Jun  4 10:20:45 xen02 kernel: [1287641.112015] br0: port 1(eth1) entering forwarding state
Jun  4 10:21:31 xen02 kernel: [1287686.932190] br0: port 1(eth1) entering disabled state

Following is the network card info shown using lspci

03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)

Module info. I am using latest firmware from debian.

# modinfo bnx2
filename:       /lib/modules/2.6.32-5-xen-amd64/kernel/drivers/net/bnx2.ko
firmware:       bnx2/bnx2-rv2p-09ax-5.0.0.j3.fw
firmware:       bnx2/bnx2-rv2p-09-5.0.0.j3.fw
firmware:       bnx2/bnx2-mips-09-5.0.0.j3.fw
firmware:       bnx2/bnx2-rv2p-06-5.0.0.j3.fw
firmware:       bnx2/bnx2-mips-06-5.0.0.j3.fw
version:        2.0.2
license:        GPL
description:    Broadcom NetXtreme II BCM5706/5708/5709/5716 Driver
author:         Michael Chan <mchan@broadcom.com>
srcversion:     8F18532C0D4FFBC79D95E9D
alias:          pci:v000014E4d0000163Csv*sd*bc*sc*i*
alias:          pci:v000014E4d0000163Bsv*sd*bc*sc*i*
alias:          pci:v000014E4d0000163Asv*sd*bc*sc*i*
alias:          pci:v000014E4d00001639sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016ACsv*sd*bc*sc*i*
alias:          pci:v000014E4d000016AAsv*sd*bc*sc*i*
alias:          pci:v000014E4d000016AAsv0000103Csd00003102bc*sc*i*
alias:          pci:v000014E4d0000164Csv*sd*bc*sc*i*
alias:          pci:v000014E4d0000164Asv*sd*bc*sc*i*
alias:          pci:v000014E4d0000164Asv0000103Csd00003106bc*sc*i*
alias:          pci:v000014E4d0000164Asv0000103Csd00003101bc*sc*i*
depends:
vermagic:       2.6.32-5-xen-amd64 SMP mod_unload modversions
parm:           disable_msi:Disable Message Signaled Interrupt (MSI) (int)

To check your current Ring buffer setting, use below command

# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             1020
RX Mini:        0
RX Jumbo:       4080
TX:             255
Current hardware settings:
RX:             255
RX Mini:        0
RX Jumbo:       0
TX:             255

To manually increase it to 1020, which is max in my case, use below command

ethtool -G eth1 rx 1020

To make your changes permanent add that command in network configuration file like below.On Debian this file is /etc/network/interfaces. My interface is br0 bridge

iface br0 inet static
        address 192.168.0.221
        netmask 255.255.255.0
        gateway 192.168.0.101
        bridge_ports eth1
        up ethtool -G eth1 rx 1020
        up ethtool -G eth0 rx 1020
Share Button

Filed under Networking

Tagged ,

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>