Friday, June 17, 2011

NX-OS 'Vlans Err-disabled on Trunk' due to 'L4 protocol CAM entry allocation failure'

n7010# sh cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute

Device-ID             Local Intrfce Hldtme Capability  Platform      Port ID
n7010(JAF1447ALEM)     Eth2/41       177    R S I s   N7K-C7010     Eth2/42

n7010(JAF1447ALEM)     Eth2/42       177    R S I s   N7K-C7010     Eth2/41

n7010#
n7010# sh spanning-tree

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    8193
             Address     f025.72a5.a3c1
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    8193   (priority 8192 sys-id-ext 1)
             Address     f025.72a5.a3c1
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth2/41          Desg FWD 4         128.297  P2p
Eth2/42          Back BLK 4         128.298  P2p

n7010#
n7010# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
n7010(config)# int vlan 1
n7010(config-if)# ip access-group Test_ACL in
Note: ACL Test_ACL does not exist. Traffic will be implicitly denied
n7010(config-if)#
n7010(config-if)# exit
n7010(config)#
n7010(config)# ip access-list Test_ACL
n7010(config-acl)# deny 1 any any
n7010(config-acl)# deny 2 any any
n7010(config-acl)# deny 3 any any
n7010(config-acl)# deny 4 any any
n7010(config-acl)# deny 5 any any
n7010(config-acl)# deny 6 any any
n7010(config-acl)# deny 7 any any
ERROR: L4 protocol CAM entry allocation failure
n7010(config-acl)#
n7010(config-acl)# sh access-list Test_ACL

IP access list Test_ACL
        10 deny icmp any any
        20 deny igmp any any
        30 deny 3 any any
        40 deny 4 any any
        50 deny 5 any any
        60 deny tcp any any
n7010(config-acl)#
n7010(config-acl)# int vlan 1
n7010(config-if)# no ip access-group Test_ACL in
n7010(config-if)#
n7010(config-if)# ip access-list Test_ACL
n7010(config-acl)# deny 7 any any
n7010(config-acl)# sh ip access-list Test_ACL

IP access list Test_ACL
        10 deny icmp any any
        20 deny igmp any any
        30 deny 3 any any
        40 deny 4 any any
        50 deny 5 any any
        60 deny tcp any any
        70 deny 7 any any
n7010(config-acl)#
n7010(config-acl)# int vlan 1
n7010(config-if)# ip access-group Test_ACL in
ERROR: L4 protocol CAM entry allocation failure
n7010(config-if)#
n7010(config-if)# int e2/41
n7010(config-if)# shut
n7010(config-if)#
n7010(config-if)# int vlan 1
n7010(config-if)# ip access-group Test_ACL in
n7010(config-if)#
n7010(config-if)# clear log log
2011 Jun 17 19:24:58 n7010 %$ VDC-1 %$ %SYSLOG-1-SYSTEM_MSG : Logging logfile (messages) cleared by user
n7010(config-if)#
n7010(config-if)# int e2/41
n7010(config-if)# no shut
n7010(config-if)# sh spanning-tree
No spanning tree instance exists.
n7010(config-if)#
n7010(config-if)# sh int e2/41 trunk

--------------------------------------------------------------------------------
Port          Native  Status        Port
              Vlan                  Channel
--------------------------------------------------------------------------------
Eth2/41       1       trunking      --

--------------------------------------------------------------------------------
Port          Vlans Allowed on Trunk
--------------------------------------------------------------------------------
Eth2/41       1-3967,4048-4093

--------------------------------------------------------------------------------
Port          Vlans Err-disabled on Trunk
--------------------------------------------------------------------------------
Eth2/41       1-8

--------------------------------------------------------------------------------
Port          STP Forwarding
--------------------------------------------------------------------------------
Eth2/41       none

--------------------------------------------------------------------------------
Port          Vlans in spanning tree forwarding state and not pruned
--------------------------------------------------------------------------------
Eth2/41       none

--------------------------------------------------------------------------------
Port          Vlans Forwarding on FabricPath
--------------------------------------------------------------------------------
Eth2/41       none
n7010(config-if)#
n7010(config-if)# sh log log
2011 Jun 17 19:24:58 n7010 %SYSLOG-1-SYSTEM_MSG : Logging logfile (messages) cleared by user
2011 Jun 17 19:25:04 n7010 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet2/41 is admin up .
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-SPEED: Interface Ethernet2/42, operational speed changed to 1 Gbps
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-IF_DUPLEX: Interface Ethernet2/42, operational duplex mode changed to Full
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet2/42, operational Receive Flow Control state changed to off
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet2/42, operational Transmit Flow Control state changed to off
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-SPEED: Interface Ethernet2/41, operational speed changed to 1 Gbps
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-IF_DUPLEX: Interface Ethernet2/41, operational duplex mode changed to Full
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet2/41, operational Receive Flow Control state changed to off
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet2/41, operational Transmit Flow Control state changed to off
2011 Jun 17 19:25:07 n7010 %ACLMGR-3-ACLMGR_VERIFY_FAIL: Verify failed: client 8100016E, L4 protocol CAM entry allocation failure
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-IF_SEQ_ERROR: Error ("L4 protocol CAM entry allocation failure") communicating with MTS_SAP_ACLMGR for opcode MTS_OPC_ETHPM_PORT_LOGICAL_BRINGUP (RID_PORT: Ethernet2/42)
2011 Jun 17 19:25:07 n7010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 1-8 on Interface Ethernet2/42 are being suspended. (Reason: L4 protocol CAM entry allocation failure)
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-IF_UP: Interface Ethernet2/42 is up in mode trunk
2011 Jun 17 19:25:07 n7010 %ACLMGR-3-ACLMGR_VERIFY_FAIL: Verify failed: client 8100016E, L4 protocol CAM entry allocation failure
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-IF_SEQ_ERROR: Error ("L4 protocol CAM entry allocation failure") communicating with MTS_SAP_ACLMGR for opcode MTS_OPC_ETHPM_PORT_LOGICAL_BRINGUP (RID_PORT: Ethernet2/41)
2011 Jun 17 19:25:07 n7010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 1-8 on Interface Ethernet2/41 are being suspended. (Reason: L4 protocol CAM entry allocation failure)
2011 Jun 17 19:25:07 n7010 %ETHPORT-5-IF_UP: Interface Ethernet2/41 is up in mode trunk
n7010(config-if)#

Root Cause:
L4 TCAM allocation failed due to 4 lines of IPv4 ACL entries with uncommon IP protocol numbers (those that are not translated to well-known tcp, udp, icmp, igmp, ospf, eigrp, etc).

1 comment:

  1. Hi Yap,

    Thats a very good blog I had a first hand exp while migrating N7K, you know vlan wont go in Err-disable state if the box is in Use (Int vlan Up or ACL is applied) because it will give you warning and wont accept the command. Vlan will go err disable only when you have applied ACL on an interface which is in down state and as soon as you make interface uo you will see allocation failure. Therefore simulating this in real time is quite difficult.. Just sharing my experience..good work keep it up!

    ReplyDelete