+ Reply to Thread
Results 1 to 14 of 14
  1. Senior Member sucanushie's Avatar
    Join Date
    Apr 2013
    Location
    Canada
    Posts
    159

    Certifications
    MCTS, Windows 7, MCITP: EDA Win7,CCNA Security,CCNA Voice, CCNP R&S
    #1

    Default Nexus VPC with Vmware VDS

    Does anyone have any experience setting up VPC LACP connection with vmware virtual distributed switch?

    I have 2 Nexus switches and 4 esx hosts with 2 10G nics each.

    I've connected 1 10G nic per host to each nexus switch. 4 connections per nexus switch.

    I created the LAG group on the VDS and added each nic into the port channel.

    On the nexus switch only one link on each is coming up, the others are suspend.

    Here is the config on the nexus side.
    SW1
    interface Ethernet1/5
    description NTNX1
    switchport mode trunk
    switchport trunk allowed vlan 106,116
    channel-group 4 mode active
    no shutdown


    interface Ethernet1/6
    description NTNX2
    switchport mode trunk
    switchport trunk allowed vlan 106,116
    channel-group 4 mode active
    no shutdown


    interface Ethernet1/7
    description NTNX3
    switchport mode trunk
    switchport trunk allowed vlan 106,116
    channel-group 4 mode active
    no shutdown


    interface Ethernet1/8
    description NTNX4
    switchport mode trunk
    switchport trunk allowed vlan 106,116
    channel-group 4 mode active
    no shutdown

    interface port-channel4
    speed 10000
    description NTNX1 VPC
    switchport mode trunk
    switchport trunk allowed vlan 106,116
    spanning-tree port type edge trunk
    vpc 4

    Switch 2

    interface Ethernet1/5
    description NTNX1
    switchport mode trunk
    switchport trunk allowed vlan 106,116
    channel-group 4 mode active
    no shutdown


    interface Ethernet1/6
    description NTNX2
    switchport mode trunk
    switchport trunk allowed vlan 106,116
    channel-group 4 mode active
    no shutdown


    interface Ethernet1/7
    description NTNX3
    switchport mode trunk
    switchport trunk allowed vlan 106,116
    channel-group 4 mode active
    no shutdown


    interface Ethernet1/8
    description NTNX4
    switchport mode trunk
    switchport trunk allowed vlan 106,116
    channel-group 4 mode active
    no shutdown

    interface port-channel4
    speed 10000
    description NTNX1 VPC
    switchport mode trunk
    switchport trunk allowed vlan 106,116
    spanning-tree port type edge trunk
    vpc 4

    and this is what I'm seeing with the sh port-channel summary

    4 Po4(SU) Eth LACP Eth1/5(P) Eth1/6(s) Eth1/7(s)
    Eth1/8(s)


    Thanks in advance.
    Reply With Quote Quote  

  2. SS -->
  3. Went to the dark side.... Moderator networker050184's Avatar
    Join Date
    Jul 2007
    Posts
    11,665

    Certifications
    CCNA, CCNP, CCIP, JNCIA-JUNOS, JNCIS-SP, JNCIP-SP, MCA200
    #2
    Yeah we have quite a few of these set up. Suspended usually mean it isn't receiving LACP packets on those ports. Have you made sure you VDS is all good to go?
    An expert is a man who has made all the mistakes which can be made.
    Reply With Quote Quote  

  4. Senior Member sucanushie's Avatar
    Join Date
    Apr 2013
    Location
    Canada
    Posts
    159

    Certifications
    MCTS, Windows 7, MCITP: EDA Win7,CCNA Security,CCNA Voice, CCNP R&S
    #3
    It looks like it's setup on the VDS correctly.

    I setup the nexus VPC config first, then the VDS after, I'm not sure if the order matters.

    Unfortunately I didn't setup the VDS, the Vmware guys did it, and I only know a little bit from a few doc's I've read.

    It's a net new setup, so I may as him to delete the VDS and start over with me watching.

    I'm confident my config on the nexus side was correct, I just didn't know if maybe it was a limitation on the VDS side of things.

    There isn't a whole lot of documentation out there regarding Nexus VPC and VDS Lag groups.
    Reply With Quote Quote  

  5. Padawan d4nz1g's Avatar
    Join Date
    May 2013
    Location
    Brazil
    Posts
    424

    Certifications
    CCNP, BCNE, CCNA, CCNA Sec, MCSA2k8, IPv6 Silver
    #4
    sh lacp interface ex/x - see if the ports are receiving lacp pdus

    show vpc consistency [global | interface] - make sure that your vpc is configured correctly

    As networker050184 mentioned, LACP may be misconfigured on these ports...You can confirm it with the show lacp command.
    2017 - CCIE RS
    Labbing, labbing, labbing.
    Reply With Quote Quote  

  6. Senior Member sucanushie's Avatar
    Join Date
    Apr 2013
    Location
    Canada
    Posts
    159

    Certifications
    MCTS, Windows 7, MCITP: EDA Win7,CCNA Security,CCNA Voice, CCNP R&S
    #5
    Interface Ethernet1/7 is suspended
    Channel group is 4 port channel is Po4
    PDUs sent: 185246
    PDUs rcvd: 4221
    Markers sent: 0
    Markers rcvd: 0
    Marker response sent: 0
    Marker response rcvd: 0
    Unknown packets rcvd: 0
    Illegal packets rcvd: 0
    Lag Id: [ [(7f9b, 0-23-4-ee-be-1, 8004, 8000, 107), (ffff, c-c4-7a-49-8c-ce, b, ff, 8002)] ]
    Operational as aggregated link since Wed Dec 31 20:00:00 1969


    Local Port: Eth1/7 MAC Address= 0-23-4-ee-be-1
    System Identifier=0x8000,0-23-4-ee-be-1
    Port Identifier=0x8000,0x107
    Operational key=32772
    LACP_Activity=active
    LACP_Timeout=Long Timeout (30s)
    Synchronization=NOT_IN_SYNC
    Collecting=false
    Distributing=false
    Partner information refresh timeout=Long Timeout (90s)
    Actor Admin State=(Ac-1:To-1:Ag-1:Sy-0:Co-0i-0e-0:Ex-0)
    Actor Oper State=(Ac-1:To-0:Ag-1:Sy-0:Co-0i-0e-0:Ex-0)
    Neighbor: 0x8002
    MAC Address= c-c4-7a-49-8c-ce
    System Identifier=0xffff, Port Identifier=0xff,0x8002
    Operational key=11
    LACP_Activity=unknown
    LACP_Timeout=Long Timeout (30s)
    Synchronization=NOT_IN_SYNC
    Collecting=false
    Distributing=false
    Partner Admin State=(Ac-0:To-1:Ag-0:Sy-0:Co-0i-0e-0:Ex-0)
    Partner Oper State=(Ac-1:To-0:Ag-1:Sy-0:Co-0i-0e-0:Ex-0)
    Aggregate or Individual(True=1)= 2

    Legend:
    Type 1 : vPC will be suspended in case of mismatch


    Name Type Local Value Peer Value
    ------------- ---- ---------------------- -----------------------
    Shut Lan 1 No No
    STP Port Type 1 Edge Trunk Port Edge Trunk Port
    STP Port Guard 1 None None
    STP MST Simulate PVST 1 Default Default
    lag-id 1 [(7f9b, [(7f9b,
    0-23-4-ee-be-1, 8004, 0-23-4-ee-be-1, 8004,
    0, 0), (ffff, 0, 0), (ffff,
    c-c4-7a-49-8d-e, b, 0, c-c4-7a-49-8d-e, b, 0,
    0)] 0)]
    mode 1 active active
    Speed 1 10 Gb/s 10 Gb/s
    Duplex 1 full full
    Port Mode 1 trunk trunk
    Native Vlan 1 1 1
    MTU 1 1500 1500
    Admin port mode 1
    Switchport MAC Learn 2 Enable Enable
    vPC card type 1 Empty Empty
    Allowed VLANs - 106,116 106,116
    Local suspended VLANs - - -
    Reply With Quote Quote  

  7. Padawan d4nz1g's Avatar
    Join Date
    May 2013
    Location
    Brazil
    Posts
    424

    Certifications
    CCNP, BCNE, CCNA, CCNA Sec, MCSA2k8, IPv6 Silver
    #6
    Well, you are receiving lacpdus but the state is not in sync...weird...

    Do you see anything when issuing sh lacp neighbor interface port-channel xx?
    2017 - CCIE RS
    Labbing, labbing, labbing.
    Reply With Quote Quote  

  8. Senior Member sucanushie's Avatar
    Join Date
    Apr 2013
    Location
    Canada
    Posts
    159

    Certifications
    MCTS, Windows 7, MCITP: EDA Win7,CCNA Security,CCNA Voice, CCNP R&S
    #7
    Flags: S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
    A - Device is in Active mode P - Device is in Passive mode
    port-channel4 neighbors
    Partner's information
    Partner Partner Partner
    Port System ID Port Number Age Flags
    Eth1/5 65535,c-c4-7a-49-8d-e 0x8002 145003 SA


    LACP Partner Partner Partner
    Port Priority Oper Key Port State
    255 0xb 0x3d


    Partner's information
    Partner Partner Partner
    Port System ID Port Number Age Flags
    Eth1/6 65535,c-c4-7a-49-8d-34 0x8002 148616 SA


    LACP Partner Partner Partner
    Port Priority Oper Key Port State
    255 0xb 0x5


    Partner's information
    Partner Partner Partner
    Port System ID Port Number Age Flags
    Eth1/7 65535,c-c4-7a-49-8c-ce 0x8002 0 SA


    LACP Partner Partner Partner
    Port Priority Oper Key Port State
    255 0xb 0x5


    Partner's information
    Partner Partner Partner
    Port System ID Port Number Age Flags
    Eth1/8 65535,c-c4-7a-4c-b-5a 0x8002 0 SA


    LACP Partner Partner Partner
    Port Priority Oper Key Port State
    255 0xb 0xd
    Reply With Quote Quote  

  9. Senior Member sucanushie's Avatar
    Join Date
    Apr 2013
    Location
    Canada
    Posts
    159

    Certifications
    MCTS, Windows 7, MCITP: EDA Win7,CCNA Security,CCNA Voice, CCNP R&S
    #8
    Just an update.

    Was able to work on this today.

    If I create a VPC for each host(1 link going to each switch) and created a lag group for each on the VDS and they all came up no issues.

    I tried to lower the bundle down to 4 links and only 2 came up and 2 were suspended as I was seeing when I was trying to do 8.
    Reply With Quote Quote  

  10. Senior Member sucanushie's Avatar
    Join Date
    Apr 2013
    Location
    Canada
    Posts
    159

    Certifications
    MCTS, Windows 7, MCITP: EDA Win7,CCNA Security,CCNA Voice, CCNP R&S
    #9
    starting to think that it's a limitation.

    I took VPC out of the equation.

    2 host plugged into the same switch. Both ports are members of the same port channel and both NICS on the esx side are members of the lag group.

    I get the same result. One link up the other is suspended.
    Reply With Quote Quote  

  11. Padawan d4nz1g's Avatar
    Join Date
    May 2013
    Location
    Brazil
    Posts
    424

    Certifications
    CCNP, BCNE, CCNA, CCNA Sec, MCSA2k8, IPv6 Silver
    #10
    It looks like everything is fine with your nexus boxes...

    I'm not a vmware guy, but what version are you using? I know there was a limitation when creating LAGs with vmware dvs, but I don't recall what version solved this issue.
    2017 - CCIE RS
    Labbing, labbing, labbing.
    Reply With Quote Quote  

  12. Senior Member sucanushie's Avatar
    Join Date
    Apr 2013
    Location
    Canada
    Posts
    159

    Certifications
    MCTS, Windows 7, MCITP: EDA Win7,CCNA Security,CCNA Voice, CCNP R&S
    #11
    I was thinking the Distributed Switch acted as one logical switch for all host, but that doesn't seem to be the case.

    You can only bundle physical nics from the same host.

    So in my case with 4 hosts with 2 nics each, I would have to create 4 separate lag groups, one per host.

    I was thinking the Distributed Switch acted almost like VSS.

    We ended up just making all 8 links individual trunks and leveraged VMware to teaming and failover.
    Reply With Quote Quote  

  13. Senior Member joelsfood's Avatar
    Join Date
    Sep 2014
    Location
    Chicago, IL
    Posts
    973

    Certifications
    CCIE:DC, CCNP:DC, CCNA:DC, CCDA, VCP:DCV, VCP:NV, JNCIA-JUNOS
    #12
    SOrry I missed this thread earlier. Yes, VDS is a management device, more than a virtual switch chassis. It allows you to create port groups once and have them configured on all hosts in the VDS, but for uplinks, each host is still addressed separately.
    Reply With Quote Quote  

  14. Senior Member sucanushie's Avatar
    Join Date
    Apr 2013
    Location
    Canada
    Posts
    159

    Certifications
    MCTS, Windows 7, MCITP: EDA Win7,CCNA Security,CCNA Voice, CCNP R&S
    #13
    Yeah, my lack of Vmware knowledge heh.

    Time to maybe tackle VCP:NV maybe haha
    Reply With Quote Quote  

  15. Padawan d4nz1g's Avatar
    Join Date
    May 2013
    Location
    Brazil
    Posts
    424

    Certifications
    CCNP, BCNE, CCNA, CCNA Sec, MCSA2k8, IPv6 Silver
    #14
    Quote Originally Posted by sucanushie View Post
    I was thinking the Distributed Switch acted as one logical switch for all host, but that doesn't seem to be the case.

    Oh boy, I missed this detail haha xD
    2017 - CCIE RS
    Labbing, labbing, labbing.
    Reply With Quote Quote  

+ Reply to Thread

Social Networking & Bookmarks