+ Reply to Thread
Page 1 of 2 1 2 Last
Results 1 to 25 of 27
  1. Senior Member Dr_Atomic's Avatar
    Join Date
    Mar 2009
    Posts
    184
    #1

    Default Solving a network/ethernet loop

    If you're experiencing an ethernet loop on a switch(es), what's the troubleshooting process for this? I remember this in my CCNA studies, but it doesn't come to me. I'm researching it, also, in the meantime. Thanks.
    Reply With Quote Quote  

  2. SS -->
  3. Senior Member /usr's Avatar
    Join Date
    Dec 2003
    Location
    West Virginia
    Posts
    1,776
    #2
    I suppose the first troubleshooting step to verify a loop would be traffic capturing. You could make a safe assumption based on which segments are experiencing throughput issues, based on user feedback or abnormal activity LED's, but the only sure way that I know of is to capture duplicate frames.

    The first plan of action is to break the loop, or manually disable all redundant network paths, then re-enable them one at a time, hopefully finding the cause of the loop in the process.

    Other people might chime in with other, more valuable input.
    Reply With Quote Quote  

  4. was here.
    Join Date
    Apr 2008
    Posts
    4,504
    #3
    Why don't you have STP enabled? :P
    Reply With Quote Quote  

  5. Senior Member chrisone's Avatar
    Join Date
    Nov 2009
    Location
    Los Angeles
    Posts
    1,569

    Certifications
    SilentBreakSecurity - DarkSideOps, CISSP, CCDP, CCNP R/S, CCNP Security (Secure, FW) , C|EH , PA ACE
    #4
    Although there are many methods, which i will admit i dont know them all, i would suggest checking the spanning tree, roots, and see if you see any ports flapping stp states. Start narrowing it down from there. Also check the MAC address tables, you probably shouldnt be seeing a bunch of MACs or a root port on a port where a computer/host should be connected to.

    Rule of thumb is:

    Use RSTP
    Use PORTFAST (only on hosts, non uplinks)
    Configure Globally BPDU Gaurd
    Configure Globally Root Gaurd

    I cant think of anything else at the moment. Maybe someone can chime in on some tips.
    2017 Goals: Dark Side OPS: Custom Pentesting (complete), eCPPT (in progress), LFCS (in progress), OSCP
    Reply With Quote Quote  

  6. Senior Member Netwurk's Avatar
    Join Date
    May 2005
    Location
    Philadelphia, PA
    Posts
    1,155

    Certifications
    CCNA, Network+, A+ (CCNP Progress: 1/3)
    #5
    I once caused a huge loop by bringing down a port channel without first doing a shut on all the ports. Luckily I did this at home and not work.

    Every light on every port in my lab started blinking like crazy. Luckily I knew what caused it and did a quick no shut on the ports that were formerly part of the channel.

    So for troubleshooting a loop, let's put it this way - if you've got one, you'll know it.
    Reply With Quote Quote  

  7. Senior Member
    Join Date
    Aug 2008
    Posts
    3,951
    #6
    If a loop develops, you'll know it pretty quick, your network will come to a screeching halt. Depending on the switches involved, you may no longer be able to get access to them, even from the console, because their CPU is probably pegged. I've only seen a loop in production once, and when I found out my switches were unaccessible from remote, I checked the syslog server to see which interface came up last, and had someone on site pull the cable from that port.
    Reply With Quote Quote  

  8. Member wolverene13's Avatar
    Join Date
    Jul 2008
    Location
    Maitland, Florida
    Posts
    86

    Certifications
    Network+, CCNA, CCNP
    #7

    Default Depends

    Quote Originally Posted by tiersten View Post
    Why don't you have STP enabled? :P
    I get this question all the time at work when customers cause broadcast storms on our Ethernet network. I'm not sure about the original poster's situation, but in our network, we have a Cisco 3400-ME switch at each customer's site so that we have end-to-end visibility to the customer's circuit. The customer plugs their equipment into the dot1q-tunnel ports on the ME-3400. We use QinQ tunneling, which essentially wraps the customer's VLAN inside our VLAN that we've assigned to that customer and we are transparent to the customer and the customer's traffic is invisible to us, aside from the bandwidth it eats up. So, whatever they put inside the tunnel is of no concern to us. However, there is a downside to all this. While we do run STP, the STP instance only applies to our network, not the customers'. So, if the customer creates a loop on their network and the traffic leaves that site and traverses the Ethernet cloud to another one of their sites, our STP doesn't see it because it doesn't have an instance for the customer's internal VLAN, being that we don't even know it exists. This manifests itself on our network in the form of overutilized trunks. As a result, trunks start bouncing and all our other customers in that particular area of that particular state are affected because our STP has to keep reconverging. So, our STP works, but that does no good to stop loops on our customers' networks. Think of it like a swarm of angry bees inside a tree trunk. There's a lot of craziness going on inside the tree, but you can't see it because you're on the outside. Normally in this situation, we try to find a point of commonality between all of the customers who call in when this is happening and correlate it with the network alarms we get through Netcool (our network monitoring system). Starting from that point, we find out which trunks are being killed with traffic, and follow that traffic until we get to the Cisco 7609 that has the offending customer on it. If we notice a customer with with a 1-Gig circuit is using 996 Megs or something, we shut the port down to restore service and advise the customer that they need to fix the issue before we turn them back up.
    Reply With Quote Quote  

  9. Senior Member
    Join Date
    Apr 2008
    Location
    Tampa, Fl
    Posts
    1,097

    Certifications
    A Few....
    #8
    Quote Originally Posted by Dr_Atomic View Post
    If you're experiencing an ethernet loop on a switch(es), what's the troubleshooting process for this? I remember this in my CCNA studies, but it doesn't come to me. I'm researching it, also, in the meantime. Thanks.
    You should disable spanning-tree on your switches, plug them up with two crossover cables, plug in a PC (to generate some broadcast packets) and have fun... you can also use a tool like yersinia to generate some STP attack packets that will essentially have the same effect on the switches (slowing them down to a crawl). That's what a home lab is for... happy "geeking"! HTH.

    -Peanut
    Reply With Quote Quote  

  10. Senior Member
    Join Date
    Sep 2008
    Location
    Sweden
    Posts
    392

    Certifications
    CCNP R&S, MCSE 2000, MCTS-640, CCNA:Sec
    #9
    Quote Originally Posted by peanutnoggin View Post
    You should disable spanning-tree on your switches, plug them up with two crossover cables, plug in a PC (to generate some broadcast packets) and have fun... you can also use a tool like yersinia to generate some STP attack packets that will essentially have the same effect on the switches (slowing them down to a crawl). That's what a home lab is for... happy "geeking"! HTH.

    -Peanut
    Hah, I did this with a couple cisco switches. If I turned down storm control to about 2Mbit rising and 1 sinking(?), They could just handle the load before croaking. I ended up using half those values, though. (Residential/college broadband) BPDU-filter on access ports.
    Reply With Quote Quote  

  11. Senior Member Netwurk's Avatar
    Join Date
    May 2005
    Location
    Philadelphia, PA
    Posts
    1,155

    Certifications
    CCNA, Network+, A+ (CCNP Progress: 1/3)
    #10
    I ran a layer 2 MAC flooding attack from a linux box on several of my switches while I was labbing BCMSN. It's a good way to see how necessary port security is. With no security, the only unsecured switch that could keep going against the attack was my old CatOS 2926. The reason I think was its relatively huge mac address table. It just refused to go down despite looping endless flood commands its way.

    My 3550's, 2950's, 3500's, and 2900's were dead in the water in less than a minute.

    I was going to name the tool I used but some idiot would then download it and become an instant hacker.

    Reply With Quote Quote  

  12. Senior Member Dr_Atomic's Avatar
    Join Date
    Mar 2009
    Posts
    184
    #11
    I gotten a lot of good responses, but so far it's all pretty much theory. Could someone give me some commands I could input and check to see what I should/shouldn't see from them? Like a step-by-step check of things to look for? What would be some sample commands to use to check for loop issues?
    Reply With Quote Quote  

  13. Senior Member chrisone's Avatar
    Join Date
    Nov 2009
    Location
    Los Angeles
    Posts
    1,569

    Certifications
    SilentBreakSecurity - DarkSideOps, CISSP, CCDP, CCNP R/S, CCNP Security (Secure, FW) , C|EH , PA ACE
    #12
    read my post, i gave you some tips on how to help prevent loops. Its your job to get the commands and read up exactly what they do. They are simple and self explanatory if you understand Spanning Tree. If you have access to the CLI , look at the STP states.
    2017 Goals: Dark Side OPS: Custom Pentesting (complete), eCPPT (in progress), LFCS (in progress), OSCP
    Reply With Quote Quote  

  14. Senior Member
    Join Date
    Apr 2008
    Location
    Tampa, Fl
    Posts
    1,097

    Certifications
    A Few....
    #13
    Quote Originally Posted by Dr_Atomic View Post
    I gotten a lot of good responses, but so far it's all pretty much theory. Could someone give me some commands I could input and check to see what I should/shouldn't see from them? Like a step-by-step check of things to look for? What would be some sample commands to use to check for loop issues?
    Dr. Atomic,

    You're right... everyone is giving you theory and as Chrisone said... it's up to you to research and see how/when to use tools. That's a part of the learning curve. What you have to realize is that when someone gives you the step by step instructions... all you're going to typically learn is what they show you. Be adventurous... if you have a home lab, backup your configs (if you want to preserve them) and then start playing around. Change some of the modes of spanning-tree, change some of the port cost, priority values, etc... disable spanning-tree on links... as you go through these different configurations, document what you do and what you find... you'd be quite surprised with the amount of information you'll learn. HTH.

    -Peanut
    Reply With Quote Quote  

  15. Senior Member Netwurk's Avatar
    Join Date
    May 2005
    Location
    Philadelphia, PA
    Posts
    1,155

    Certifications
    CCNA, Network+, A+ (CCNP Progress: 1/3)
    #14
    Quote Originally Posted by Dr_Atomic View Post
    I gotten a lot of good responses, but so far it's all pretty much theory. Could someone give me some commands I could input and check to see what I should/shouldn't see from them? Like a step-by-step check of things to look for? What would be some sample commands to use to check for loop issues?
    When a loop occurs, you have limited time to track it down. After a few minutes, you might not even be able to get to the console on your devices. Set up a syslog server so that you can troubleshoot from there if all the network equipment gets pegged.

    Syslog is very easy to configure. Just get syslog running on a server and use the global IOS command logging x.x.x.x on all your devices.
    Reply With Quote Quote  

  16. Senior Member
    Join Date
    Aug 2008
    Posts
    3,951
    #15
    Quote Originally Posted by Netwurk View Post
    When a loop occurs, you have limited time to track it down. After a few minutes, you might not even be able to get to the console on your devices. Set up a syslog server so that you can troubleshoot from there if all the network equipment gets pegged.

    Syslog is very easy to configure. Just get syslog running on a server and use the global IOS command logging x.x.x.x on all your devices.
    Yup, like I said above, the one broadcast storm I've seen in production, by the time I was made aware of it, my switches weren't accessible due to CPU usage, even from the console. I had someone on site go check the syslog server and read me the last few interface events before the crap hit the fan. It was the quickest way to narrow down the problem.
    Reply With Quote Quote  

  17. Senior Member Dr_Atomic's Avatar
    Join Date
    Mar 2009
    Posts
    184
    #16
    I"m checking a production network, so I can't experiment with a server at the moment. I've input every conceivable command I can think of to check this problem. If someone could deign to provide some commands and what to look for, it would be nice.

    In other words, how would one know from being logged into a switch if there *was* a loop present causing a problem? From what command would one see the problem?
    Last edited by Dr_Atomic; 09-30-2010 at 07:25 PM.
    Reply With Quote Quote  

  18. Cisco Moderator mikej412's Avatar
    Join Date
    May 2005
    Location
    Chicago
    Posts
    10,190

    Certifications
    CCNP CCIP CCSP CCVP CCDP CCDA CCNA CS-CIPSS CS-CIPTDS CS-CIPTOS CS-CIPCSS CS-CFWS CS-CVPNS CS-CISecS ISSP 4013 4011
    #17
    Quote Originally Posted by Dr_Atomic View Post
    I"m checking a production network, so I can't experiment with a server at the moment. I've input every conceivable command I can think of to check this problem.
    Um.... what commands were those?

    Do you have a network diagram that accurately lists the redundant links? Shutdown the known redundant links.

    Find any incorrect redundant links created by an idiot randomly plugging in network cables in a wiring closet and misconfiguring switch ports using the show cdp neighbor command.

    If it's been a day and you can still log into the switches then you either don't have a loop -- or you have some nice switches and storm control enabled or maybe a loop limited to just one VLAN (or two). What's the hardware? What's the topology? Did you run the show spanning-tree command (and use any of the options)? What version(s) of spanning tree are you running?

    How are you logging? Syslog? Local Logs? Is logging to the console turned off?
    Reply With Quote Quote  

  19. Senior Member Dr_Atomic's Avatar
    Join Date
    Mar 2009
    Posts
    184
    #18
    Quote Originally Posted by mikej412 View Post
    Do you have a network diagram that accurately lists the redundant links? Shutdown the known redundant links.

    Find any incorrect redundant links created by an idiot randomly plugging in network cables in a wiring closet and misconfiguring switch ports using the show cdp neighbor command.
    So if I *do* have a redundant link, I could do a sh cdp neighbor and it would show it there? Then I could just disconnect that link to see if it helps the issue?
    Reply With Quote Quote  

  20. Went to the dark side.... Moderator networker050184's Avatar
    Join Date
    Jul 2007
    Posts
    11,646

    Certifications
    CCNA, CCNP, CCIP, JNCIA-JUNOS, JNCIS-SP, JNCIP-SP, MCA200
    #19
    No offense, but it sounds like you are in WAY over your head here if you don't even know how to find redundant links. Is there not a more knowledgeable staff member you can talk to? You are probably going to make things worse if you are just winging it man.
    An expert is a man who has made all the mistakes which can be made.
    Reply With Quote Quote  

  21. Senior Member Dr_Atomic's Avatar
    Join Date
    Mar 2009
    Posts
    184
    #20
    Quote Originally Posted by networker050184 View Post
    No offense, but it sounds like you are in WAY over your head here if you don't even know how to find redundant links. Is there not a more knowledgeable staff member you can talk to? You are probably going to make things worse if you are just winging it man.
    It's just me, pal. I've been thrown to the wolves on this one.
    Reply With Quote Quote  

  22. Senior Member
    Join Date
    Sep 2008
    Location
    Sweden
    Posts
    392

    Certifications
    CCNP R&S, MCSE 2000, MCTS-640, CCNA:Sec
    #21
    Quote Originally Posted by Netwurk View Post
    I ran a layer 2 MAC flooding attack from a linux box on several of my switches while I was labbing BCMSN. It's a good way to see how necessary port security is. With no security, the only unsecured switch that could keep going against the attack was my old CatOS 2926. The reason I think was its relatively huge mac address table. It just refused to go down despite looping endless flood commands its way.

    My 3550's, 2950's, 3500's, and 2900's were dead in the water in less than a minute.

    I was going to name the tool I used but some idiot would then download it and become an instant hacker.

    Well I did have port security set, so mac overflow wasn't an issue, still CPU util went up like crazy.

    Also, macof! There, I dun did it!

    Information wants to be free goddammit
    Reply With Quote Quote  

  23. Cisco Moderator mikej412's Avatar
    Join Date
    May 2005
    Location
    Chicago
    Posts
    10,190

    Certifications
    CCNP CCIP CCSP CCVP CCDP CCDA CCNA CS-CIPSS CS-CIPTDS CS-CIPTOS CS-CIPCSS CS-CFWS CS-CVPNS CS-CISecS ISSP 4013 4011
    #22
    Quote Originally Posted by Dr_Atomic View Post
    So if I *do* have a redundant link, I could do a sh cdp neighbor and it would show it there? Then I could just disconnect that link to see if it helps the issue?
    Exactly what is the issue. If you did have a loop and misconfigured/disabled STP you probably wouldn't be able to access the switches.

    If you don't have a (current) network diagram, then you should be able to map the Cisco equipment (and links) using the show cdp command. Of course you'd also check the configurations first to see if CDP had been disabled anywhere before you waste time drawing an incomplete network map.

    But if you did have a good network diagram, you'd probably want to first look for redundant links that shouldn't be there.
    Reply With Quote Quote  

  24. Senior Member Netwurk's Avatar
    Join Date
    May 2005
    Location
    Philadelphia, PA
    Posts
    1,155

    Certifications
    CCNA, Network+, A+ (CCNP Progress: 1/3)
    #23
    Quote Originally Posted by creamy_stew View Post
    Also, macof! There, I dun did it!

    Information wants to be free
    My macof experiment post was really off-topic for this thread, but at the time I mistakenly thought the loop issue was solved. Oops.

    Although I'm not really sure if our buddy even has a loop issue, but who know?

    Oliver, how big a network are we talking about? What is making you think you have a loop?

    Found you a Cisco page with loop advice, the section "Troubleshooting Forwarding Loops" should be helpful.

    Troubleshooting STP on Catalyst Switches Running Cisco IOS System Software - Cisco Systems
    Reply With Quote Quote  

  25. Cisco Moderator mikej412's Avatar
    Join Date
    May 2005
    Location
    Chicago
    Posts
    10,190

    Certifications
    CCNP CCIP CCSP CCVP CCDP CCDA CCNA CS-CIPSS CS-CIPTDS CS-CIPTOS CS-CIPCSS CS-CFWS CS-CVPNS CS-CISecS ISSP 4013 4011
    #24
    Quote Originally Posted by Netwurk View Post
    What is making you think you have a loop?
    Or has he already moved on to a different thread/problem theory and not told us here?

    From this thread: Bouncing ports

    Quote Originally Posted by Dr_Atomic View Post
    I was told there might be a network loop

    show interface summary and look for anything out of the ordinary (assuming you know what would be ordinary)

    show interface counters (ditto)
    Reply With Quote Quote  

  26. Member wolverene13's Avatar
    Join Date
    Jul 2008
    Location
    Maitland, Florida
    Posts
    86

    Certifications
    Network+, CCNA, CCNP
    #25

    Default Here

    Quote Originally Posted by Dr_Atomic View Post
    I"m checking a production network, so I can't experiment with a server at the moment. I've input every conceivable command I can think of to check this problem. If someone could deign to provide some commands and what to look for, it would be nice.

    In other words, how would one know from being logged into a switch if there *was* a loop present causing a problem? From what command would one see the problem?
    "show interfaces" is all you really need. Then you look for ports that are maxed out. Those ports are where the traffic caused by the loop is. If you see maxed out input traffic on a trunk (meaning the loop traffic is coming into that device from somewhere else), go to the device on the other end of the trunk and issue a "show interfaces" command on that device. Keep doing this until you reach a device that only has maxed out output traffic on the trunks. This means that the culprit is directly connected to the device you are currently logged into and the loop traffic is originating on the device you are currently logged into, so you then check traffic on access ports. Once you find a maxed out access port, you know that the device or host connected to it is what is causing the loop. "show log" will also help in this scenario. I lot of times you'll see MAC flapping messages in the logs on the device where the loop is occurring because the switch is seeing the same MAC on two different ports.
    Reply With Quote Quote  

+ Reply to Thread
Page 1 of 2 1 2 Last

Social Networking & Bookmarks