+ Reply to Thread
Results 1 to 19 of 19

Thread: Help please

  1. Member joshgibson82's Avatar
    Join Date
    Oct 2007
    Location
    NC
    Posts
    80

    Certifications
    CCNA, CCNP, CWNA
    #1

    Default Help please

    I'm coming to this board because I know there is a wide variety of experience in here. I hope someone can help me solve a mystery in my network. I have an 1811 router connected to a metro ethernet connection. It has 3Mbps symmetrical bandwidth. When I get to speedtest.net and do a speed test, the download starts off at around 3 Meg, then quickly drops to maybe 13 Kbps. When I try to download files from certain websites (wireshark.org for example), the same thing happens. The download will start out around 400 KBps, then drop to around 5KBps. I have taken a network trace and do not see any problems. I can post that trace if someone would like to see it. I am using CBAC on the 1811 and suspect there may be a software bug with it. I have taken a laptop directly to the metro ethernet switch and connected, downloaded wireshark and other files very quickly. I am posting my Firewall config:

    ip inspect log drop-pkt
    ip inspect name FW_OUT cuseeme
    ip inspect name FW_OUT dns
    ip inspect name FW_OUT ftp
    ip inspect name FW_OUT https
    ip inspect name FW_OUT icmp
    ip inspect name FW_OUT tftp
    ip inspect name FW_OUT tcp
    ip inspect name FW_OUT udp
    ip inspect name FW_OUT pptp
    ip inspect name FW_OUT http
    ip inspect name FW_OUT h323
    ip inspect name FW_OUT imap
    ip inspect name FW_OUT pop3
    ip inspect name FW_OUT rcmd
    ip inspect name FW_OUT realaudio
    ip inspect name FW_OUT rtsp
    ip inspect name FW_OUT esmtp
    ip inspect name FW_OUT sqlnet
    ip inspect name FW_OUT streamworks
    ip inspect name FW_OUT vdolive
    ip inspect name FW_OUT telnet
    ip inspect name FW_OUT ssh
    ip ips notify SDEE

    interface FastEthernet1
    description Internet Connection
    ip address x.x.x.x 255.255.255.192
    ip access-group FW_IN in
    no ip redirects
    no ip proxy-arp
    ip nat outside
    ip inspect FW_OUT out
    ip virtual-reassembly
    ip tcp adjust-mss 1460
    speed 100
    full-duplex
    no cdp enable
    crypto map TO-JOSH

    (the metro E switch is manually set to 100/full duplex and has an MTU of 1518 )

    interface Vlan1
    description $FW_INSIDE$
    ip address 192.168.10.2 255.255.255.0
    no ip redirects
    no ip proxy-arp
    ip nat inside
    ip virtual-reassembly
    ip tcp adjust-mss 1452

    ip nat inside source route-map NAT_POOL interface FastEthernet1 overload

    access-list 100 permit ip 192.168.10.0 0.0.0.255 any
    route-map NAT_POOL permit 1
    match ip address 100

    ip access-list extended FW_IN
    permit icmp any host <F1 ip address>
    deny ip any any log



    TOPOLOGY:

    (192.168.10.0 /24 segment is VLAN 1) ----> 1811 interface F1 for WAN ------------> Metro Ethernet switch --------------> Internet

    Again, most all internet sessions work fine, except when I try to do large downloads it seems. I have played with the tcp adjust-mss size and also the mtu on the F1 interface but nothing has made any difference. PLEASE HELP!!!!!!!!!!! It's driving me crazy!
    Last edited by joshgibson82; 11-19-2009 at 12:09 AM.
    Reply With Quote Quote  

  2. SS -->
  3. Member joshgibson82's Avatar
    Join Date
    Oct 2007
    Location
    NC
    Posts
    80

    Certifications
    CCNA, CCNP, CWNA
    #2

    Default one more note

    I'd be more than happy to run any debug commands someone would want to help troubleshoot this. I'm just out of ideas at this point.
    Reply With Quote Quote  

  4. APA
    APA is offline
    Senior Member APA's Avatar
    Join Date
    Jun 2006
    Location
    Sydney, Australia
    Posts
    956

    Certifications
    CompTIA, Microsoft, Juniper & Cisco (Check Signature)
    #3
    output from the following

    1811
    sh int fa 1

    Metro Switch
    sh int xxx

    Also why are you adjusting mss for??? I don't see any PPP connections or a need to adjust??? (Mind you I can only assume from what you have pasted)

    To me it sounds like the TCP window sizing is being cut down....hence the fast start and gradual decrease....

    As a test can you remove any mss modifications and try the download...
    Reply With Quote Quote  

  5. Member joshgibson82's Avatar
    Join Date
    Oct 2007
    Location
    NC
    Posts
    80

    Certifications
    CCNA, CCNP, CWNA
    #4
    I have completely removed all the adjust MSS statements and no change, so I put them back. I do not have access to the Metro E switch as it belongs to the ISP.
    Reply With Quote Quote  

  6. Member joshgibson82's Avatar
    Join Date
    Oct 2007
    Location
    NC
    Posts
    80

    Certifications
    CCNA, CCNP, CWNA
    #5
    I am getting this:




    *Nov 19 11:02:11.402: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948
    *Nov 19 11:02:41.418: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948

    Also, I have removed all the adjust mss statements. all default now.
    Reply With Quote Quote  

  7. Member joshgibson82's Avatar
    Join Date
    Oct 2007
    Location
    NC
    Posts
    80

    Certifications
    CCNA, CCNP, CWNA
    #6
    Quote Originally Posted by APA View Post
    output from the following

    1811
    sh int fa 1

    .

    #sh int f1
    FastEthernet1 is up, line protocol is up
    Hardware is PQ3_TSEC, address is 0016.47e9.0dfd (bia 0016.47e9.0dfd)
    Description: Internet Connection
    Internet address is x.x.x.x/26
    MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
    reliability 255/255, txload 1/255, rxload 1/255
    Encapsulation ARPA, loopback not set
    Keepalive set (10 sec)
    Full-duplex, 100Mb/s, 100BaseTX/FX
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input 00:00:00, output 00:00:00, output hang never
    Last clearing of "show interface" counters never
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 37
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 216000 bits/sec, 28 packets/sec
    5 minute output rate 72000 bits/sec, 22 packets/sec
    1468746 packets input, 900256088 bytes
    Received 43018 broadcasts, 0 runts, 0 giants, 0 throttles
    1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    0 watchdog
    0 input packets with dribble condition detected
    1588942 packets output, 711440820 bytes, 0 underruns
    0 output errors, 0 collisions, 3 interface resets
    0 babbles, 0 late collision, 0 deferred
    0 lost carrier, 0 no carrier
    0 output buffer failures, 0 output buffers swapped out
    Reply With Quote Quote  

  8. Senior Member jovan88's Avatar
    Join Date
    May 2008
    Location
    Sydney, Australia
    Posts
    388

    Certifications
    CCNP R&S, CCNP Sec
    #7
    which IOS are you using
    Reply With Quote Quote  

  9. Senior Member
    Join Date
    Apr 2008
    Location
    Tampa, Fl
    Posts
    1,097

    Certifications
    A Few....
    #8
    Is this happening on all clients? try adjusting your MTU size on your client... (unless its all clients having this problem).

    HTH.
    Reply With Quote Quote  

  10. APA
    APA is offline
    Senior Member APA's Avatar
    Join Date
    Jun 2006
    Location
    Sydney, Australia
    Posts
    956

    Certifications
    CompTIA, Microsoft, Juniper & Cisco (Check Signature)
    #9
    sh ip inspect session (first start a legit flow from a client on the 192.168.x.x network)

    There are numerous factors that could be causing the issues you are experiencing... so we would have to go through them one by one...

    What clients are you testing from? Windows? Linux?
    Reply With Quote Quote  

  11. Member joshgibson82's Avatar
    Join Date
    Oct 2007
    Location
    NC
    Posts
    80

    Certifications
    CCNA, CCNP, CWNA
    #10
    Yes, this is happening on all clients including windows xp pro, vista, and Opensuse 10.3. I will have to wait until I get off work to do any show commands, but I know the IOS is in the 12.4 train. I will get the exact one later. I was thinking the same thing (could be IOS bug).
    Reply With Quote Quote  

  12. Senior Member SysAdmin4066's Avatar
    Join Date
    Feb 2009
    Location
    California
    Posts
    443

    Certifications
    CCNP, CCNA, MCITP EA, MCSE, MCSA, multiple MCTS, MCP, CISSP, CTP
    #11
    Have you tried removing your ip inspect statements? Just to see if it is in fact the ip inspect. You said that when you connect directly to the metro E, your download speeds are unaffected. Try by first removing all of the IP Inspect statements. Then if the problem goes away, try to add each one individually back and test each statement. What I would do is take the router back to completely open, no config besides whats absolutely necessary for routing. Then add the security one by one and test.
    Reply With Quote Quote  

  13. Went to the dark side.... Moderator networker050184's Avatar
    Join Date
    Jul 2007
    Posts
    11,649

    Certifications
    CCNA, CCNP, CCIP, JNCIA-JUNOS, JNCIS-SP, JNCIP-SP, MCA200
    #12
    Quote Originally Posted by SysAdmin4066 View Post
    Have you tried removing your ip inspect statements? Just to see if it is in fact the ip inspect. You said that when you connect directly to the metro E, your download speeds are unaffected. Try by first removing all of the IP Inspect statements. Then if the problem goes away, try to add each one individually back and test each statement. What I would do is take the router back to completely open, no config besides whats absolutely necessary for routing. Then add the security one by one and test.

    That is what I would try also. You have to narrow down the issue as much as possible.
    An expert is a man who has made all the mistakes which can be made.
    Reply With Quote Quote  

  14. Member joshgibson82's Avatar
    Join Date
    Oct 2007
    Location
    NC
    Posts
    80

    Certifications
    CCNA, CCNP, CWNA
    #13
    c181x-adventerprisek9-mz.124-6.T2.bin is the IOS i'm running and also, when I removed the firewall from the interface, and the inbound ACL, the download from wireshark went extremely fast. Process of elimination now. Thanks for the idea guys!
    Reply With Quote Quote  

  15. Member joshgibson82's Avatar
    Join Date
    Oct 2007
    Location
    NC
    Posts
    80

    Certifications
    CCNA, CCNP, CWNA
    #14
    UPgraded code to 12-4-15.T11 and configured this inspect set:


    ip inspect log drop-pkt
    ip inspect name FW_OUT http
    ip inspect name FW_OUT https
    ip inspect name FW_OUT ssh
    ip inspect name FW_OUT ftp
    ip inspect name FW_OUT echo
    ip inspect name FW_OUT tcp
    ip inspect name FW_OUT udp
    ip inspect name FW_OUT icmp
    ip inspect name FW_OUT smtp
    ip inspect name FW_OUT dns

    And got this output when downloading wireshark:
    *Nov 20 00:14:29.622: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:193142182 1500 bytes is out-of-order; expected seq:193111774. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:29.622: %FW-6-DROP_PKT: Dropping http session 69.89.22.118:80 192.168.10.10:34029 due to Out-Of-Order Segment with ip ident 40914 tcpflags 0x8010 seq.no 193142182 ack 830037319
    *Nov 20 00:14:29.666: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:14:35.286: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:194323750 1500 bytes is out-of-order; expected seq:194297686. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:40.530: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:195725414 1500 bytes is out-of-order; expected seq:195689214. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:41.666: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:14:41.686: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:195768854 1500 bytes is out-of-order; expected seq:195693558. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:42.834: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:195777542 1500 bytes is out-of-order; expected seq:195697902. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:48.118: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:196847614 1500 bytes is out-of-order; expected seq:196809966. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:49.298: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:196885262 1500 bytes is out-of-order; expected seq:196814310. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:55.854: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:198145022 1500 bytes is out-of-order; expected seq:198108822. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:57.694: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:15:02.663: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:199988326 1500 bytes is out-of-order; expected seq:199963710. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:02.667: %FW-6-DROP_PKT: Dropping http session 69.89.22.118:80 192.168.10.10:34029 due to Out-Of-Order Segment with ip ident 45829 tcpflags 0x8010 seq.no 199988326 ack 830037319
    *Nov 20 00:15:04.607: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:200686262 1500 bytes is out-of-order; expected seq:200645718. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:05.771: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:200719566 1500 bytes is out-of-order; expected seq:200650062. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:07.695: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:15:12.059: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:201985118 1500 bytes is out-of-order; expected seq:201940230. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:13.435: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:202583142 1500 bytes is out-of-order; expected seq:202549838. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:18.859: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TCP Timer", ipl= 4, pid= 111, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:15:18.907: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:204000734 1500 bytes is out-of-order; expected seq:203958742. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:25.539: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:205941054 1500 bytes is out-of-order; expected seq:205907750. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:30.931: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:207209502 1500 bytes is out-of-order; expected seq:207183438. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:32.695: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:15:36.667: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:208628542 1500 bytes is out-of-order; expected seq:208593790. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:36.667: %FW-6-DROP_PKT: Dropping http session 69.89.22.118:80 192.168.10.10:34029 due to Out-Of-Order Segment with ip ident 52060 tcpflags 0x8010 seq.no 208628542 ack 830037319
    *Nov 20 00:15:38.383: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:209287382 1500 bytes is out-of-order; expected seq:209248286. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:40.391: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:210202518 1500 bytes is out-of-order; expected seq:210176454. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:45.963: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:211649070 1500 bytes is out-of-order; expected seq:211609974. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:47.695: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8

    This version of code definitely provides more detail but I don't really know what it means.
    Reply With Quote Quote  

  16. Member joshgibson82's Avatar
    Join Date
    Oct 2007
    Location
    NC
    Posts
    80

    Certifications
    CCNA, CCNP, CWNA
    #15
    *Nov 20 00:29:30.905: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:4261185239 1500 bytes is out-of-order; expected seq:4261149039. Reason: TCP reassembly queue overflow - session 192.168.10.10:45419 to 69.89.22.118:80



    And these!
    Reply With Quote Quote  

  17. Member joshgibson82's Avatar
    Join Date
    Oct 2007
    Location
    NC
    Posts
    80

    Certifications
    CCNA, CCNP, CWNA
    #16
    So for anyone who cares out there..... c181x-adventerprisek9-mz.124-24.T2.bin code seems to have fixed all issues. Time will tell if the fix is permanent.
    Reply With Quote Quote  

  18. Senior Member hypnotoad's Avatar
    Join Date
    Dec 2007
    Posts
    915

    Certifications
    BS&MS-CompSci, CCNA, CCNP, Hyper-V, CCAI
    #17
    Quote Originally Posted by joshgibson82 View Post
    I am getting this:




    *Nov 19 11:02:11.402: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948
    *Nov 19 11:02:41.418: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948

    Also, I have removed all the adjust mss statements. all default now.
    I get this all the time too -- no idea what runs on 34948 but it comes in 24/7 from all over.
    Reply With Quote Quote  

  19. Senior Member
    Join Date
    May 2009
    Posts
    111

    Certifications
    CCNP, CCNA Voice
    #18
    Try

    no ip inspect tcp reasembly queue length

    maybe?
    Reply With Quote Quote  

  20. Senior Member cyberguypr's Avatar
    Join Date
    May 2007
    Location
    Chicago, IL
    Posts
    5,779

    Certifications
    GCFE, GCED, GCIH, CISSP, CCSP, and others that should never be mentioned
    #19
    Considering it's been a year and a half since the OP posted I think either he solved it or got fired.
    Reply With Quote Quote  

+ Reply to Thread

Social Networking & Bookmarks