ASA5520 to 5505 L2L VPN

[Netstudent]

I have an ipsec l2l vpn between a 5520 and a 5505 running 804. The tunnel drops everytime ISAKMP rekeys. The configs are correct, and I have already gotten TAC and several engineers to review the config.

Basically what happens is on the far-end 5505 ISAKMP tears down the tunnel about 300 seconds before the actual timer expires. Timer is set to 3600s for isakmp and ipsec.

When this happens, the ISAKMP SA on the 5505 shows this:

IKE Peer: x.x.x.x
    Type    : user            Role    : initiator 
    Rekey   : no              State   : MM_WAIT_MSG2?
    Encrypt : aes-256 ?        Hash    : SHA       
    Auth    : preshared       Lifetime: 0 ?

5505# "show cry ipsec sa"

There are no ipsec sas

The 5520 at the head end shows this:

1   IKE Peer: x.x.x.x
    Type    : L2L             Role    : initiator
    Rekey   : no              State   : MM_ACTIVE
    Encrypt : 3des            Hash    : SHA
    Auth    : preshared       Lifetime: 3600
    Lifetime Remaining: 339

So the headend is still counting down and the far-end has already tore down the tunnel.

ALso I am running captures on UDP:500 on the outside interface with the peers specified in the ACL on both firewalls. The 5505 sends and sends but the 5520 does not reply.

On the 5505 you can see the ISAKMP packets going out and the 5520 not replying (2.2.2.2 would be the headend):

   61: 12:46:58.337827 1.1.1.1.500 > 2.2.2.2.500:  udp 108
  62: 12:47:12.422265 1.1.1.1.500 > 2.2.2.2.500:  udp 68
  63: 12:47:12.426094 1.1.1.1.500 > 2.2.2.2.500:  udp 84

This goes on for about 5 minutes OR untill I clear the crypto isakmp sa on the headend. When I clear the crytpo or if I wait 5-6 minutes then I will see bidirectional packets coming and going.

It may also be worth nothing that we just upgraded to 804 and the problem started to occur around the same time as the upgrade, but nobody has confirmed that this is a code issue.

WHen I run debugs on the 5505 I see that ISAKMP tries to start a rekey but does not get a reply from the 5520.

<167>Oct 03 2008 14:49:40: %ASA-7-713906: IP = 2.2.2.2, Starting phase 1 rekey

<165>Oct 03 2008 14:49:40: %ASA-5-713041: IP = 2.2.2.2, IKE Initiator: Rekeying Phase 1, Intf outside, IKE Peer 2.2.2.2 local Proxy Address N/A, remote Proxy Address N/A,  Crypto map (N/A)

<167>Oct 03 2008 14:49:40: %ASA-7-715046: IP = 2.2.2.2, constructing ISAKMP SA payload

<167>Oct 03 2008 14:49:40: %ASA-7-715046: IP = 2.2.2.2, constructing Fragmentation VID + extended capabilities payload

<167>Oct 03 2008 14:49:40: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108

<167>Oct 03 2008 14:49:48: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108

<167>Oct 03 2008 14:49:56: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108

<167>Oct 03 2008 14:50:04: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108

<167>Oct 03 2008 14:50:12: %ASA-7-715065: IP = 2.2.2.2, IKE MM Initiator FSM error history (struct &0xd7910528)  <state>, <event>:  MM_DONE, EV_ERROR-->MM_WAIT_MSG2, EV_RETRY-->MM_WAIT_MSG2, EV_TIMEOUT-->MM_WAIT_MSG2, NullEvent-->MM_SND_MSG1, EV_SND_
MSG-->MM_SND_MSG1, EV_START_TMR-->MM_SND_MSG1, EV_RESEND_MSG-->MM_WAIT_MSG2, EV_RETRY

<167>Oct 03 2008 14:50:12: %ASA-7-713906: IP = 2.2.2.2, IKE SA MM:b3fdfbb2 terminating:  flags 0x01000422, refcnt 0, tuncnt 0

<167>Oct 03 2008 14:50:12: %ASA-7-713906: IP = 2.2.2.2, sending delete/delete with reason message

<163>Oct 03 2008 14:50:12: %ASA-3-713902: IP = 2.2.2.2, Removing peer from peer table failed, no match!

<164>Oct 03 2008 14:50:12: %ASA-4-713903: IP = 2.2.2.2, Error: Unable to remove PeerTblEntry

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Group = 2.2.2.2, IP = 2.2.2.2, Expired IKE SA with 1 phase 2 centries still associated

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Group = 2.2.2.2, IP = 2.2.2.2, sending delete/delete with reason message

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing blank hash payload

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing IPSec delete payload

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing qm hash payload

<167>Oct 03 2008 14:50:17: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE SENDING Message (msgid=170945d) with payloads : HDR + HASH (8) + DELETE (12) + NONE (0) total length : 68

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Group = 2.2.2.2, IP = 2.2.2.2, Active unit receives a delete event for remote peer 2.2.2.2.



<167>Oct 03 2008 14:50:17: %ASA-7-715009: Group = 2.2.2.2, IP = 2.2.2.2, IKE Deleting SA: Remote Proxy 0.0.0.0, Local Proxy 10.10.0.0

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Group = 2.2.2.2, IP = 2.2.2.2, IKE SA MM:63bba120 terminating:  flags 0x0120c002, refcnt 0, tuncnt 0

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Group = 2.2.2.2, IP = 2.2.2.2, sending delete/delete with reason message

<166>Oct 03 2008 14:50:17: %ASA-6-602304: IPSEC: An inbound LAN-to-LAN SA (SPI= 0x78B07B12) between 1.1.1.1 and 2.2.2.2(user= 2.2.2.2) has been deleted.

<166>Oct 03 2008 14:50:17: %ASA-6-602304: IPSEC: An outbound LAN-to-LAN SA (SPI= 0xB4FA3DE6) between 1.1.1.1 and 2,2,2,2 (user= 2.2.2.2) has been deleted.

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing blank hash payload

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing IKE delete payload

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing qm hash payload

<167>Oct 03 2008 14:50:17: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE SENDING Message (msgid=1a400590) with payloads : HDR + HASH (8) + DELETE (12) + NONE (0) total length : 80

<167>Oct 03 2008 14:50:17: %ASA-7-715077: Pitcher: received key delete msg, spi 0x78b07b12

<167>Oct 03 2008 14:50:17: %ASA-7-715077: Pitcher: received key delete msg, spi 0x78b07b12

<167>Oct 03 2008 14:50:17: %ASA-7-715077: Pitcher: received key delete msg, spi 0xb4fa3de6

<163>Oct 03 2008 14:50:17: %ASA-3-713902: Group = 2.2.2.2, IP = 2.2.2.2, Removing peer from peer table failed, no match!

<164>Oct 03 2008 14:50:17: %ASA-4-713903: Group = 2.2.2.2, IP = 2.2.2.2, Error: Unable to remove PeerTblEntry

<164>Oct 03 2008 14:50:17: %ASA-4-113019: Group = 2.2.2.2 Username = 2.2.2.2, IP = 2.2.2.2, Session disconnected. Session Type: IPsec, Duration: 0h:51m:37s, Bytes xmt: 84531039, Bytes rcv: 255159181, Reason: User Requested

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Ignoring msg to mark SA with dsID 507904 dead because SA deleted

<167>Oct 03 2008 14:50:17: %ASA-7-715077: Pitcher: received a key acquire message, spi 0x0

<165>Oct 03 2008 14:50:17: %ASA-5-713041: IP = 2.2.2.2, IKE Initiator: New Phase 1, Intf inside, IKE Peer 2.2.2.2 local Proxy Address 10.10.0.0, remote Proxy Address 0.0.0.0,  Crypto map (outside_map)

<167>Oct 03 2008 14:50:17: %ASA-7-715046: IP = 2.2.2.2, constructing ISAKMP SA payload

<167>Oct 03 2008 14:50:17: %ASA-7-715046: IP = 2.2.2.2, constructing Fragmentation VID + extended capabilities payload

<167>Oct 03 2008 14:50:17: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108

<167>Oct 03 2008 14:50:17: %ASA-7-715077: Pitcher: received a key acquire message, spi 0x0

<166>Oct 03 2008 14:50:17: %ASA-6-713219: IP = 2.2.2.2, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.

<167>Oct 03 2008 14:50:18: %ASA-7-715077: Pitcher: received a key acquire message, spi 0x0

<166>Oct 03 2008 14:50:18: %ASA-6-713219: IP = 2.2.2.2, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.

<167>Oct 03 2008 14:50:19: %ASA-7-715077: Pitcher: received a key acquire message, spi 0x0

<166>Oct 03 2008 14:50:19: %ASA-6-713219: IP = 2.2.2.2, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.

<167>Oct 03 2008 14:50:19: %ASA-7-715077: Pitcher: received a key acquire message, spi 0x0

<166>Oct 03 2008 14:50:19: %ASA-6-713219: IP = 2.2.2.2, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.

Here is a snippet of debug. The KEY_ACUIRE messages continue while the 5505 sends a ISAKMP rekey payload but the 5520 does not reply.

Again the tunnel runs fine untill a rekey. And the tunnel will finally come back up if I let it sit there long enough. Also the routing is fine, its routing over the internet.

So has anyone encountered this type of behavior?

[networker050184]

I haven't seen this before, but if it started happening only after a code upgrade I'd revert back and see if it fixes the issue. If it does you might be missing something in the config that is different between the version. Or it could be a bug...



Nice to see you around Netstudent

[dtlokee]

I would change the timers fir IKE and IPSEC so that they are not trying to re-key at the same time, are you using PFS or DPD?

[Netstudent]

Hey thanks for the reply DT. I tried several different variations on isakmp and ipsec timers in an attempt to find a stable config. None worked. Also had isakmp keepalives enabled on both sides. I just did a zero downtime code change on the HA pair . Went from 8.0.4 to 8.0.4(7) which is an interim release not available to the public yet. Sure enough, that fixed it. I even made a few crypto map changes and the tunnel only dropped like 4 or 5 ping packets during the change. If I would have done that before, I would have been clearing cryptos like a mad man trying to get it back up.

Nice to see you around Netstudent

Thanks, I'm still around. Just lurking lately. I'm trying to get done with school. I'm so burned out with school, and I'm trying to concentrate on finishing without slacking off my grades. The end is hard. I am starting the ASA specialist track though because of work. I think I can get that done real quick. I have enough learning credits for the SNAA course and the SNAf topics I do on a daily basis. Not trying to go Security Pro or anything yet, but perimeter security should be in every engineers toolkit.

[networker050184]

Good luck with school and hopefully you will get a little down time for those certs!

[Ahriakin]

Have you tried keying the lifetime off of Data sent instead of absolute time? It's possible one unit has a clock issue, unlikely but worth seeing if you can isolate it to the function it is using to decide it's time to rekey the SA.

[techistore]

Hi Netstudent,

We are in to the exactly same scenario. We have same hardware 5505 and 5520 and trying to acheive the same LAN-to-LAN connections. We currently have a case open with Cisco and we referred this thread to them so they will give us the code which fixed your issue. Apparently they are asking us to disable idle timeouts in the group policy settings and then see if it is fixing the issue.

Can you please check if the new code you have has the idle time out disabled? If not can you share the code with us then...

[cisco_trooper]

man, i'm glad to see this topic. i was just pondering upgrading my ASAs to the 804 code. With more than one Site To Site VPN in production, perhaps I'll wait until I hear more about this....

[Netstudent]

Hi Netstudent,

We are in to the exactly same scenario. We have same hardware 5505 and 5520 and trying to acheive the same LAN-to-LAN connections. We currently have a case open with Cisco and we referred this thread to them so they will give us the code which fixed your issue. Apparently they are asking us to disable idle timeouts in the group policy settings and then see if it is fixing the issue.

Can you please check if the new code you have has the idle time out disabled? If not can you share the code with us then...

With the new 804 code, the vpn-idle-timeout was set to 30. TAC never told me that the idle timeout was an issue and I never turned it off. They did mention keepalives and DPD apckets which I made sure was on. But I will add that when all this was happening, I did not see consistent DPD R-U-THERE packets going back and forth between the peers. AFter the change to the interim version, it was going back forth between the peers about once a minute consistently.

But TAC never gave me a difinitive answer, they were like "Try this didn't work? okay try this, didn;t work okay try this. Until finally they gave me the dang code.

Even with the interim version, the idle timouts are still on and none of my tunnels are failing.

If your in a tight spot and need to stabilize and TAC isn't giving you what you need, then PM me.

Humdinger# show run all group-policy
group-policy DfltGrpPolicy internal
group-policy DfltGrpPolicy attributes
 banner none
 wins-server none
 dns-server none
 dhcp-network-scope none
 vpn-access-hours none
 vpn-simultaneous-logins 3
 vpn-idle-timeout 30
 vpn-session-timeout none
 vpn-filter none
 ipv6-vpn-filter none
 vpn-tunnel-protocol IPSec
 password-storage disable
 ip-comp disable
 re-xauth disable
 group-lock none
 pfs disable
 ipsec-udp disable
 ipsec-udp-port 10000
 split-tunnel-policy tunnelall
 split-tunnel-network-list none
 default-domain none
 split-dns none
 intercept-dhcp 255.255.255.255 disable
 secure-unit-authentication disable
 user-authentication disable
 user-authentication-idle-timeout 30
 ip-phone-bypass disable
 leap-bypass disable
 nem disable
 backup-servers keep-client-config
 msie-proxy server none
 msie-proxy method no-modify
 msie-proxy except-list none
 msie-proxy local-bypass disable
 msie-proxy pac-url none
 vlan none
 nac-settings none
 address-pools none
 ipv6-address-pools none
 smartcard-removal-disconnect enable
 client-firewall none
 client-access-rule none
 webvpn
  url-list none
  filter none
  homepage none
  html-content-filter none
  port-forward name Application Access
  port-forward disable
  mapi disable
  http-proxy disable
  sso-server none
  svc dtls enable
  svc mtu 1406
  svc keep-installer installed
  svc keepalive 20
  svc rekey time none
  svc rekey method none
  svc dpd-interval client 30
  svc dpd-interval gateway 30
  svc compression deflate
  svc modules none
  svc profiles none
  svc ask none
  ike-retry-timeout 10
  ike-retry-count 3
  customization none
  keep-alive-ignore 4
  http-comp gzip
  download-max-size 2147483647
  upload-max-size 2147483647
  post-max-size 2147483647
  user-storage none
  storage-objects value cookies,credentials
  storage-key none
  hidden-shares none
  smart-tunnel disable
  activex-relay enable
  unix-auth-uid 65534
  unix-auth-gid 65534
  file-entry enable
  file-browsing enable
  url-entry enable

[cisco_trooper]

Just got bit by this.