The company I work for (Acme Networking) does “realtime” data transfers of their production servers to an off-site disaster recovery facility. This lets them bring up a near current copy of our servers in a disaster recovery scenario at an off-site location. However, this generates a great deal of traffic, typically crushing our Internet pipe.
Enter Cisco’s ASA Firewall policies: Enabling QoS policing
Our disaster recovery site receives our replicated data over a LAN-to-LAN VPN tunnel. Initially, this made creating a QoS policy limiting their bandwidth quite easy. Later however, it added complexity (we’ll get to that).
To limit a LAN-to-LAN tunnel to a specific amount of bandwidth you will need to create a class-map that matches the tunnel-group, a corresponding policy-map, and then apply that policy-map to an interface. We do all that in the example below, keeping the tunnel to 50Mbps or less.
! Create the class-map ! Class-maps just "identify" the traffic class-map DR-Tunnel-Group match flow ip destination-address match tunnel-group 18.104.22.168 exit ! Create the policy-map ! Policy-maps tell the firewall what to do with the traffic ! you identified in the class-map policy-map Police-DR-50Mbps class DR-Tunnel-Group police output 52428750 4000000 exit exit ! Now you have to tell the firewall *where* to take the action ! specified in the policy-map ! In our case we are applying it to the outside interface of the firewall service-policy Police-SG-50Mbps interface outside
While this is a great solution for keeping LAN-to-LAN tunnels in line with an environment that has constrained bandwidth, it isn’t very dynamic. Before QoS, our disaster recovery traffic killed our Internet bandwidth and caused production issues. Our Recovery Point Objective however was close to real-time. After QoS we had plenty of available Internet bandwidth during the day, but the continued constraint on our backups during off-hours affected the point-in-time we could recover to in the event of a failure.
Enter Cisco’s ASA Firewall policies: Enabling QoS policies based on time-range
We decided the best approach would be to continue limiting our backup traffic to 50Mbps during business hours (7AM – 8 PM ET, Monday – Friday) and allowing 100Mbps at all other times and days. Since there is no “match time-range xyz” in the class-map function (Cisco, if you are reading this, please make this a feature! I’ll take a Cisco shirt for payment of my idea) we have to use access-lists to identify the traffic.
Since this is a LAN-to-LAN VPN, we already have an access-list identifying our interesting traffic (hereafter I’m going to call it an encryption domain), so this makes it very easy to setup two more access-lists; one defining our encryption domain with a time-range of “business hours” and the other defining the encryption domain with a time-range of “off hours”. Just to give a starting point, here is a subset of our crypto config:
crypto map crypto-map 30 match address ED-DR-L2L crypto map crypto-map 30 set pfs crypto map crypto-map 30 set peer 22.214.171.124 crypto map crypto-map 30 set ikev1 transform-set ESP-AES-256-SHA ESP-AES-192-SHA ESP-AES-128-SHA crypto map crypto-map 30 set nat-t-disable access-list ED-DR-L2L extended permit ip object-group OG-Acme-To-DR-ED object-group OG-DR-To-Acme-ED object-group network OG-Acme-To-DR-ED network-object 172.18.0.0 255.255.0.0 network-object 192.168.0.0 255.255.0.0 network-object 172.21.0.0 255.255.255.0 network-object 172.25.1.0 255.255.255.0 object-group network OG-DR-To-Acme-ED network-object 10.15.86.176 255.255.255.240 network-object 10.15.97.0 255.255.255.240 exit
Because the encryption domain is already specified, the time based access-lists become easy:
time-range TR-OffHours periodic weekdays 20:00 to 23:59 periodic weekdays 0:00 to 7:00 periodic weekend 0:00 to 23:59 time-range TR-OnHours periodic weekdays 7:01 to 19:59 exit access-list Policy-DR-OffHours extended permit ip object-group OG-Acme-To-DR-ED object-group OG-DR-To-Acme-ED time-range TR-OffHours access-list Policy-DR-OnHours extended permit ip object-group OG-Acme-To-DR-ED object-group OG-DR-To-Acme-ED time-range TR-OnHours class-map DR-Tunnel-OffHours match access-list Policy-DR-OffHours class-map DR-Tunnel-OnHours match access-list Policy-DR-OnHours exit policy-map Police-DR class DR-Tunnel-OffHours police output 104857000 8000000 class DR-Tunnel-OnHours police output 52428500 4000000 exit exit service-policy Police-DR interface outside
This is an especially elegant solution because if we make any changes to the VPN encryption domain via the object-groups, those changes will be “replicated” to the QoS policies as well.
However, here lies the problem with this configuration: Once a TCP session has been established and assigned a policy (based on the time of day it was when the session was established) it will forever flow over that policy. So, a TCP session established at 4PM ET which is only allowed 50Mbps will still only receive 50Mbps if the session is still active at 10PM ET. Conversely, a TCP session established at 6AM ET which is allowed 100Mbps will continue to soak up throughput well into business hours if the session stays established until 11AM ET. Normally, this is not a big deal. Web based TCP connections are short lived, minutes at most. Even FTP or some other file transmission is probably going to use different TCP sessions throughout its relatively short connection lifetime. But this traffic is backup / DR replication. The backup software installed on the servers maintains the same TCP session with the DR site throughout the day. So what our policy ends up doing is allowing a combined total of 150Mbps (depending on when a session was initialized) regardless of time or day.
Enter Cisco’s ASA Firewall policies: VPN Access Hours
For the solution we have to go back to time-ranges. We can make the tunnel available during certain hours and days and unavailable otherwise via group-policy:
time-range TR-DR-VPN periodic weekdays 0:00 to 7:00 periodic weekdays 7:10 to 19:59 periodic weekdays 20:10 to 23:59 periodic weekend 0:00 to 23:59 exit ! This command already set by default ! Just making sure for documentation to keep it in config ! You can verify the command is set by issuing 'show run all | include sysopt no sysopt connection preserve-vpn-flows ! This is the default group-policy set by the tunnel group group-policy GroupPolicy1 attributes vpn-access-hours value TR-DR-VPN exit ! Tear down the VPN so that it re-reads the new configuration clear crypto ikev1 sa 126.96.36.199
So here is the problem with this solution: The ‘vpn-access-hours’ value does not disconnect the tunnel when the time is outside of scope. It only prevents the firewall from initiating or responding to tunnel creation events during the off times (c’mon Cisco, get this stuff fixed!). So, you still need to disconnect the VPN tunnel at the beginning of the transition times so that the TCP sessions can time out.
We have elected to use our management server to SSH into the firewall at 7:02 and 20:02 every weekday and issue a ‘clear crypto ikev1 sa 188.8.131.52’ automatically with a service account. However, if you are running version 9.2(1) or later on your firewall, you can use code similar to the following to handle things locally:
event manager applet clear-vpn-peer-184.108.40.206_1 event timer absolute time 7:02:00 output none action 1 cli command "clear crypto ikev1 sa 220.127.116.11" event manager applet clear-vpn-peer-18.104.22.168_2 event timer absolute time 20:02:00 output none action 1 cli command "clear crypto ikev1 sa 22.214.171.124" exit
To recap: If you want to make a standard QoS policy change policing the amount of bandwidth a VPN tunnel utilizes, use figure 1’s config to craft a variation of your own. If you would like to rate-limit a subset of outbound Internet traffic (not going over a VPN tunnel) then figure 3 should provide most of the configuration you need. On the other hand, if you need time-of-day QoS policing on a VPN tunnel, then combine figures 3 and 4 and you should have a pretty bullet-proof policy in place.
NotesThis config was written on an ASA running software version 9.5(x)
The peer IP address in these configs is a class D address to obfuscate the real peer address.
These configs were bounced off of Cisco TAC for accuracy and appropriateness.