VPN and IPsec Basics

Virtual Private Networks (VPNs) allow the creation of private networks across a public domain (eg: Internet) for private and secured data transmission.

VPNs are secure and inexpensive. There are 2 types of VPN implementation – using IPsec to create authentication and encryption services; and establishing tunnels via tunneling protocols.

Tunneling is method of overcoming protocol restrictions by encapsulating or wrapping packets from a protocol in another protocol and transmits the encapsulated packet over a network that supports the encapsulating protocol.

Below describes the most common tunneling protocols:

Layer 2 Forwarding (L2F)	A Cisco-proprietary tunneling protocol created for Virtual Private Dial-Up Networks (VPDNs), where devices create secure connections to a corporate network using dial-up connections. L2F was later replaced by L2TP (backward compatible with L2F).
Point-to-Point Tunneling Protocol (PPTP)	Created by Microsoft for secure data transmission between remote networks and corporate network.
Layer 2 Tunneling Protocol (L2TP)	Created by Cisco and Microsoft to replace L2F and PPTP. L2TP merged both L2F and PPTP capabilities into one tunneling protocol.
Generic Routing Encapsulation (GRE)	Another Cisco-proprietary tunneling protocol. It forms (unencrypted) virtual point-to-point links which are able to encapsulate a variety of protocols inside IP packets. It can also be used to protect the private address space from being advertised to any public domain.

Note: There are many types of tunnels. VPNs are tunnels; GRE creates tunnels; Secure Shell (SSH) is also a form of tunnel. Tunnels can encrypt data so that only the other side can see it, as with SSH; they can make a remote network appears local, as with GRE; or they can do both, as with VPN.

IPsec is a suite of industry standard network layer (L3) protocols and algorithms for securing communications over the IP-based networks by authenticating and/or encrypting IP packets. IPsec also includes the IKE protocol for establishing shared security information (Security Association, SA) between 2 network devices or end systems to support secure communication.
Note: Other Internet security protocols, eg: SSL and SSH, are operate from the Transport layer up to the Application layer (L4 – L7).
Note: The official term of “IPsec” as defined by the IETF is often wrongly addressed as “IPSec”.

IPsec secures a path between a pair of gateways, a pair of hosts, or a gateway and a host.

IPsec unable to encrypt non-IP traffic. For such a situation, a GRE tunnel would first be created to encrypt the non-IP traffic and followed by using IPsec to encrypt the GRE tunnel. Additionally, Multicast can only be passed on GRE tunnels.

IPsec-based VPN is comprised of 2 parts – Internet Key Exchange (IKE) protocol and IPsec security protocols – Authentication Header (AH) and Encapsulating Security Payload (ESP). Below describes the flow of IPsec events:
i) IKE Phase 1: IKE Security Negotiation – IKE negotiates how to protect IKE by establishing an authenticated and secure channel between 2 IKE peers called the IKE Security Association. IKE Phase 1 is consists of Main Mode or Aggressive Mode. The peer that initiates the session will propose or offer at least one or more configured ISAKMP policies which specify a combination of encryption algorithm, hash algorithm, authentication type, Diffie-Hellman group, and the lifetime. The remote peer will then try to find a matching configured policy that has the same parameters as the one being sent by its peer. If no matching policy is found, IKE will terminate the negotiation. If a policy is mutually agreed upon, IKE will complete the negotiation process and an ISAKMP SA will be created. Additionally, peers in an IPsec session must authenticate themselves among each other during IKE Phase 1 Main Mode exchange before IKE can proceed.
ii) IKE Phase 2: IPsec Security Negotiation – IKE negotiates how to protect IPsec by negotiating the IPsec security associations (SAs) and generating the keys for IPsec. IKE Phase 2 negotiation is done in only 1 mode – Quick Mode. The peer that initiates the session will propose or offer at least one or more configured transforms which specify a combination of authentication and/or encryption algorithm. The remote peer will then try to find a matching configured transform that has the same parameters as the one being sent by its peer. If no matching transform is found, IKE will terminate negotiation and an IPsec VPN will not be established. If a policy is mutually agreed upon, IKE will complete the negotiation process and an IPsec SA will be created.
iii) IPsec transfers the actual data in the VPN tunnel using the authentication and encryption methods agreed upon the IKE negotiation process.

Internet Key Exchange (IKE) allows 2 VPN endpoints verify the identity of each other (using pre-shared keys or RSA) in IKE Phase 1, and negotiate the methods (security policies) for secured data transmission in IKE Phase 2. IKE manages VPN connections by defining a set of Security Associations (SAs) for each connection. Each SA has its own SAID.

In a VPN, before a communication path is considered secure, the VPN endpoints must be authenticated. IPsec uses the following authentication methods to authenticate peers:

Pre-Shared Keys	Secret key values that are manually configured on each peer.
RSA signatures	Use the exchange of digital certificates to authenticate the peers.

IKE is a hybrid protocol that uses part of Oakley and part of SKEME inside the ISAKMP framework; hence IKE is formerly known as ISAKMP/Oakley. IKE typically uses ISAKMP for key exchange and establishment of SAs, although other methods can be used.

IKE establishes both ISAKMP and IPsec SAs for an IPsec VPN session. IKE first negotiates an ISAKMP SA with the peer. It is possible to configure multiple policy statements with different parameters, and then allow the peers to negotiate and establish a mutual agreement.

An IPsec SA defines the security algorithms or parameters associated with particular connection – the IPsec protocol (AH, ESP, or both), the session keys used for data encryption, etc. IPsec SAs are unidirectional (simplex); hence there is always more than 1 IPsec SA per IPsec connection. In cases where only either AH or ESP is used, 2 SAs will be created for each connection – one for each the incoming and outgoing traffic. In cases where AH and ESP are used in conjunction, 4 SAs will be created.

The Internet Security Association and Key Management Protocol (ISAKMP) defines the procedures for authenticating peers, IKASAMP and IPsec SAs establishment, negotiation, modification, and deletion; key generation, and threat mitigation (eg: DoS and replay attacks).

Instead of ISAKMP (the use of ipsec-isakmp keyword along with the crypto map global configuration command), manual keying (the use of ipsec-manual keyword along with the crypto map global configuration command) which require manual entry of the shared secret session keys (used for hashing and encryption) on both crypto endpoints is also possible.

IPsec operation requires both ends to be configured with the same transform set, which specifies the methods for encrypt and decrypt the data. IPsec uses 2 primary security protocols – Authentication Header (AH) and Encapsulating Security Payload (ESP). These protocols are used for secured data transmission through an IPsec-based VPN tunnel. IPsec-based VPNs can be established using AH only, ESP only, or both AH and ESP.

The Authentication Header (AH) protocol provides authentication for both the IP header and data of a packet using a one-way hash function. The sender first generates a one-way hash, and then the receiver generates the same one-way hash. If the packet has changed in any way, it won’t be authenticated and will be dropped. IPsec relies upon AH to guarantee authenticity. AH provides integrity check on the entire packet, but it doesn’t provide any encryption services.

ESP only provides integrity check (and encrypts) on the data of a packet (and the ESP header); while AH checks the entire packet – both header and data. AH is used for authentication only, while ESP can be used for either encryption or authentication only; or both.

Although AH and ESP are typically used independently, they are often being used together to provide data encryption service. Note: It is important to use authentication even if encryption is used, as encrypt-only implementations are subject to some forms of effective attacks.

ESP provides the following 4 functionalities or capabilities:

Confidentiality (Encryption)	Provided through the use of symmetric encryption algorithms, eg: DES, 3DES. Confidentiality can be selected separately from all other services, but the confidentiality selected must be the same on all VPN endpoints.
Data Origin Authentication and Connectionless Integrity	Joint services offered as an option in conjunction with the confidentiality option. Authentication ensures that the connection is established with the desired system.
Anti-Replay Protection	This service works only if data origin authentication is selected. It is based upon the receiver – it is effective only if the receiver checks the sequence number of the received packets with a sliding window of the destination gateway or host to prevent replay attacks.
Traffic Flow Confidentiality	This service works only when tunnel mode is selected. It is most effective if implemented at a security gateway, where the source-destination patterns of attacks is visible.

The degree of security of IPsec VPN is based on the encryption algorithm used and the length of the pre-shared key. The longer the key, the harder it is to be broken.

IPsec is not bound to any specific encryption or authentication algorithm, keying or technology, or security algorithms, which allows IPsec to support newer and better algorithms.

IPsec supports the following 3 types of encryption algorithms:

Data Encryption Standard (DES)	Uses a 56-bit key that ensures high performance encryption. Uses a symmetric key cryptosystem.
Triple DES (3DES)	A variant of DES that breaks data into 64-bit blocks. 3DES then processes each block 3 times, each time with an independent 56-bit key, hence providing significant improvement in encryption strength over DES. Uses a symmetric key cryptosystem.
Advanced Encryption Standard (AES)	Provides stronger encryption than DES and is more efficient than 3DES. Key lengths can be 128-, 192-, and 256-bit keys.

Encryption algorithms (eg: DES, 3DES, and AES) require a symmetric shared secret key to perform encryption and decryption. The Diffie-Hellman Key Exchange (D-H) is a public key exchange process that allows 2 parties that have no prior knowledge of each other to negotiate symmetric shared secret keys used for encryption and decryption over an insecure channel. The shared secret keys are negotiated between crypto endpoints dynamically, which only the prime modulus size for use in the D-H exchange is needed to be specified. IKE uses the D-H keys to encrypt the ISAKMP SA when establishing the IPsec SAs. Additionally, D-H is also used to generate shared secret keys to be used in the ciphers specified in the IPsec transforms, which are then used in conjunction with the D-H keys by the IPsec to encrypt and decrypt the data passes through the IPsec VPN tunnel.

IPsec uses a data integrity algorithm called Hash-based Message Authentication Code (HMAC) that adds a hash to the message to ensure data integrity. The hash guarantees the integrity of the original message. If the transmitted hash matches the received hash, the message is considered has not been tampered.

IPsec uses the following 2 HMAC algorithms:

Message Digest Algorithm 5 (MD5)	Uses a 128-bit shared secret key. The message and 128-bit shared secret key are combined and run through the MD5 hash algorithm, producing a 128-bit hash. This hash is then added to the original message and sent to the destination host.
Secure Hash Algorithm 1 (SHA-1)	Uses a 160-bit shared secret key. The message and 160-bit shared secret key are combined and run through the SHA-1 hash algorithm, producing a 160-bit hash. This hash is then added to the original message and sent to the destination host.

Below lists the 2 modes of IPsec operation:

Transport	Only the payload of the IP packet is encrypted and/or authenticated. The routing is intact, as the IP header is neither modified nor encrypted. It is used for host-to-host or end-to-end communications (end systems perform the security processing).
Tunnel	The entire IP packet is encrypted and/or authenticated. The original packet is encapsulated entirely into a new packet with a new set of source and destination IP addresses for routing to work. It is used for network-to-network or portal-to-portal communications (gateways or routers perform the security processing). It is the traditional and default mode of IPsec VPNs.

The Tunnel mode is most commonly used to secure existing IP traffic for communication between end systems on networks connected to IPsec-enabled routers. With routers or VPN endpoints performing the IPsec encryption, no changes are required to the software and drivers on the end systems – the IPsec implementation is transparent to end systems.

IPsec in AH Transport Mode

IPsec in AH Tunnel Mode

IPsec in ESP Transport Mode

IPsec in ESP Tunnel Mode

The main challenge and problem faced by IKE and IPsec is NAT, as both protocols were not designed to work through NAT. NAT Traversal (NAT-T) has evolved as a method of enabling IPsec-protected IP packets to work well in NAT environments by encapsulating ISAKMP and ESP packets into UDP Port 4500 packets.
The IPsec NAT Transparency feature was introduced in Cisco IOS Release 12.2T.
The access control list configuration named UDP Port 4500 as “non500-isakmp”. :-)