Part 2: How IPsec works - IPsec, IPv4 and IPv6
How IPsec Works
IPsec uses the Authentication Header (AH) and the Encapsulating Security Payload (ESP) to apply security to IP packets. These protocols define IP header options (for IPv4) or header extensions (for IPv6). Both AH and ESP headers include a Security Parameter Index (SPI). The SPI, along with the security protocol in use (AH or ESP) and the destination IP address, combine to form the Security Association (SA).
The sending host knows what kind of security to apply to the packet by looking in a Security Policy Database (SPD). The sending host determines what policy is appropriate for the packet, depending on various selectors (for example, destination IP address or transport layer ports), by looking in the SPD. The SPD indicates what the policy is for a particular packet: either the packet requires IPsec processing of some sort, in which case it is passed to the IPsec module for processing; or it does not, in which case it is simply passed along for normal IP processing. Outbound packets must be checked against the SPD to see what kind (if any) of IPsec processing to apply. Inbound packets are checked against the SPD to see what kind of IPsec service should be present in those packets.
A second database, called the Security Association Database (SAD), includes all security parameters associated with all active SAs. When an IPsec host wants to send a packet, it checks the appropriate selectors to determine the Security Policy Database security policy for that referenced destination/port/application. If the SPD references a particular Security Association, the host can look up the SA in the Security Association Database to identify appropriate security parameters for that packet.
Key management is another important aspect of IPsec. Two important key management specifications associated with IPsec are: the Internet Security Association and Key Management Protocol (ISAKMP) and the Internet Key Exchange (IKE).
ISAKMP, a generalized protocol for establishing Security Associations and cryptographic keys within an Internet environment, defines the procedures and packet formats needed to establish, negotiate, modify, and delete Security Associations. It also defines payloads for exchanging key generation and authentication data. These formats provide a consistent framework for transferring this data, independent of how the key is generated or what types of encryption or authentication algorithms are being used.
ISAKMP was designed to provide a framework that can be used by any security protocols that use Security Associations, not just IPsec. To be useful for a particular security protocol, a Domain of Interpretation or DOI must be defined. The DOI groups related protocols for the purpose of negotiating Security Associations--security protocols that share a DOI choose protocol and cryptographic transforms from a common namespace. They also share key exchange protocol identifiers, as well as a common interpretation of payload data content.
While ISAKMP and the IPsec DOI provide a framework for authentication and key exchange, ISAKMP does not actually define how those functions are executed. The Internet Key Exchange (or IKE) protocol, working within the framework defined by ISAKMP, does define a mechanism for hosts to perform these exchanges.
By defining a separate protocol for the generalized formats required to do key and Security Association exchanges, ISAKMP can be used as a base to build specific key exchange protocols. The foundation protocol can be used for any security protocol, and does not have to be replaced if an existing key exchange protocol is replaced for some reason, such as if a security flaw was found in the protocol.
IPsec, IPv4, and IPv6
IPsec provides security services for either IPv4 or IPv6, but the way it provides those services is slightly different in each. IPv4 uses header options: every IP packet contains 20 bytes-worth of required fields, and any packet that has any "special" requirements can use up to 40 bytes for those options. This tends to complicate packet processing, since routers must check the length of each packet it receives for forwarding--even though many of those header options are related to end-to-end functions such as security, with which routers are not otherwise concerned.
IPv6 simplifies header processing: every IPv6 packet header is the same length, 40 bytes, but any options can be accommodated in extension headers that follow the IPv6 header. IPsec services are provided through these extensions.
The ordering of IPsec headers, whether within IPv4 or IPv6, has significance. For example, it makes sense to encrypt a payload with the ESP header, and then use the Authentication Header to provide data integrity on the encrypted payload. In this case, the AH header appears first, followed by the ESP header and encrypted payload. Reversing the order, by doing data integrity first and then encrypting the whole lot, means that the recipient can be sure of who originated the data, but not necessarily certain of who did the encryption.