TCP Three-Way Handshake

The Handshake: A High-Level View

Before diving into the technical details, let's start with a simple analogy. Imagine two people, Alice and Bob, establishing a two-way radio communication.

1. Alice to Bob: "Bob, this is Alice. Can you hear me? Over." (This is the SYN packet)
2. Bob to Alice: "Yes, Alice, I can hear you. Can you hear me? Over." (This is the SYN-ACK packet)
3. Alice to Bob: "Yes Bob, I can hear you loud and clear. Over and out." (This is the ACK packet)

At this point, both have confirmed that they can send and receive messages. The channel is open. The TCP three-way handshake works on this exact principle.

To see this in action, watch the first part of the following video.

The Technical Purpose: Synchronizing Sequence Numbers

The handshake does more than just confirm that both parties are reachable. Its primary technical purpose is to synchronize the initial sequence numbers (ISNs) between the client and the server.

As we discussed previously, TCP ensures data is reassembled in the correct order. It does this by assigning a sequence number to every byte of data it sends. The three-way handshake is where the starting numbers for this sequencing are agreed upon.

To get a detailed, step-by-step breakdown of this process, please read the first few sections of the following article.

To summarize and visualize the process described in the article:

1. Client -> Server: SYN

   • The client initiates the connection.
   • It sends a TCP segment with the SYN (Synchronize) flag set.
   • This packet contains the client's randomly generated Initial Sequence Number (ISN). Let's call it ISN_C.

2. Server -> Client: SYN-ACK

   • The server receives the client's SYN packet.
   • It responds with a segment that has both the SYN and ACK (Acknowledge) flags set.
   • This packet contains the server's own randomly generated Initial Sequence Number, ISN_S.
   • Crucially, the Acknowledgment Number field is set to ISN_C + 1. This confirms to the client that its SYN packet was received.

3. Client -> Server: ACK

   • The client receives the server's SYN-ACK.
   • It sends one final segment with the ACK flag set.
   • The Acknowledgment Number field in this packet is set to ISN_S + 1, confirming receipt of the server's SYN packet.

At this point, the connection is ESTABLISHED. Both sides have acknowledged each other's sequence numbers, and data transfer can begin.

The image below illustrates this entire flow, including the changing states of the client and server.

A Deeper Dive: Flags and Numbers in Action

Let's reinforce this with a more detailed video walkthrough. The following video explores exactly how the flags, sequence numbers, and acknowledgment numbers appear in the TCP headers during the exchange.

An interesting detail from that video is the concept of the "phantom byte". Notice that the SYN flag itself consumes one sequence number, even though no actual data was sent. This is why the acknowledgment for a packet with sequence X and the SYN flag set is X+1. This is a design feature that ensures the synchronization and acknowledgment mechanism is consistent.

Connection States and Security Implications

Understanding the handshake is not just academic; it has direct applications in security. The states that a connection moves through (LISTEN, SYN_SENT, SYN_RECEIVED, ESTABLISHED) are important for network analysis and are exploited in common attacks.

Let's explore this using another resource.

Let's expand on those two key security concepts:

1. SYN Flood (Denial of Service Attack)
This attack exploits the server's "half-open" connection state.

• An attacker sends a high volume of SYN packets to a server, often from spoofed (fake) IP addresses.
• The server responds to each SYN with a SYN-ACK and allocates resources (memory, CPU time) for each new connection, placing them in the SYN_RECEIVED state.
• Because the attacker never sends the final ACK, these connections remain "half-open," consuming server resources.
• Eventually, the server's connection table fills up, and it can no longer accept new connections from legitimate users, resulting in a denial of service.

2. Port Scanning (SYN Scan)
This is a fundamental reconnaissance technique for a penetration tester.

• To check if a port is open, a scanning tool like Nmap sends a SYN packet to the target port.
• If the port is open: The target server responds with a SYN-ACK. The scanner now knows the port is open. Instead of completing the handshake with an ACK, it sends a RST (Reset) packet to tear down the connection. This is called a "half-open" or "stealth" scan because it avoids creating a full connection that might be logged.
• If the port is closed: The target server responds with a RST packet immediately.
• If no response is received: The port is likely firewalled.

Conclusion

In this lesson, we deconstructed the TCP three-way handshake, moving from a simple analogy to the precise mechanics of the protocol.

Key Takeaways:

• The handshake is a three-step process: SYN, SYN-ACK, ACK.
• Its primary purpose is to establish a connection and synchronize initial sequence numbers (ISNs) between the client and server.
• The ISNs are randomized to enhance security against attacks like TCP session hijacking.
• This process is fundamental to a penetration tester's toolkit, forming the basis for SYN flood attacks and SYN-based port scanning.

Now that we've seen how a reliable connection is established at the Transport Layer (Layer 4), our next lesson will move down the OSI model to the Network Layer (Layer 3). We will examine the learning outcome: Explain the structure of an IP address and the function of a subnet mask, to understand how devices are uniquely identified and located on a network.