Mitigating TCP connection resets in AWS

Some time ago, we encountered broken connections in our logs. After some time of inactivity, new connections got TCP RST. We started to dig deeper to find out what happened.

It turned out that what we’ve experienced was a common issue in the AWS environment.

AWS has Network Load Balancers (NLBs) that have idle timeout as stated in the documentation:

For each TCP request that a client makes through a Network Load Balancer, the state of that connection is tracked. If no data is sent through the connection by either the client or target for longer than the idle timeout, the connection is closed. If a client or a target sends data after the idle timeout period elapses, it receives a TCP RST packet to indicate that the connection is no longer valid.

Elastic Load Balancing sets the idle timeout value for TCP flows to 350 seconds. You cannot modify this value. Clients or targets can use TCP keepalive packets to reset the idle timeout. Keepalive packets sent to maintain TLS connections cannot contain data or payload.

Many others discovered this too. [1] [2].

With this knowledge we could do two things with connections that go through load balancers:

1. Close the connection before 350 seconds of inactivity

2. Enable Keep-Alive for TCP connections

Let’s explore both options:

Close the connection before 350 seconds

To prevent the closing of the TCP connection we need to close it first. We can achieve this by introducing timeout and close connections before they are closed by AWS.

To make this in reactive Spring WebClient (and underlying Reactor HttpClient) we need to specify maxIdleTime in ConnectionProvider:

import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.netty.http.client.HttpClient;
import reactor.netty.resources.ConnectionProvider;
        
    webClientBuilder
        .clientConnector(
                new ReactorClientHttpConnector(
                        HttpClient.create(ConnectionProvider
                            .builder("aws-connection-provider")
                            .maxIdleTime(Duration.ofSeconds(300)) // must be less than 350 s
                            .build())));

The above configuration option is a bit complicated, but it only has to be done once.

Enable Keep-Alive for TCP connections

In some scenarios, we want to avoid delays incurred by re-establishing TCP connections by generating more traffic and keeping the connection alive. Such setup is trickier because it additionally requires tweaking the keep alive setting in the operating system.

Let’s start with WebClient/HttpClient configuration:

import io.netty.channel.ChannelOption;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.netty.http.client.HttpClient;
        
    webClientBuilder
        .clientConnector(
            new ReactorClientHttpConnector(
                HttpClient.create()
                    .option(ChannelOption.SO_KEEPALIVE, true)));

The above setting enables TCP keepalive flag, but as said before it’s not enough.

We need to adjust the net.ipv4.tcp_keepalive_time kernel parameter which determines the frequency of sending the TCP keepalive packets to keep a connection alive if it is currently unused. In Linux-based systems this parameter default value is 7200 seconds (2h).

Most applications are run inside Docker containers and to adjust such parameter we need to pass the --sysctl option in the docker run command. This may not be possible for the normal user though.

Fortunately when in AWS we can set it in the ECS task definition:

            "systemControls": [
                {
                    "namespace": "net.ipv4.tcp_keepalive_time", 
                    "value": "300"
                }
            ]

Setup is even more complicated than the first one. But it enables us to have the connection ready without the time-consuming TCP connection establishing phase. Please note that keeping connections may not be suitable for you, especially if you have a lot of them, and you do not wish to increase traffic just for the sake of keeping connections alive.

Conclusion & TL;DR

In this post, we learned about the 350 seconds timeout in AWS and how to deal with it in reactive Spring HTTP clients - either by finishing TCP connections before AWS timeout or keeping them alive.