How do I troubleshoot client SSL/TLS negotiation errors when I connect to an Application Load Balancer that uses HTTPS?

Short description

A negotiation error occurs when the client TLS can't establish an SSL session with the Application Load Balancer. You get this error because the load balancer's security policy doesn't support the connecting client's security protocol or cipher suite. The error causes the clientTLSnegotiationErrorCount metric to increment.

To establish a secure connection, make sure that your client's initial SSL/TLS handshake meets the following requirements:

• Matches one or more cipher suites that are supported by the Application Load Balancer's security policy.
• Has an SSL/TLS protocol version that's specified in the Application Load Balancer's security policy.

Resolution

Review the Application Load Balancer's connection logs during the time period when the ClientTLSNegotiationErrorCount metric incremented. The logs show the security protocols and ciphers that the client presented to the Application Load Balancer.
Note: Make sure that you activated your connection logs. Or, if you have access to the client that's failing to connect, take a packet capture, and then review the Client Hello packet. For more information on packet capture tests, see How do I troubleshoot network performance issues between EC2 Linux or Windows instances in a VPC and an on-premises host over the internet gateway?

Determine the protocols and ciphers that are supported by your load balancer's security policy

Application Load Balancers don't support custom security policies. For more information about security policies, including the default security policy, see Application Load Balancer security policies.

You can review the attached security policies of your Application Load Balancer in two ways:

• Use the Amazon Elastic Compute Cloud (Amazon EC2) console.
• Use AWS Command Line Interface (AWS CLI) and the describe-listeners command.

Note: The AWS CLI section assumes that your setup meets the following prerequisites:

• AWS CLI is configured with a user who has the appropriate IAM permissions to describe against the Elastic Load Balancing Service.
• If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

From the Amazon EC2 console

To review the security policy, complete the following steps:

1. Open the Amazon EC2 console.
2. On the navigation pane, under Load Balancing, choose Load Balancers.
3. Select the Application Load Balancer that you want. Then, choose Listeners and rules.
4. In the table, expand the Security Policy column. Review the supported security protocols and cipher suites.

From the AWS CLI

To review the security policy, run the following commands:

1. To locate the Application Load Balancer's Amazon Resource Name (ARN), run the describe-load-balancers command:

   aws elbv2 describe-load-balancers --region your_region | grep 'LoadBalancerName\|LoadBalancerArn'

2. To locate the Application Load Balancer's security policy, run the describe-listeners command:

   aws elbv2 describe-listeners --region your_region --load-balancer-arn your_loadbalancer_arn | grep 'SslPolicy'

   Note: Replace your_region with the Region that contains your resources. Replace your_loadbalancer_arn with your load balancer's ARN.

Use connection logs to determine the security protocols and cipher suites presented by the client

Use the Amazon Simple Storage Service (Amazon S3) console to decompress the connection logs and show the information in them. Then, download the files for further review.
Note: Use the bash tools zcat, awk, and sed to process the connection logs.

The bash tool awk parses the following field numbers or values within the log:

($1) timestamp 
($2) client_ip  
($3) client_port  
($4) listener_port 
($5) tls_protocol 
($6) tls_cipher 
($7) tls_handshake_latency 
($8) leaf_client_cert_subject 
($9) leaf_client_cert_validity 
($10) leaf_client_cert_serial_number 
($11) tls_verify_status

The following sections describe the different ways that you can use to process the connection logs.

Identify the time period contained within the connection logs
Note: Logged time values are in UTC.

zcat *.log.gz | gawk '{print substr($1,0,16)}'| sort | uniq | sed '1p;$!d'

2024-03-30T18:14
2024-03-30T19:04

Identify or compare the errors present within the connection logs
Note: The tls_verify_status field is primarily for mTLS connections. However, the field can be used to show a failed TLS connection attempt because of a mismatch between one or more protocols or cipher suites.

zcat *.log.gz | gawk '{print $11}' | sort | uniq -c | sort -r

1354 Failed:UnmappedConnectionError
 176 Success

Identify unique clients that show the most connection errors

zcat *.log.gz | gawk '$11 ~ "Failed"' | gawk '{print $2}' | sort | uniq -c | sort -r

440 a.a.a.a
436 b.b.b.b
386 c.c.c.c
 90 d.d.d.d
  1 e.e.e.e
  1 f.f.f.f

Identify the cipher suite and protocol used by the client

Note the protocol, cipher suite, and the subsequent error.

zcat *.log.gz | gawk '($11 ~ "Failed" && $6 != "-")' | gawk '{print $2,$5,$6,$11}' | sort | uniq -c | sort -r

40 a.a.a.a TLSv1.2 ECDHE-RSA-AES256-SHA Failed:UnmappedConnectionError
30 b.b.b.b TLSv1.2 ECDHE-RSA-AES256-SHA Failed:UnmappedConnectionError
15 c.c.c.c TLSv1.2 ECDHE-RSA-AES256-SHA Failed:UnmappedConnectionError
 2 d.d.d.d TLSv1.2 ECDHE-RSA-AES256-SHA Failed:UnmappedConnectionError
 2 e.e.e.e TLSv1.2 ECDHE-RSA-AES256-SHA Failed:UnmappedConnectionError
 1 f.f.f.f TLSv1.2 ECDHE-RSA-AES256-SHA Failed:UnmappedConnectionError

Use packet captures to determine the security protocol and cipher suite presented by the client

After you take a packet capture on the client that initiated the connection, review the Client Hello packet that's sent to the Application Load Balancer.

Use Tshark for packet capture
Note: You can also use Wireshark (GUI) instead of Tshark (Terminal) to analyze the packet capture.

The Tshark example uses the following information:

• Client IP address: 192.168.150.100
• Application Load Balancer IP address: 10.10.10.10
• Destination port: 443

Complete the following steps:
Note: Replace your_file.pcapng with the name of your file.

1. To find all the packets in the conversation between the client and the Application Load Balancer's IP addresses, run the following command:

   tshark -r your_file.pcapng '(ip.addr == 10.10.10.10 && tcp.port == 443)'
   
   69   1.428028 192.168.150.100 → 10.10.10.10 TCP 66 61747 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
   70   1.481769 10.10.10.10 → 192.168.150.100 TCP 66 443 → 61747 [SYN, ACK] Seq=0 Ack=1 Win=26883 Len=0 MSS=1460 SACK_PERM=1 WS=256
   71   1.481905 192.168.150.100 → 10.10.10.10 TCP 54 61747 → 443 [ACK] Seq=1 Ack=1 Win=131328 Len=0
   72   1.490709 192.168.150.100 → 10.10.10.10 TLSv1 244 Client Hello
   74   1.541926 10.10.10.10 → 192.168.150.100 TCP 56 443 → 61747 [ACK] Seq=1 Ack=191 Win=28160 Len=0
   75   1.541926 10.10.10.10 → 192.168.150.100 TCP 56 [TCP Dup ACK 74#1] 443 → 61747 [ACK] Seq=1 Ack=191 Win=28160 Len=0
   76   1.541926 10.10.10.10 → 192.168.150.100 TLSv1.2 61 Alert (Level: Warning, Description: Close Notify)
   77   1.541926 10.10.10.10 → 192.168.150.100 TCP 56 443 → 61747 [FIN, ACK] Seq=8 Ack=191 Win=28160 Len=0
   78   1.542087 192.168.150.100 → 10.10.10.10 TCP 54 61747 → 443 [ACK] Seq=191 Ack=9 Win=131328 Len=0
   79   1.543501 192.168.150.100 → 10.10.10.10 TCP 54 61747 → 443 [FIN, ACK] Seq=191 Ack=9 Win=131328 Len=0
   95   1.614454 10.10.10.10 → 192.168.150.100 TCP 56 443 → 61747 [ACK] Seq=9 Ack=192 Win=28160 Len

   The preceding output shows the following information:

   The client (192.168.15.100) initiates a connection with the Application Load Balancer (10.10.10.10).
   Shortly after the initial TCP handshake (SYN, SYN-ACK, ACK), Client Hello is followed by an ACK from the Application Load Balancer (Packet 72).
   The Application Load Balancer then issues a Close Notify to show that the session must close (Packet 76).

2. Locate the SSL/TLS protocol and cipher suites used in the Client Hello packets. These packets are used in the conversation between the client IP address (192.168.15.100) and the load balancer's IP address (10.10.10.10).

   To move the Client Hello packet (tls.handshake.type == 1) for the conversation into its own PCAP file, run the following command:

   tshark -r your_file.pcapng -O -P -w ALB_Sessions.pcap '(ip.addr == 10.10.10.10 && tcp.port == 443) && (tls.handshake.type==1)'

3. To move the cipher suites from the ALB_Sessions.pcap file to a text file, run the following command:

   tshark -r ALB_Sessions.pcap -Y ssl.handshake.ciphersuites -Vx > TLS_outfile.txt

4. To concatenate the file and use grep to search for the Version: field, run the following command. Review the TLS version used in the Client Hello packet.

   cat TLS_outfile.txt | grep "Version:"
       
   0100 .... = Version: 4
       Version: TLS 1.0 (0x0301)
           Version: TLS 1.2 (0x0303)

5. To concatenate the file and use grep to search for the Cipher Suite: field, run the following command. Review the cipher suites used within the Client Hello packet.

   cat TLS_outfile.txt | grep "Cipher Suite:"
   
   Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
   Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)

6. To compare the hex values to the original file, run the following command:

   tshark -r your_file.pcapng -T fields -Y '(tls.handshake.type==1 && ip.addr == 10.10.10.10)' -e tls.handshake.version -e tls.handshake.ciphersuite
   
   0x0303  0xc014,0x00ff

   The command reads the file.pcapng file and parses it for the Client Hello packet that was sent to the Application Load Balancer's IP address (10.10.10.10). The command then prints the tls.handshake.version and tls.handshake.ciphersuites as individual columns. Compare the output to confirm that these are the hex values that appeared in the TLS_outfile.txt file.

Analyze the packet capture

In this example, the only cipher suite presented by the client was TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA. This information follows the IANA naming schema, but the Application Load Balancer uses the OpenSSL naming schema.
Note: Use this table to translate IANA to the OpenSSL naming schema. TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA is equal to ECDHE-RSA-AES256-SHA.

Update your load balancer's security policy

To accommodate the client's supported protocols or ciphers, perform the following actions:

1. Open the Amazon EC2 console.
2. On the navigation pane, choose Load Balancers.
3. Select the load balancer.
4. On the Listeners and rules tab, select the text in the Protocol:Port column to open the detail page for the listener.
5. On the Details page, choose Actions, and then choose Edit listener.
6. In the Secure listener settings section, for Security policy, choose a new security policy.
7. Choose Save changes.

Update the client's supported security protocols and cipher suites

To update the client's supported security protocols and cipher suites, refer to the documentation of the operating system of the device that's sending the request.

Related information

What is an SSL/TLS certificate?

How do I troubleshoot the connection errors that I get with the Application Load Balancers over HTTP?

How do I identify and resolve client connection issues when I use mTLS with the Application Load Balancer?