Differences between revisions 1 and 7 (spanning 6 versions)
Revision 1 as of 2019-06-10 17:04:56
Size: 63
Editor: PeterWu
Comment: dummy
Revision 7 as of 2019-06-13 23:08:25
Size: 12019
Editor: PeterWu
Comment: mention DSB
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
Describe TLS here.
(Dummy page, I cannot seem to rename SSL)
##language:en
<<TableOfContents(4)>>

= Transport Layer Security (TLS) =
Transport Layer Security (TLS) provides security in the communication between two hosts. It provides integrity, authentication and confidentiality. It is used most commonly in web browsers, but can be used with any protocol that uses TCP as the transport layer.

Secure Sockets Layer (SSL) is the predecessor of the TLS protocol. These names are often used interchangeably which can lead to some confusion:
 * A configuration that uses the ''SSL protocol'' (SSLv2/SSLv3) is insecure. The ''TLS protocol'' should be used instead.
 * X.509 certificates for authentication are sometimes also called ''SSL Certificates''.
 * Some applications (such as email) use a single port for both unencrypted and encrypted sessions. To change from unencrypted to encrypted, ''(START)TLS'' is used. When a single port directly uses the TLS protocol, it is often referred to as ''SSL''.
 * For historical reasons, software (Wireshark included) refer to SSL or SSL/TLS while it actually means the TLS protocol since that is nowadays what everyone uses.

== Protocol dependencies ==
 * [[TCP]]: Typically, TLS uses [[TCP]] as its transport protocol.

== TLS dissection in Wireshark ==
The TLS dissector is fully functional and even supports advanced features such as decryption of TLS if appropriate secrets are provided ([[#TLS_Decryption]]).

Since Wireshark 3.0, the TLS dissector has been renamed from SSL to TLS. Use of the `ssl` display filter will emit a warning.

== TLS Decryption ==
Wireshark supports TLS decryption when appropriate secrets are provided. The two available methods are:

 * Key log file using per-session secrets ([[#Using_the_.28Pre.29-Master-Secret]]).
 * Decryption using an RSA private key ([[#RSA_Keys]]).

A key log file is a universal mechanism that always enables decryption, even if a Diffie-Hellman (DH) key exchange is in use. The RSA private key only works in a limited number of cases.

The key log file is a text file generated by applications such as Firefox, Chrome and curl when the `SSLKEYLOGFILE` environment variable is set. To be precise, their underlying library (NSS, OpenSSL or boringssl) writes the required per-session secrets to a file. This file can subsequently be configured in Wireshark ([[#Using_the_.28Pre.29-Master-Secret]]).

The RSA private key file can only be used in the following circumstances:

 * The cipher suite selected by the server is not using (EC)DHE.
 * The protocol version is SSLv3, (D)TLS 1.0-1.2. It does ''not'' work with TLS 1.3.
 * The private key matches the ''server'' certificate. It does not work with the ''client'' certificate, nor the Certificate Authority (CA) certificate.
 * The session has not been resumed. The handshake must include the ClientKeyExchange handshake message.

The key log file is generally recommended since it works in all cases, but requires the continuous ability to export the secrets from either the client or server application.
The only advantage of the RSA private key is that it needs to be configured only once in Wireshark to enable decryption, subject to the above limitations.

== Preference Settings ==
Go to ''Edit'' -> ''Preferences''. Open the ''Protocols'' tree and select ''TLS''.
Alternatively, select a TLS packet in the packet list, right-click on the ''TLS'' layer in the packet details view and open the ''Protocol preferences'' menu.

The notable TLS protocol preferences are:

 * (Pre)-Master-Secret log filename (tls.keylog_file): path to read the TLS key log file for decryption.
 * RSA keys list: opens a dialog to configure RSA private keys for decryption. Deprecated in favor of the ''Preferences'' -> ''RSA Keys'' dialog.
 * Pre-Shared-Key: used to configure the decryption key for PSK cipher suites. Not generally used.
 * TLS debug file (tls.debug_logfile): path to write internal details about the decryption process. Will contain the results of decryption and the keys that were used in this process. This can be used to diagnose why decryption fails.

The following TCP protocol preferences are also required to enable TLS decryption:

 * Allow subdissector to reassemble TCP streams. Enabled by default.
 * Reassemble out-of-order segments (since Wireshark 3.0, disabled by default).

Starting with Wireshark 3.0, a new RSA Keys dialog can be found at ''Edit'' -> ''Preferences'' -> ''RSA Keys''. In this dialog, use the ''Add new keyfile...'' button to select a file. You will be prompted for a password if necessary. The ''Add new token...'' button can be used to add keys from a HSM which might require using ''Add new provider...'' to select select a DLL/.so file, and additional vendor-specific configuration.

The RSA key file can either be a ''PEM'' format private key or a PKCS#12 keystore (typically a file with a .pfx or .p12 extension). The PKCS#12 key is a binary file, but the PEM format is a text file which looks like this:
 {{{
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDReQzlKVeAK8b5
TRcRBhSi9IYwHX8Nqc8K4HeDRvN7HiBQQP3bhUkVekdoXpRLYVuc7A8h1BLr93Qw
...
KOi8FZl+jhG+p8vtpK5ZAIyp
-----END PRIVATE KEY-----
}}}

The deprecated ''RSA keys list'' dialog may be removed at some point. To configure keys, use the ''RSA keys'' dialog instead. To change the protocol for decrypted network data, right-click on a TLS packet and use ''Decode As'' to change the ''Current'' protocol for the ''TLS port''. The ''IP address'' and ''Port'' fields are unused.

== Example capture file ==
See also [[SampleCaptures#SSL_with_decryption_keys]].

 * [[https://bugs.wireshark.org/bugzilla/attachment.cgi?id=11612|dump.pcapng]] TLSv1.2 capture with 73 cipher suites, you need this [[https://bugs.wireshark.org/bugzilla/attachment.cgi?id=11616|premaster.txt]] file for decrypting the traffic. (linked from https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=9144)
 * tls12-dsb.pcapng - TLS 1.2 trace with decryption keys embedded.
 * https://github.com/wireshark/wireshark/blob/master/test/captures - The test suite contains various TLS traces.

== Display Filter ==
A complete list of TLS display filter fields can be found in the [[https://www.wireshark.org/docs/dfref/t/tls.html|display filter reference]]

Show only the TLS based traffic:

 {{{tls}}}

== Capture Filter ==
You cannot directly filter TLS protocols while capturing. However, if you know the [[TCP]] port used (see above), you can filter on that one, for example using {{{tcp port 443}}}.

## Note: external documentations link to this section using #Using_the_.28Pre.29-Master-Secret
== Using the (Pre)-Master-Secret ==
The master secret enables TLS decryption in Wireshark and can be supplied via the Key Log File. The pre-master secret is the result from the key exchange and can be converted to a master secret by Wireshark. This pre-master secret can be obtained when a RSA private key is provided and a RSA key exchange is in use.

Step-by-step instructions to decrypt TLS traffic from Chrome or Firefox in Wireshark:

 1. Close the browser completely (check your task manager just to be sure).
 2. Set environment variable `SSLKEYLOGFILE` to the absolute path of a writable file.
 3. Start the browser.
 4. Verify that the location from step 2 is created.
 5. In Wireshark, go to ''Preferences'' -> ''Protocols'' -> ''TLS'', and change the ''(Pre)-Master-Secret log filename'' preference to the path from step 2.
 6. Start the Wireshark capture.
 7. Open a website, for example https://www.wireshark.org/
 8. Check that the decrypted data is visible. For example, using the `tls and (http or http2)` filter.

For Windows, an environment variable can be set globally as described in [[https://redflagsecurity.net/2019/03/10/decrypting-tls-wireshark/|this walkthrough]], but this is not recommended since it is easy to forget about and may be a security issue since it allows decryption of all your TLS traffic. A better way to set the environment variable is via a batch file. Create a file `start-fx.cmd` with:
 {{{
@echo off
set SSLKEYLOGFILE=%USERPROFILE%\Desktop\keylogfile.txt
open firefox
}}}

Change the `SSLKEYLOGFILE` path as needed, and replace `firefox` with `chrome` for Google Chrome. This mechanism currently (2019) does not work for Safari, Microsoft Edge, and others since their TLS libraries (Microsoft SChannel/Apple SecureTransport) do not support this mechanism. This mechanism works for applications other than web browsers as well, but it dependent on the TLS library used by the application.

Examples of other applications:
 * Applications using OpenSSL could use a [[https://security.stackexchange.com/q/80158/2630|GDB or a LD_PRELOAD trick]] to extract the secrets. This includes Python.
 * For Java programs, pre-master secrets can be extracted from the TLS debug log preference, or output directly in the format Wireshark requires via [[http://jsslkeylog.sourceforge.net/|this agent]].
 * Python scripts can be [[https://github.com/joernheissler/SslMasterKey|edited]] to dump keys as well.

For a survey of supported TLS applications and libraries, see also [[https://lekensteyn.nl/files/wireshark-ssl-tls-decryption-secrets-sharkfest18eu.pdf#page=19|page 19 of Peter Wu's SSL/TLS Decryption SharkFest'18 EU presentation]].

=== Embedding decryption secrets in a pcapng file ===
Since Wireshark 3.0 you can embed the TLS key log file in a '''pcapng''' file. This makes it much easier to distribute capture files with decryption secrets, and makes switching between capture files easier since the TLS protocol preference does not have to be updated. To add the contents of key log file `keys.txt` to capture file `in.pcap` and write the result to `out-dsb.pcapng`:

 {{{editcap --inject-secrets tls,keys.txt in.pcap out-dsb.pcapng}}}

The ''dsb'' suffix stands for Decryption Secrets Block (DSB) and is part of the [[https://github.com/pcapng/pcapng|pcapng specification]].

A key log file might contain keys that are not related to a capture file. To ensure that unnecessary keys are not leaked, you can use the `inject-tls-secrets.sh` script from https://gist.github.com/Lekensteyn/f64ba6d6d2c6229d6ec444647979ea24 to filter the key log file and add the required secrets to a capture file. This has been tested with Linux and macOS, but not Windows. Example:

 {{{
git clone https://gist.github.com/Lekensteyn/f64ba6d6d2c6229d6ec444647979ea24 ~/its
~/its/inject-tls-secrets.sh keys.txt some.pcap
}}}

== See also ==
Some other protocols are derived from TLS. This includes:
 * [[DTLS]] is based on the TLS standard and runs on top of [[UDP]] as a transport protocol.
 * [[QUIC]] is an in-development protocol that uses TLS for its encryption, Wireshark's status can be tracked at [[https://github.com/quicwg/base-drafts/wiki/Tools#wireshark|]].

== External links ==
 * https://en.wikipedia.org/wiki/Transport_Layer_Security Wikipedia article for TLS
 * https://sharkfesteurope.wireshark.org/assets/presentations16eu/07.pdf SharkFest'16 EU presentation by Sake Blok on troubleshooting SSL with Wireshark/Tshark (or watch the video of the presentation at https://youtu.be/oDaDY9QCnXk)
 * https://lekensteyn.nl/files/wireshark-ssl-tls-decryption-secrets-sharkfest18eu.pdf SharkFest'18 EU presentation by Peter Wu on TLS decryption (video for an earlier talk in Asia at https://youtu.be/bwJEBwgoeBg)
 * https://lekensteyn.nl/files/wireshark-tls-debugging-sharkfest19us.pdf SharkFest'19 US presentation by Peter Wu describing TLS decryption and use of embedded decryption secrets (https://youtu.be/Ha4SLHceF6w).
 * [[https://security.stackexchange.com/questions/20803/how-does-ssl-tls-work|How does SSL/TLS work? - Information Security Stack Exchange]]
 * [[https://blog.cloudflare.com/keyless-ssl-the-nitty-gritty-technical-details/|Keyless SSL: The Nitty Gritty Technical Details]] with a good introduction in TLS

----
 . CategoryHowTo

Transport Layer Security (TLS)

Transport Layer Security (TLS) provides security in the communication between two hosts. It provides integrity, authentication and confidentiality. It is used most commonly in web browsers, but can be used with any protocol that uses TCP as the transport layer.

Secure Sockets Layer (SSL) is the predecessor of the TLS protocol. These names are often used interchangeably which can lead to some confusion:

  • A configuration that uses the SSL protocol (SSLv2/SSLv3) is insecure. The TLS protocol should be used instead.

  • X.509 certificates for authentication are sometimes also called SSL Certificates.

  • Some applications (such as email) use a single port for both unencrypted and encrypted sessions. To change from unencrypted to encrypted, (START)TLS is used. When a single port directly uses the TLS protocol, it is often referred to as SSL.

  • For historical reasons, software (Wireshark included) refer to SSL or SSL/TLS while it actually means the TLS protocol since that is nowadays what everyone uses.

Protocol dependencies

  • TCP: Typically, TLS uses TCP as its transport protocol.

TLS dissection in Wireshark

The TLS dissector is fully functional and even supports advanced features such as decryption of TLS if appropriate secrets are provided (#TLS_Decryption).

Since Wireshark 3.0, the TLS dissector has been renamed from SSL to TLS. Use of the ssl display filter will emit a warning.

TLS Decryption

Wireshark supports TLS decryption when appropriate secrets are provided. The two available methods are:

A key log file is a universal mechanism that always enables decryption, even if a Diffie-Hellman (DH) key exchange is in use. The RSA private key only works in a limited number of cases.

The key log file is a text file generated by applications such as Firefox, Chrome and curl when the SSLKEYLOGFILE environment variable is set. To be precise, their underlying library (NSS, OpenSSL or boringssl) writes the required per-session secrets to a file. This file can subsequently be configured in Wireshark (#Using_the_.28Pre.29-Master-Secret).

The RSA private key file can only be used in the following circumstances:

  • The cipher suite selected by the server is not using (EC)DHE.
  • The protocol version is SSLv3, (D)TLS 1.0-1.2. It does not work with TLS 1.3.

  • The private key matches the server certificate. It does not work with the client certificate, nor the Certificate Authority (CA) certificate.

  • The session has not been resumed. The handshake must include the ClientKeyExchange handshake message.

The key log file is generally recommended since it works in all cases, but requires the continuous ability to export the secrets from either the client or server application. The only advantage of the RSA private key is that it needs to be configured only once in Wireshark to enable decryption, subject to the above limitations.

Preference Settings

Go to Edit -> Preferences. Open the Protocols tree and select TLS. Alternatively, select a TLS packet in the packet list, right-click on the TLS layer in the packet details view and open the Protocol preferences menu.

The notable TLS protocol preferences are:

  • (Pre)-Master-Secret log filename (tls.keylog_file): path to read the TLS key log file for decryption.
  • RSA keys list: opens a dialog to configure RSA private keys for decryption. Deprecated in favor of the Preferences -> RSA Keys dialog.

  • Pre-Shared-Key: used to configure the decryption key for PSK cipher suites. Not generally used.
  • TLS debug file (tls.debug_logfile): path to write internal details about the decryption process. Will contain the results of decryption and the keys that were used in this process. This can be used to diagnose why decryption fails.

The following TCP protocol preferences are also required to enable TLS decryption:

  • Allow subdissector to reassemble TCP streams. Enabled by default.
  • Reassemble out-of-order segments (since Wireshark 3.0, disabled by default).

Starting with Wireshark 3.0, a new RSA Keys dialog can be found at Edit -> Preferences -> RSA Keys. In this dialog, use the Add new keyfile... button to select a file. You will be prompted for a password if necessary. The Add new token... button can be used to add keys from a HSM which might require using Add new provider... to select select a DLL/.so file, and additional vendor-specific configuration.

The RSA key file can either be a PEM format private key or a PKCS#12 keystore (typically a file with a .pfx or .p12 extension). The PKCS#12 key is a binary file, but the PEM format is a text file which looks like this:

  • -----BEGIN PRIVATE KEY-----
    MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDReQzlKVeAK8b5
    TRcRBhSi9IYwHX8Nqc8K4HeDRvN7HiBQQP3bhUkVekdoXpRLYVuc7A8h1BLr93Qw
    ...
    KOi8FZl+jhG+p8vtpK5ZAIyp
    -----END PRIVATE KEY-----

The deprecated RSA keys list dialog may be removed at some point. To configure keys, use the RSA keys dialog instead. To change the protocol for decrypted network data, right-click on a TLS packet and use Decode As to change the Current protocol for the TLS port. The IP address and Port fields are unused.

Example capture file

See also SampleCaptures#SSL_with_decryption_keys.

Display Filter

A complete list of TLS display filter fields can be found in the display filter reference

Show only the TLS based traffic:

  • tls

Capture Filter

You cannot directly filter TLS protocols while capturing. However, if you know the TCP port used (see above), you can filter on that one, for example using tcp port 443.

Using the (Pre)-Master-Secret

The master secret enables TLS decryption in Wireshark and can be supplied via the Key Log File. The pre-master secret is the result from the key exchange and can be converted to a master secret by Wireshark. This pre-master secret can be obtained when a RSA private key is provided and a RSA key exchange is in use.

Step-by-step instructions to decrypt TLS traffic from Chrome or Firefox in Wireshark:

  1. Close the browser completely (check your task manager just to be sure).
  2. Set environment variable SSLKEYLOGFILE to the absolute path of a writable file.

  3. Start the browser.
  4. Verify that the location from step 2 is created.
  5. In Wireshark, go to Preferences -> Protocols -> TLS, and change the (Pre)-Master-Secret log filename preference to the path from step 2.

  6. Start the Wireshark capture.
  7. Open a website, for example https://www.wireshark.org/

  8. Check that the decrypted data is visible. For example, using the tls and (http or http2) filter.

For Windows, an environment variable can be set globally as described in this walkthrough, but this is not recommended since it is easy to forget about and may be a security issue since it allows decryption of all your TLS traffic. A better way to set the environment variable is via a batch file. Create a file start-fx.cmd with:

  • @echo off
    set SSLKEYLOGFILE=%USERPROFILE%\Desktop\keylogfile.txt
    open firefox

Change the SSLKEYLOGFILE path as needed, and replace firefox with chrome for Google Chrome. This mechanism currently (2019) does not work for Safari, Microsoft Edge, and others since their TLS libraries (Microsoft SChannel/Apple SecureTransport) do not support this mechanism. This mechanism works for applications other than web browsers as well, but it dependent on the TLS library used by the application.

Examples of other applications:

  • Applications using OpenSSL could use a GDB or a LD_PRELOAD trick to extract the secrets. This includes Python.

  • For Java programs, pre-master secrets can be extracted from the TLS debug log preference, or output directly in the format Wireshark requires via this agent.

  • Python scripts can be edited to dump keys as well.

For a survey of supported TLS applications and libraries, see also page 19 of Peter Wu's SSL/TLS Decryption SharkFest'18 EU presentation.

Embedding decryption secrets in a pcapng file

Since Wireshark 3.0 you can embed the TLS key log file in a pcapng file. This makes it much easier to distribute capture files with decryption secrets, and makes switching between capture files easier since the TLS protocol preference does not have to be updated. To add the contents of key log file keys.txt to capture file in.pcap and write the result to out-dsb.pcapng:

  • editcap --inject-secrets tls,keys.txt in.pcap out-dsb.pcapng

The dsb suffix stands for Decryption Secrets Block (DSB) and is part of the pcapng specification.

A key log file might contain keys that are not related to a capture file. To ensure that unnecessary keys are not leaked, you can use the inject-tls-secrets.sh script from https://gist.github.com/Lekensteyn/f64ba6d6d2c6229d6ec444647979ea24 to filter the key log file and add the required secrets to a capture file. This has been tested with Linux and macOS, but not Windows. Example:

  • git clone https://gist.github.com/Lekensteyn/f64ba6d6d2c6229d6ec444647979ea24 ~/its
    ~/its/inject-tls-secrets.sh keys.txt some.pcap

See also

Some other protocols are derived from TLS. This includes:


TLS (last edited 2019-06-13 23:08:25 by PeterWu)