PcapNg

The PCAP Next Generation Dump File Format (or pcapng for short) is an attempt to overcome the limitations of the currently widely used (but limited) libpcap format.

The pcapng file format specification is still work in progress, see:

• http://www.winpcap.org/ntar/draft/PCAP-DumpFileFormat.html

• (2012-05) New pcap-ng mailing list (and archives)

• (2012-07) Pcap-ng specification anonymous subversion repository

• Some extensions to the PCAP-NG format from the Linux-Sensor project.

Example pcapng Capture File

Included below are some example pcapng files. While the block headers for these files appears to be formatted correctly, the payloads contained within these may NOT necessarily be well-formed (this is especially true of the "http.bigendian.ntar" file). Examples of pcapng files are currently very scarce. A few of these files were found elsewhere on the net (most likely the at the winpcap.org site referenced above). The "http.littlendian.ntar" file was found as an attachment(under a different name) in an email regarding pcapng. The http.bigendian.ntar is a version of this same file that was edited by hand to illustrate how a big-endian and little-endian machine might write the same data.

Please note: The recommended file name extension for pcapng files is .pcapng

• http.littleendian.ntar (SHB, IDB, 43 * PB)

• http.bigendian.ntar (same as http.littleendian but buggy! mixture of big and little endian byte swapping -- ULFL, There were apparently mistakes made when hand-editing this file. It is not a valid pcapng file and Wireshark won't load it. -- CM)

• test001.ntar (SHB)

• test002.ntar (SHB, IDB)

• test003.ntar (SHB, IDB)

• test004.ntar (SHB)

• test005.ntar (SHB, IDB)

• test006.ntar (SHB, IDB, 1 * PB)

• test007.ntar (SHB, IDB, 1 * SPB) NOTE: Wireshark refuses to load this file. It appears to be corrupt. -- CM

• test008.ntar (SHB, IDB, SHB, IDB) NOTE: Wireshark refuses to load this file since it contains multiple SHB's, which is not yet supported. -- CM

• test009.ntar (SHB, IDB)

• test010.ntar (SHB, IDB, 1000 * PB) NOTE: Wireshark refuses to load this file. It appears to be corrupt. -- CM

• icmp2.ntar (SHB, IDB, 8 * PB). This file contains 8 ICMP packets (a ping operation). Timestamps have nanosecond precision and the packets contain a valid FCS. The correct dissection of packets and timestamps (relative) is the screenshot .

ntartest - a simplistic standalone pcapng (ntar) file reader

Included below is the C source code to a very simplistic program to read and dump header information about a pcapng (a.k.a. ntar) file. This program has been successfully compiled using gcc and used on several different types of systems including Linux, cygwin and Solaris 9.

• ntartest.c

To compile this program use the command:

• # gcc -o ntartest ntartest.c

To use the program provide a pcapng file as the first argument to the program:

• # ntartest http.littleendian.pcapng

File

File 5.08+ should be able to identify pcap-ng files (ml thread).

NTAR

The Network Trace Archival and Retrieval library is able to read and write pcap-ng files. Caveats: i) frozen since 2007, ii) no plugins for EPB, ISB, and NRB blocks. Mailing list archives.

• Upstream has a more recent version of the library that includes support for EPB blocks. Unfortunately this version is not available for download on the website.

• (2012-07) NTAR library anonymous subversion repository (ml announcement)

Libpcap

As of the 1.1.0 release, libpcap adds limited support for reading pcap-ng files, see CHANGES.

Wireshark

As of Wireshark 1.2.0, pcapng files can be read and written, and live captures can be done in pcapng format as well as pcap format. There were a number of bugs in 1.2.0's support that were fixed in 1.2.1.

The current limitations for pcapng format are:

• Only a single section
• Only blocks SHB, IDB, PB, EPB, SPB (others will be ignored)
• Lots of Options not implemented
• Writing files is mostly untested
• When merging files, mergecap doesn't retain each IDB's snaplen
• mergecap won't merge pcapng files with different encapsulations and intermixed timestamps

tshark 1.6 should be able to read and write NRB blocks (command line options: -W n and -H hosts_file).

dumpcap 1.4/1.6/1.7 uses the libpcap_write_.*_block functions from pcapio.c to write the pcapng blocks. It currently writes SHB, IDB, EPB and ISB blocks.

• [v1.6.x] dumpcap -i eth0 -n -w file.pcapng
• [v1.7.x] dumpcap -i eth0 -w file.pcapng
  • Capture file will have the following blocks: SHB, IDB, EPB, EPB, ..., ISB.
• [v1.7.x] dumpcap -i eth0 -i eth1 -i eth2 -w file.pcapng
  • Capture file will have the following pcap-ng blocks: SHB, IDB, IDB, IDB, EPB, EPB, ..., ISB, ISB, ISB.
  • You can also supply a different capture filter for each interface: dumpcap -i eth0 -f udp -i eth1 -i eth2 -f tcp -w file.pcapng

  • TODO: the "-i any" behavior needs to be specified (see bug #3096 comments 46+)

• [v1.7.1 rev >= 41137 and rev < 41160] dumpcap -i eth0 -i eth1 -i eth2 -w file.pcapng

  • Capture file will have the following pcap-ng blocks: SHB, IDB, ISB, IDB, ISB, IDB, ISB, EPB, EPB, ..., ISB, ISB, ISB.

• [v1.7.1 rev >= 41160] dumpcap -i eth0 -i eth1 -i eth2 -w file.pcapng

  • Stopped writing ISB blocks immediately after IDB blocks
  • Capture file will have the following pcap-ng blocks: SHB, IDB, IDB, IDB, EPB, EPB, ..., ISB, ISB, ISB.

Options working

These options are working:

• opt_endofopt
• if_tsresol (was if_tsaccur) - only values 6 (microseconds - the default) and 9 (nanoseconds) currently supported
• if_fcslen

These options will only be displayed on the console (missing API from wiretap to Wireshark):

• opt_comment - work in progress: see ticket 3096

• shb_hardware
• shb_os
• shb_userappl
• if_name
• if_description
• if_speed
• pack_flags
• epb_flags

Options status

• dumpcap 1.4/1.6/1.7 (rev < 41026) only supports (writes) the following options:

  • SHB option 04 - shb_userappl
  • IDB option 02 - if_name
  • IDB option 11 - if_filter (if capture filter specified on the command line)
  • ISB option 04 - isb_ifrecv
  • ISB option 05 - isb_ifdrop

• dumpcap 1.7.1 rev 41026 added support for the following three SHB options:

  • SHB option 01 - opt_comment ( currently NULL as of rev 41046 )
  • SHB option 02 - shb_hardware ( currently NULL as of rev 41046 )
  • SHB option 03 - shb_os ( value from get_runtime_version_info() )

• Pcapng files created with dumpcap 1.7.1 rev 41026+ can't be opened with previous versions of wireshark. The problem is that the Wiretap API code in previous versions can't handle SHB options longer than 100 bytes (and the current shb_os is longer than that). A patch has already been committed for version 1.6.6 (rev 41037) and is queued to be backported for version 1.4.12. Additional information in bug 6846.

  • Rev 41086 - minimizes the exposure of bug #6846 by using get_os_version_info() instead of get_runtime_version_info() to fill the value of shb_os (and thus keeping its length below 100 bytes).

• Rev 41082 - Improved Wiretap support of IDB blocks and IDB options.

  • Rev 41097 - IDB, EPB options longer than 100 bytes support in branch 1.6

• rev 41136, rev 41137 - dumpcap: ISB options

• rev 41137 - wiretap: read all ISB options and write ISB blocks

  • It also started writing an ISB block immediately after each IDB block
  • The capture file created by 'dumpcap -i eth0 -i eth1 -w file.pcapng' will now have the following pcap-ng blocks: SHB, IDB, ISB, IDB, ISB, EPB, EPB, ..., ISB, ISB.

  • rev 41160 - stopped writing ISB blocks immediately after IDB blocks

• rev 41143, rev 41152 - wiretap: pcapng sanity checks

• rev 41153 - more backports to branch 1.6

• rev 41339, rev 41341, rev 41351, rev 41352 - if_filter: first byte specifies filter type (0 = libpcap filter string, 1 = libpcap filter bytecode, ...)

• rev 41455 - time resolution: if_tsresol = 6 (usec) or 9 (nsec)

• rev 41682 - dumpcap's pcapio no longer writes nul-terminated strings (block options values)

• rev 41683 - Wiretap API: write ISB blocks

Future

So after implementing the "skeleton" to read the pcapng file, there's a lot more to do:

• enhance/change/exchange current wiretap API (in part or in whole?)
• hierarchically store information in Wireshark
• somehow display these additional infos in the Wireshark GUI

Wishlist

List here things users would like Wireshark to support in PCAP-NG (this was recorded at the Developer's Roundtable at Sharkfest '10).

Existing pcap-ng features:

• packet dropped count
• annotations (comments)
• local IP address

• interface & direction

• hostname <-> IP address database

Existing, but insufficiently specified, pcap-ng features:

• capture filter used
• timezone

Features not yet in pcap-ng:

• GPS location (of capturing computer) -- see Kismet
• Wireless spectrum information / physical layer meta-data (other than what's already carried in headers such as the radiotap header for 802.11)
• Recording the original frame numbers of packets from the original packet capture file when saving packets to a new file in order to make cross-referencing them easier.
• SSL session keys (when decrypting SSL with a private key, save the Master Secrets of the SSL sessions, so the tracefile can be shown decrypted without access to the private key)

FAQ

File format conversion

From .pcap to .pcapng

Using editcap or tshark:

• editcap -F pcapng file.pcap file.pcapng
• tshark -F pcapng -r file.pcap -w file.pcapng

From .pcapng to .pcap

Using editcap:

• editcap -F libpcap -T ether file.pcapng file.pcap

Note: you need to specify an encapsulation type (-T <encapsulation type>) (1)

Using tcpdump/libpcap (4.1.0+/1.1.0+) (1,2):

• tcpdump -r file.pcapng -w file.pcap

Merging pcapng files

Again, you need to manually specify an encapsulation type. For example:

• mergecap -T ether -w new.pcapng file1.pcapng file2.pcapng ...

Note: This operation can't still be done via Wireshark's File/Merge ... (as of rev 40759).

Create pcap files instead of pcapng files

You may use the dumpcap's -P command line option if you need to create capture files in the old libpcap format (instead of the new default pcapng):

• dumpcap -P -i eth0 -w file.pcap