C-DNS: A DNS Packet Capture Format
draft-ietf-dnsop-dns-capture-format-01
The information below is for an old version of the document.
| Document | Type |
This is an older version of an Internet-Draft that was ultimately published as RFC 8618.
|
|
|---|---|---|---|
| Authors | John Dickinson , Jim Hague , Sara Dickinson , Terry Manderson , John Bond | ||
| Last updated | 2017-02-21 | ||
| Replaces | draft-dickinson-dnsop-dns-capture-format | ||
| RFC stream | Internet Engineering Task Force (IETF) | ||
| Formats | |||
| Reviews | |||
| Additional resources | Mailing list discussion | ||
| Stream | WG state | WG Document | |
| Document shepherd | (None) | ||
| IESG | IESG state | Became RFC 8618 (Proposed Standard) | |
| Consensus boilerplate | Yes | ||
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-ietf-dnsop-dns-capture-format-01
+------------------+----------+-------------------------------------+ | Field | Type | Description | +------------------+----------+-------------------------------------+ | ae-type | Unsigned | The type of event. The following | | | | events types are currently defined: | | | | 0. TCP reset. | | | | 1. ICMP time exceeded. | | | | 2. ICMP destination unreachable. | | | | 3. ICMPv6 time exceeded. | | | | 4. ICMPv6 destination unreachable. | | | | 5. ICMPv6 packet too big. | | | | | | ae-code | Unsigned | A code relating to the event. | | | | Optional. | | | | | | ae-address-index | Unsigned | The index in the IP address table | | | | of the client address. | | | | | | ae-count | Unsigned | The number of occurrences of this | | | | event during the block collection | | | | period. | +------------------+----------+-------------------------------------+ 7.20. Malformed packet records This optional table records the original wire format content of malformed packets (see Section 8). +----------------+--------+-----------------------------------------+ | Field | Type | Description | +----------------+--------+-----------------------------------------+ | time-useconds | A | Packet timestamp as an offset in | | | | microseconds from the Block preamble | | | | Timestamp. | | | | | | time-pseconds | A | Picosecond component of the timestamp. | | | | Optional. | | | | | | packet-content | Byte | The packet content in wire format. | | | string | | +----------------+--------+-----------------------------------------+ 8. Malformed Packets In the context of generating a C-DNS file it is assumed that only those packets which can be parsed to produce a well-formed DNS message are stored in the C-DNS format. This means as a minimum: Dickinson, et al. Expires August 25, 2017 [Page 23] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 o The packet has a well-formed 12 bytes DNS Header o The section counts are consistent with the section contents o All of the resource records can be parsed In principle, packets that do not meet these criteria could be classified into two categories: o Partially malformed: those packets which can be decoded sufficiently to extract * a DNS header (and therefore a DNS transaction ID) * a QDCOUNT * the first question in the QUESTION section if QDCOUNT is greater than 0 but suffer other issues while parsing. This is the minimum information required to attempt packet matching as described in Section 10.1 o Completely malformed: those packets that cannot be decoded to this extent. An open question is whether there is value in attempting to process partially malformed packets in an analogous manner to well formed packets in terms of attempting to match them with the corresponding query or response. This could be done by creating 'placeholder' records during packet matching with just the information extracted as above. If the packet were then matched the resulting C-DNS Q/R data item would include a flag to indicate a malformed record (in addition to capturing the wire format of the packet). An advantage of this would be that it would result in more meaningful statistics about matched packets because, for example, some partially malformed queries could be matched to responses. However it would only apply to those queries where the first QUESTION is well formed. It could also simplify the downstream analysis of C-DNS files and the reconstruction of packet streams from C-DNS. A disadvantage is that this adds complexity to the packet matching and data representation, could potentially lead to false matches and some additional statistics would be required (e.g. counts for matched-partially-malformed, unmatched-partially-malformed, completely-malformed). Dickinson, et al. Expires August 25, 2017 [Page 24] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 9. C-DNS to PCAP It is possible to re-construct PCAP files from the C-DNS format in a lossy fashion. Some of the issues with reconstructing both the DNS payload and the full packet stream are outlined here. The reconstruction depends on whether or not all the optional sections of both the query and response were captured in the C-DNS file. Clearly, if they were not all captured, the reconstruction will be imperfect. Even if all sections of the response were captured, one cannot reconstruct the DNS response payload exactly due to the fact that some DNS names in the message on the wire may have been compressed. Section 9.1 discusses this is more detail. Some transport information is not captured in the C-DNS format. For example, the following aspects of the original packet stream cannot be re-constructed from the C-DNS format: o IP fragmentation o TCP stream information: * Multiple DNS messages may have been sent in a single TCP segment * A DNS payload may have be split across multiple TCP segments * Multiple DNS messages may have be sent on a single TCP session o Malformed DNS messages if the wire format is not recorded o Any Non-DNS messages that were in the original packet stream e.g. ICMP Simple assumptions can be made on the reconstruction: fragmented and DNS-over-TCP messages can be reconstructed into single packets and a single TCP session can be constructed for each TCP packet. Additionally, if malformed packets and Non-DNS packets are captured separately, they can be merged with packet captures reconstructed from C-DNS to produce a more complete packet stream. Dickinson, et al. Expires August 25, 2017 [Page 25] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 9.1. Name Compression All the names stored in the C-DNS format are full domain names; no DNS style name compression is used on the individual names within the format. Therefore when reconstructing a packet, name compression must be used in order to reproduce the on the wire representation of the packet. [RFC1035] name compression works by substituting trailing sections of a name with a reference back to the occurrence of those sections earlier in the packet. Not all name server software uses the same algorithm when compressing domain names within the responses. Some attempt maximum recompression at the expense of runtime resources, others use heuristics to balance compression and speed and others use different rules for what is a valid compression target. This means that responses to the same question from different name server software which match in terms of DNS payload content (header, counts, RRs with name compression removed) do not necessarily match byte-for-byte on the wire. Therefore, it is not possible to ensure that the DNS response payload is reconstructed byte-for-byte from C-DNS data. However, it can at least, in principle, be reconstructed to have the correct payload length (since the original response length is captured) if there is enough knowledge of the commonly implemented name compression algorithms. For example, a simplistic approach would be to try each algorithm in turn to see if it reproduces the original length, stopping at the first match. This would not guarantee the correct algorithm has been used as it is possible to match the length whilst still not matching the on the wire bytes but, without further information added to the C-DNS data, this is the best that can be achieved. Appendix B presents an example of two different compression algorithms used by well-known name server software. 10. Data Collection This section describes a non-normative proposed algorithm for the processing of a captured stream of DNS queries and responses and matching queries/responses where possible. For the purposes of this discussion, it is assumed that the input has been pre-processed such that: 1. All IP fragmentation reassembly, TCP stream reassembly, and so on, has already been performed Dickinson, et al. Expires August 25, 2017 [Page 26] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 2. Each message is associated with transport metadata required to generate the Primary ID (see Section 10.2.1) 3. Each message has a well-formed DNS header of 12 bytes and (if present) the first RR in the Question section can be parsed to generate the Secondary ID (see below). As noted earlier, this requirement can result in a malformed query being removed in the pre-processing stage, but the correctly formed response with RCODE of FORMERR being present. DNS messages are processed in the order they are delivered to the application. It should be noted that packet capture libraries do not necessary provide packets in strict chronological order. TODO: Discuss the corner cases resulting from this in more detail. 10.1. Matching algorithm A schematic representation of the algorithm for matching Q/R data items is shown in the following diagram: Figure showing the packet matching algorithm format (PNG) [5] Figure showing the packet matching algorithm format (SVG) [6] Further details of the algorithm are given in the following sections. 10.2. Message identifiers 10.2.1. Primary ID (required) A Primary ID is constructed for each message. It is composed of the following data: 1. Source IP Address 2. Destination IP Address 3. Source Port 4. Destination Port 5. Transport 6. DNS Message ID Dickinson, et al. Expires August 25, 2017 [Page 27] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 10.2.2. Secondary ID (optional) If present, the first question in the Question section is used as a secondary ID for each message. Note that there may be well formed DNS queries that have a QDCOUNT of 0, and some responses may have a QDCOUNT of 0 (for example, responses with RCODE=FORMERR or NOTIMP). In this case the secondary ID is not used in matching. 10.3. Algorithm Parameters 1. Query timeout 2. Skew timeout 10.4. Algorithm Requirements The algorithm is designed to handle the following input data: 1. Multiple queries with the same Primary ID (but different Secondary ID) arriving before any responses for these queries are seen. 2. Multiple queries with the same Primary and Secondary ID arriving before any responses for these queries are seen. 3. Queries for which no later response can be found within the specified timeout. 4. Responses for which no previous query can be found within the specified timeout. 10.5. Algorithm Limitations For cases 1 and 2 listed in the above requirements, it is not possible to unambiguously match queries with responses. This algorithm chooses to match to the earliest query with the correct Primary and Secondary ID. 10.6. Workspace A FIFO structure is used to hold the Q/R data items during processing. 10.7. Output The output is a list of Q/R data items. Both the Query and Response elements are optional in these items, therefore Q/R data items have one of three types of content: Dickinson, et al. Expires August 25, 2017 [Page 28] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 1. A matched pair of query and response messages 2. A query message with no response 3. A response message with no query The timestamp of a list item is that of the query for cases 1 and 2 and that of the response for case 3. 10.8. Post Processing When ending capture, all remaining entries in the Q/R data item FIFO should be treated as timed out queries. 11. IANA Considerations None 12. Security Considerations Any control interface MUST perform authentication and encryption. Any data upload MUST be authenticated and encrypted. 13. Acknowledgements The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for many useful discussions on binary formats, compression and packet matching. Also Jan Vcelak and Wouter Wijngaards for discussions on name compression and Paul Hoffman for a detailed review of the document and the C-DNS CDDL. Thanks also to Robert Edmonds and Jerry Lundstroem for review. Also, Miek Gieben for mmark [7] 14. Changelog draft-ietf-dnsop-dns-capture-format-01 o Many editorial improvements by Paul Hoffman o Included discussion of malformed packet handling o Improved Appendix C on Comparison of Binary Formats o Now using C-DNS field names in the tables in section 8 Dickinson, et al. Expires August 25, 2017 [Page 29] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 o A handful of new fields included (CDDL updated) o Timestamps now include optional picoseconds o Added details of block statistics draft-ietf-dnsop-dns-capture-format-00 o Changed dnstap.io to dnstap.info o qr_data_format.png was cut off at the bottom o Update authors address o Improve wording in Abstract o Changed DNS-STAT to C-DNS in CDDL o Set the format version in the CDDL o Added a TODO: Add block statistics o Added a TODO: Add extend to support pico/nano. Also do this for Time offset and Response delay o Added a TODO: Need to develop optional representation of malformed packets within C-DNS and what this means for packet matching. This may influence which fields are optional in the rest of the representation. o Added section on design goals to Introduction o Added a TODO: Can Class be optimised? Should a class of IN be inferred if not present? draft-dickinson-dnsop-dns-capture-format-00 o Initial commit 15. References 15.1. Normative References [RFC1035] Mockapetris, P., "Domain names - implementation and specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, November 1987, <http://www.rfc-editor.org/info/rfc1035>. Dickinson, et al. Expires August 25, 2017 [Page 30] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <http://www.rfc-editor.org/info/rfc2119>. [RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049, October 2013, <http://www.rfc-editor.org/info/rfc7049>. 15.2. Informative References [ditl] DNS-OARC, "DITL", 2016, <https://www.dns- oarc.net/oarc/data/ditl>. [dnscap] DNS-OARC, "DNSCAP", 2016, <https://www.dns-oarc.net/tools/ dnscap>. [dnstap] dnstap.info, "dnstap", 2016, <http://dnstap.info/>. [dsc] Wessels, D. and J. Lundstrom, "DSC", 2016, <https://www.dns-oarc.net/tools/dsc>. [I-D.daley-dnsxml] Daley, J., Morris, S., and J. Dickinson, "dnsxml - A standard XML representation of DNS data", draft-daley- dnsxml-00 (work in progress), July 2013. [I-D.greevenbosch-appsawg-cbor-cddl] Vigano, C. and H. Birkholz, "CBOR data definition language (CDDL): a notational convention to express CBOR data structures", draft-greevenbosch-appsawg-cbor-cddl-09 (work in progress), September 2016. [I-D.hoffman-dns-in-json] Hoffman, P., "Representing DNS Messages in JSON", draft- hoffman-dns-in-json-10 (work in progress), October 2016. [packetq] .SE - The Internet Infrastructure Foundation, "PacketQ", 2014, <https://github.com/dotse/PacketQ>. [pcap] tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>. [pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G. Harris, "pcap-ng", 2016, <https://github.com/pcapng/ pcapng>. Dickinson, et al. Expires August 25, 2017 [Page 31] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 [RFC7159] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March 2014, <http://www.rfc-editor.org/info/rfc7159>. [rrtypes] IANA, "RR types", 2016, <http://www.iana.org/assignments/ dns-parameters/dns-parameters.xhtml#dns-parameters-4>. 15.3. URIs [1] https://github.com/dns-stats/draft-dns-capture- format/blob/master/cdns_format.png [2] https://github.com/dns-stats/draft-dns-capture- format/blob/master/cdns_format.svg [3] https://github.com/dns-stats/draft-dns-capture- format/blob/master/qr_data_format.png [4] https://github.com/dns-stats/draft-dns-capture- format/blob/master/qr_data_format.svg [5] https://github.com/dns-stats/draft-dns-capture- format/blob/master/packet_matching.png [6] https://github.com/dns-stats/draft-dns-capture- format/blob/master/packet_matching.svg [7] https://github.com/miekg/mmark [8] https://www.nlnetlabs.nl/projects/nsd/ [9] https://www.knot-dns.cz/ [10] https://avro.apache.org/ [11] https://developers.google.com/protocol-buffers/ [12] http://cbor.io [13] https://github.com/kubo/snzip [14] http://google.github.io/snappy/ [15] http://lz4.github.io/lz4/ [16] http://www.gzip.org/ [17] http://facebook.github.io/zstd/ Dickinson, et al. Expires August 25, 2017 [Page 32] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 [18] http://tukaani.org/xz/ [19] https://github.com/dns-stats/draft-dns-capture- format/blob/master/file-size-versus-block-size.png [20] https://github.com/dns-stats/draft-dns-capture- format/blob/master/file-size-versus-block-size.svg Appendix A. CDDL ; CDDL specification of the file format for C-DNS, ; which describes a collection of DNS messages and ; traffic meta-data. File = [ file-type-id : tstr, ; = "C-DNS" file-preamble : FilePreamble, file-blocks : [* Block], ] FilePreamble = { major-format-version => uint, ; = 1 minor-format-version => uint, ; = 0 ? private-version => uint, ? configuration => Configuration, ? generator-id => tstr, ? host-id => tstr, } major-format-version = 0 minor-format-version = 1 private-version = 2 configuration = 3 generator-id = 4 host-id = 5 Configuration = { ? query-timeout => uint, ? skew-timeout => uint, ? snaplen => uint, ? promisc => uint, ? interfaces => [* tstr], ? server-addresses => [* IPAddress], ; Hint for later analysis ? vlan-ids => [* uint], ? filter => tstr, ? query-options => QRCollectionSections, ? response-options => QRCollectionSections, ? accept-rr-types => [* uint], Dickinson, et al. Expires August 25, 2017 [Page 33] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 ? ignore-rr-types => [* uint], ? max-block-qr-items => uint, ? collect-malformed => uint, } QRCollectionSectionValues = &( question : 0, ; Second & subsequent question sections answer : 1, authority : 2, additional: 3, ) QRCollectionSections = uint .bits QRCollectionSectionValues query-timeout = 0 skew-timeout = 1 snaplen = 2 promisc = 3 interfaces = 4 vlan-ids = 5 filter = 6 query-options = 7 response-options = 8 accept-rr-types = 9 ignore-rr-types = 10 server-addresses = 11 max-block-qr-items = 12 collect-malformed = 13 Block = { preamble => BlockPreamble, ? statistics => BlockStatistics, tables => BlockTables, queries => [* QueryResponse], ? address-event-counts => [* AddressEventCount], ? malformed-packet-data => [* MalformedPacket], } preamble = 0 statistics = 1 tables = 2 queries = 3 address-event-counts = 4 malformed-packet-data = 5 BlockPreamble = { earliest-time => Timeval } Dickinson, et al. Expires August 25, 2017 [Page 34] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 earliest-time = 1 Timeval = [ seconds : uint, microseconds : uint, ? picoseconds : uint, ] BlockStatistics = { ? total-packets => uint, ? total-pairs => uint, ? unmatched-queries => uint, ? unmatched-responses => uint, ? malformed-packets => uint, } total-packets = 0 total-pairs = 1 unmatched-queries = 2 unmatched-responses = 3 malformed-packets = 4 BlockTables = { ip-address => [* IPAddress], classtype => [* ClassType], name-rdata => [* bstr], ; Holds both Name RDATA and RDATA query-sig => [* QuerySignature] ? qlist => [* QuestionList], ? qrr => [* Question], ? rrlist => [* RRList], ? rr => [* RR], } ip-address = 0 classtype = 1 name-rdata = 2 query-sig = 3 qlist = 4 qrr = 5 rrlist = 6 rr = 7 QueryResponse = { time-useconds => uint, ; Time offset from start of block ? time-pseconds => uint, ; in microseconds and picoseconds client-address-index => uint, client-port => uint, transaction-id => uint, Dickinson, et al. Expires August 25, 2017 [Page 35] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 query-signature-index => uint, ? client-hoplimit => uint, ? delay-useconds => int, ? delay-pseconds => int, ; Has same sign as delay-useconds ? query-name-index => uint, ? query-size => uint, ; DNS size of query ? response-size => uint, ; DNS size of response ? query-extended => QueryResponseExtended, ? response-extended => QueryResponseExtended, } time-useconds = 0 time-pseconds = 1 client-address-index = 2 client-port = 3 transaction-id = 4 query-signature-index = 5 client-hoplimit = 6 delay-useconds = 7 delay-pseconds = 8 query-name-index = 9 query-size = 10 response-size = 11 query-extended = 12 response-extended = 13 ClassType = { type => uint, class => uint, } type = 0 class = 1 DNSFlagValues = &( query-cd : 0, query-ad : 1, query-z : 2, query-ra : 3, query-rd : 4, query-tc : 5, query-aa : 6, query-d0 : 7, response-cd: 8, response-ad: 9, response-z : 10, response-ra: 11, response-rd: 12, Dickinson, et al. Expires August 25, 2017 [Page 36] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 response-tc: 13, response-aa: 14, ) DNSFlags = uint .bits DNSFlagValues QueryResponseFlagValues = &( has-query : 0, has-reponse : 1, query-has-question : 2, query-has-opt : 3, response-has-opt : 4, response-has-no-question: 5, ) QueryResponseFlags = uint .bits QueryResponseFlagValues TransportFlagValues = &( tcp : 0, ipv6 : 1, query-trailingdata: 2, ) TransportFlags = uint .bits TransportFlagValues QuerySignature = { server-address-index => uint, server-port => uint, transport-flags => TransportFlags, qr-sig-flags => QueryResponseFlags, ? query-opcode => uint, qr-dns-flags => DNSFlags, ? query-rcode => uint, ? query-classtype-index => uint, ? query-qd-count => uint, ? query-an-count => uint, ? query-ar-count => uint, ? query-ns-count => uint, ? edns-version => uint, ? udp-buf-size => uint, ? opt-rdata-index => uint, ? response-rcode => uint, } server-address-index = 0 server-port = 1 transport-flags = 2 qr-sig-flags = 3 query-opcode = 4 qr-dns-flags = 5 query-rcode = 6 Dickinson, et al. Expires August 25, 2017 [Page 37] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 query-classtype-index = 7 query-qd-count = 8 query-an-count = 9 query-ar-count = 10 query-ns-count = 11 edns-version = 12 udp-buf-size = 13 opt-rdata-index = 14 response-rcode = 15 QuestionList = [ * uint, ; Index of Question ] Question = { ; Second and subsequent questions name-index => uint, ; Index to a name in the name-rdata table classtype-index => uint, } name-index = 0 classtype-index = 1 RRList = [ * uint, ; Index of RR ] RR = { name-index => uint, ; Index to a name in the name-rdata table classtype-index => uint, ttl => uint, rdata-index => uint, ; Index to RDATA in the name-rdata table } ttl = 2 rdata-index = 3 QueryResponseExtended = { ? question-index => uint, ; Index of QuestionList ? answer-index => uint, ; Index of RRList ? authority-index => uint, ? additional-index => uint, } question-index = 0 answer-index = 1 authority-index = 2 additional-index = 3 Dickinson, et al. Expires August 25, 2017 [Page 38] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 AddressEventCount = { ae-type => &AddressEventType, ? ae-code => uint, ae-address-index => uint, ae-count => uint, } ae-type = 0 ae-code = 1 ae-address-index = 2 ae-count = 3 AddressEventType = ( tcp-reset : 0, icmp-time-exceeded : 1, icmp-dest-unreachable : 2, icmpv6-time-exceeded : 3, icmpv6-dest-unreachable: 4, icmpv6-packet-too-big : 5, ) MalformedPacket = { time-useconds => uint, ; Time offset from start of block ? time-pseconds => uint, ; in microseconds and picoseconds packet-content => bstr, ; Raw packet contents } time-useconds = 0 time-pseconds = 1 packet-content = 2 IPv4Address = bstr .size 4 IPv6Address = bstr .size 16 IPAddress = IPv4Address / IPv6Address Appendix B. DNS Name compression example The basic algorithm, which follows the guidance in [RFC1035], is simply to collect each name, and the offset in the packet at which it starts, during packet construction. As each name is added, it is offered to each of the collected names in order of collection, starting from the first name. If labels at the end of the name can be replaced with a reference back to part (or all) of the earlier name, and if the uncompressed part of the name is shorter than any compression already found, the earlier name is noted as the compression target for the name. Dickinson, et al. Expires August 25, 2017 [Page 39] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 The following tables illustrate the process. In an example packet, the first name is example.com. +---+-------------+--------------+--------------------+ | N | Name | Uncompressed | Compression Target | +---+-------------+--------------+--------------------+ | 1 | example.com | | | +---+-------------+--------------+--------------------+ The next name added is bar.com. This is matched against example.com. The com part of this can be used as a compression target, with the remaining uncompressed part of the name being bar. +---+-------------+--------------+--------------------+ | N | Name | Uncompressed | Compression Target | +---+-------------+--------------+--------------------+ | 1 | example.com | | | | 2 | bar.com | bar | 1 + offset to com | +---+-------------+--------------+--------------------+ The third name added is www.bar.com. This is first matched against example.com, and as before this is recorded as a compression target, with the remaining uncompressed part of the name being www.bar. It is then matched against the second name, which again can be a compression target. Because the remaining uncompressed part of the name is www, this is an improved compression, and so it is adopted. +---+-------------+--------------+--------------------+ | N | Name | Uncompressed | Compression Target | +---+-------------+--------------+--------------------+ | 1 | example.com | | | | 2 | bar.com | bar | 1 + offset to com | | 3 | www.bar.com | www | 2 | +---+-------------+--------------+--------------------+ As an optimization, if a name is already perfectly compressed (in other words, the uncompressed part of the name is empty), then no further names will be considered for compression. B.1. NSD compression algorithm Using the above basic algorithm the packet lengths of responses generated by NSD [8] can be matched almost exactly. At the time of writing, a tiny number (<.01%) of the reconstructed packets had incorrect lengths. Dickinson, et al. Expires August 25, 2017 [Page 40] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 B.2. Knot Authoritative compression algorithm The Knot Authoritative [9] name server uses different compression behavior, which is the result of internal optimization designed to balance runtime speed with compression size gains. In brief, and omitting complications, Knot Authoritative will only consider the QNAME and names in the immediately preceding RR section in an RRSET as compression targets. A set of smart heuristics as described below can be implemented to mimic this and while not perfect it produces output nearly, but not quite, as good a match as with NSD. The heuristics are: 1. A match is only perfect if the name is completely compressed AND the TYPE of the section in which the name occurs matches the TYPE of the name used as the compression target. 2. If the name occurs in RDATA: * If the compression target name is in a query, then only the first RR in an RRSET can use that name as a compression target. * The compression target name MUST be in RDATA. * The name section TYPE must match the compression target name section TYPE. * The compression target name MUST be in the immediately preceding RR in the RRSET. Using this algorithm less than 0.1% of the reconstructed packets had incorrect lengths. B.3. Observed differences In sample traffic collected on a root name server around 2-4% of responses generated by Knot had different packet lengths to those produced by NSD. Appendix C. Comparison of Binary Formats Several binary serialisation formats were considered, and for completeness were also compared to JSON. o Apache Avro [10]. Data is stored according to a pre-defined schema. The schema itself is always included in the data file. Dickinson, et al. Expires August 25, 2017 [Page 41] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 Data can therefore be stored untagged, for a smaller serialisation size, and be written and read by an Avro library. * At the time of writing, Avro libraries are available for C, C++, C#, Java, Python, Ruby and PHP. Optionally tools are available for C++, Java and C# to generate code for encoding and decoding. o Google Protocol Buffers [11]. Data is stored according to a pre- defined schema. The schema is used by a generator to generate code for encoding and decoding the data. Data can therefore be stored untagged, for a smaller serialisation size. The schema is not stored with the data, so unlike Avro cannot be read with a generic library. * Code must be generated for a particular data schema to to read and write data using that schema. At the time of writing, the Google code generator can currently generate code for encoding and decoding a schema for C++, Go, Java, Python, Ruby, C#, Objective-C, Javascript and PHP. o CBOR [12]. Defined in [RFC7049], this serialisation format is comparable to JSON but with a binary representation. It does not use a pre-defined schema, so data is always stored tagged. However, CBOR data schemas can be described using CDDL [I-D.greevenbosch-appsawg-cbor-cddl] and tools exist to verify data files conform to the schema. * CBOR is a simple format, and simple to implement. At the time of writing, the CBOR website lists implementations for 16 languages. Avro and Protocol Buffers both allow storage of untagged data, but because they rely on the data schema for this, their implementation is considerably more complex than CBOR. Using Avro or Protocol Buffers in an unsupported environment would require notably greater development effort compared to CBOR. A test program was written which reads input from a PCAP file and writes output using one of two basic structures; either a simple structure, where each query/response pair is represented in a single record entry, or the C-DNS block structure. The resulting output files were then compressed using a variety of common general-purpose lossless compression tools to explore the compressibility of the formats. The compression tools employed were: Dickinson, et al. Expires August 25, 2017 [Page 42] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 o snzip [13]. A command line compression tool based on the Google Snappy [14] library. o lz4 [15]. The command line compression tool from the reference C LZ4 implementation. o gzip [16]. The ubiquitous GNU zip tool. o zstd [17]. Compression using the Zstandard algorithm. o xz [18]. A popular compression tool noted for high compression. In all cases the compression tools were run using their default settings. Note that this draft does not mandate the use of compression, nor any particular compression scheme, but it anticipates that in practice output data will be subject to general-purpose compression, and so this should be taken into consideration. "test.pcap", a 662Mb capture of sample data from a root instance was used for the comparison. The following table shows the formatted size and size after compression (abbreviated to Comp. in the table headers), together with the task resident set size (RSS) and the user time taken by the compression. File sizes are in Mb, RSS in kb and user time in seconds. +-------------+-----------+-------+------------+-------+-----------+ | Format | File size | Comp. | Comp. size | RSS | User time | +-------------+-----------+-------+------------+-------+-----------+ | PCAP | 661.87 | snzip | 212.48 | 2696 | 1.26 | | | | lz4 | 181.58 | 6336 | 1.35 | | | | gzip | 153.46 | 1428 | 18.20 | | | | zstd | 87.07 | 3544 | 4.27 | | | | xz | 49.09 | 97416 | 160.79 | | | | | | | | | JSON simple | 4113.92 | snzip | 603.78 | 2656 | 5.72 | | | | lz4 | 386.42 | 5636 | 5.25 | | | | gzip | 271.11 | 1492 | 73.00 | | | | zstd | 133.43 | 3284 | 8.68 | | | | xz | 51.98 | 97412 | 600.74 | | | | | | | | | Avro simple | 640.45 | snzip | 148.98 | 2656 | 0.90 | | | | lz4 | 111.92 | 5828 | 0.99 | | | | gzip | 103.07 | 1540 | 11.52 | | | | zstd | 49.08 | 3524 | 2.50 | | | | xz | 22.87 | 97308 | 90.34 | | | | | | | | Dickinson, et al. Expires August 25, 2017 [Page 43] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 | CBOR simple | 764.82 | snzip | 164.57 | 2664 | 1.11 | | | | lz4 | 120.98 | 5892 | 1.13 | | | | gzip | 110.61 | 1428 | 12.88 | | | | zstd | 54.14 | 3224 | 2.77 | | | | xz | 23.43 | 97276 | 111.48 | | | | | | | | | PBuf simple | 749.51 | snzip | 167.16 | 2660 | 1.08 | | | | lz4 | 123.09 | 5824 | 1.14 | | | | gzip | 112.05 | 1424 | 12.75 | | | | zstd | 53.39 | 3388 | 2.76 | | | | xz | 23.99 | 97348 | 106.47 | | | | | | | | | JSON block | 519.77 | snzip | 106.12 | 2812 | 0.93 | | | | lz4 | 104.34 | 6080 | 0.97 | | | | gzip | 57.97 | 1604 | 12.70 | | | | zstd | 61.51 | 3396 | 3.45 | | | | xz | 27.67 | 97524 | 169.10 | | | | | | | | | Avro block | 60.45 | snzip | 48.38 | 2688 | 0.20 | | | | lz4 | 48.78 | 8540 | 0.22 | | | | gzip | 39.62 | 1576 | 2.92 | | | | zstd | 29.63 | 3612 | 1.25 | | | | xz | 18.28 | 97564 | 25.81 | | | | | | | | | CBOR block | 75.25 | snzip | 53.27 | 2684 | 0.24 | | | | lz4 | 51.88 | 8008 | 0.28 | | | | gzip | 41.17 | 1548 | 4.36 | | | | zstd | 30.61 | 3476 | 1.48 | | | | xz | 18.15 | 97556 | 38.78 | | | | | | | | | PBuf block | 67.98 | snzip | 51.10 | 2636 | 0.24 | | | | lz4 | 52.39 | 8304 | 0.24 | | | | gzip | 40.19 | 1520 | 3.63 | | | | zstd | 31.61 | 3576 | 1.40 | | | | xz | 17.94 | 97440 | 33.99 | +-------------+-----------+-------+------------+-------+-----------+ The above results are discussed in the following sections. C.1. Comparison with full PCAP files An important first consideration is whether moving away from PCAP offers significant benefits. The simple binary formats are typically larger than PCAP, even though they omit some information such as Ethernet MAC addresses. But not only do they require less CPU to compress than PCAP, the resulting compressed files are smaller than compressed PCAP. Dickinson, et al. Expires August 25, 2017 [Page 44] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 C.2. Simple versus block coding The intention of the block coding is to perform data de-duplication on query/response records within the block. The simple and block formats above store exactly the same information for each query/ response record. This information is parsed from the DNS traffic in the input PCAP file, and in all cases each field has an identifier and the field data is typed. The data de-duplication on the block formats show an order of magnitude reduction in the size of the format file size against the simple formats. As would be expected, the compression tools are able to find and exploit a lot of this duplication, but as the de- duplication process uses knowledge of DNS traffic, it is able to retain a size advantage. This advantage reduces as stronger compression is applied, as again would be expected, but even with the strongest compression applied the block formatted data remains around 75% of the size of the simple format and its compression requires roughly a third of the CPU time. C.3. Binary versus text formats Text data formats offer many advantages over binary formats, particularly in the areas of ad-hoc data inspection and extraction. It was therefore felt worthwhile to carry out a direct comparison, implementing JSON versions of the simple and block formats. Concentrating on JSON block format, the format files produced are a significant fraction of an order of magnitude larger than binary formats. The impact on file size after compression is as might be expected from that starting point; the stronger compression produces files that are 150% of the size of similarly compressed binary format, and require over 4x more CPU to compress. C.4. Performance Concentrating again on the block formats, all three produce format files that are close to an order of magnitude smaller that the original "test.pcap" file. CBOR produces the largest files and Avro the smallest, 20% smaller than CBOR. However, once compression is taken into account, the size difference narrows. At medium compression (with gzip), the size difference is 4%. Using strong compression (with xz) the difference reduces to 2%, with Avro the largest and Protocol Buffers the smallest, although CBOR and Protocol Buffers require slightly more compression CPU. Dickinson, et al. Expires August 25, 2017 [Page 45] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 The measurements presented above do not include data on the CPU required to generate the format files. Measurements indicate that writing Avro requires 10% more CPU than CBOR or Protocol Buffers. It appears, therefore, that Avro's advantage in compression CPU usage is probably offset by a larger CPU requirement in writing Avro. C.5. Conclusions The above assessments lead us to the choice of a binary format file using blocking. As noted previously, this draft anticipates that output data will be subject to compression. There is no compelling case for one particular binary serialisation format in terms of either final file size or machine resources consumed, so the choice must be largely based on other factors. CBOR was therefore chosen as the binary serialisation format for the reasons listed in Section 6. C.6. Block size choice Given the choice of a CBOR format using blocking, the question arises of what an appropriate default value for the maximum number of query/ response pairs in a block should be. This has two components; what is the impact on performance of using different block sizes in the format file, and what is the impact on the size of the format file before and after compression. The following table addresses the performance question, showing the impact on the performance of a C++ program converting "test.pcap" to C-DNS. File size is in Mb, resident set size (RSS) in kb. +------------+-----------+--------+-----------+ | Block size | File size | RSS | User time | +------------+-----------+--------+-----------+ | 1000 | 133.46 | 612.27 | 15.25 | | 5000 | 89.85 | 676.82 | 14.99 | | 10000 | 76.87 | 752.40 | 14.53 | | 20000 | 67.86 | 750.75 | 14.49 | | 40000 | 61.88 | 736.30 | 14.29 | | 80000 | 58.08 | 694.16 | 14.28 | | 160000 | 55.94 | 733.84 | 14.44 | | 320000 | 54.41 | 799.20 | 13.97 | +------------+-----------+--------+-----------+ Increasing block size, therefore, tends to increase maximum RSS a little, with no significant effect (if anything a small reduction) on CPU consumption. Dickinson, et al. Expires August 25, 2017 [Page 46] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 The following figure plots the effect of increasing block size on output file size for different compressions. Figure showing effect of block size on file size (PNG) [19] Figure showing effect of block size on file size (SVG) [20] From the above, there is obviously scope for tuning the default block size to the compression being employed, traffic characteristics, frequency of output file rollover etc. Using a strong compression, block sizes over 10,000 query/response pairs would seem to offer limited improvements. Authors' Addresses John Dickinson Sinodun IT Magdalen Centre Oxford Science Park Oxford OX4 4GA Email: jad@sinodun.com Jim Hague Sinodun IT Magdalen Centre Oxford Science Park Oxford OX4 4GA Email: jim@sinodun.com Sara Dickinson Sinodun IT Magdalen Centre Oxford Science Park Oxford OX4 4GA Email: sara@sinodun.com Dickinson, et al. Expires August 25, 2017 [Page 47] Internet-Draft C-DNS: A DNS Packet Capture Format February 2017 Terry Manderson ICANN 12025 Waterfront Drive Suite 300 Los Angeles CA 90094-2536 Email: terry.manderson@icann.org John Bond ICANN 12025 Waterfront Drive Suite 300 Los Angeles CA 90094-2536 Email: john.bond@icann.org Dickinson, et al. Expires August 25, 2017 [Page 48]