Specification of the IP Flow Information Export (IPFIX) File Format
draft-ietf-ipfix-file-05

[Ballot discuss]
Version -05 looks very good -- thanks for your work on this!

There's one minor bug that should be still fixed in Section 9.1: it
should use "id-data" (1.2.840.113549.1.7.1) content type instead of
"anyContentType" (which is meant for certificate constraints, not
actual CMS messages).

This would be easily fixed with an RFC Editor Note:

Section 9.1:
OLD:
  eContentType        id-ct-anyContentType, -- (1.2.840.113549.1.9.16.1.0)
NEW:
  eContentType        id-data, -- (1.2.840.113549.1.7.1)

Section 9.1.4:
OLD:
  The EncapsulatedContentInfo structure contains a content type
  identifier.  Since a detached signature is being created, it does not
  encapsulate the Internet-Draft.  The fields of
  EncapsulatedContentInfo are as follows:

  eContentType  is an object identifier that uniquely specifies the
      content type.  The content type associated with the plain text
      file MUST be id-ct-anyContentType (1.2.840.113549.1.9.16.1.0).
NEW:
  The EncapsulatedContentInfo structure contains a content type
  identifier.  Since a detached signature is being created, it does not
  encapsulate the IPFIX File.  The fields of
  EncapsulatedContentInfo are as follows:

  eContentType  is an object identifier that uniquely specifies the
      content type.  The content type associated with IPFIX File
      MUST be id-data (1.2.840.113549.1.7.1).

[Ballot discuss]
Updated discuss for draft-ietf-ipfix-file-04:

1) The security mechanism most likely to see some deployment is
probably TLS/DLTS. When you use TLS, it seems preserving this
information (the fact that this messages were sent by node A over TLS
to node B) would be important. Probably this would mean specifying
"exporterCertificate" and "collectorCertificate" information elements
in Section 8.1 to store the TLS endpoints certificates (probably with
"octetArray" type -- IPFIX doesn't need to look inside these).

2) As discussed with the authors in Stockholm, encapsulating IPFIX
files in CMS is likely to result in interop problems since not all
software for reading IPFIX files will implement CMS (and thus won't
be able to read signed files, even if they don't care about the
signature). I think we had rough agreement that the simplest way
to solve this would be to do detached CMS signatures (somewhat
like RFC 5485), and not specify encryption at this time.

[Ballot discuss]
Updated as per -04:

Holding DISCUSS on MIME type/file extension registration, as raised by several ADs (including Cullen, Lisa).

The media type registration looks mostly fine. One thing to fix:

File extension(s): none

I think a file extension needs to be selected.

[Ballot comment]
Updated (one extra typo at the end).

A well written document. Some minor editorial comments and questions.

Question: Excuse my ignorance, but what is the difference between File Time Window Options Template defined in section 8.1.2 and Export Session Details Options Template defined in section 8.1.3? The former seems to provide subset of information defined in the latter.

[Ballot discuss]
Updated as per -04:

Holding DISCUSS on MIME type/file extension registration, as raised by several ADs (including Cullen, Lisa).

The media type registration looks mostly fine. One thing to fix:

File extension(s): none

I think a file extension needs to be selected.

I am also waiting for confirmation that the registration was sent out to ietf-types@ mailing list for review.

[Ballot comment]
Updated (one extra typo at the end).

A well written document. Some minor editorial comments and questions.

Question: Excuse my ignorance, but what is the difference between File Time Window Options Template defined in section 8.1.2 and Export Session Details Options Template defined in section 8.1.3? The former seems to provide subset of information defined in the latter.


6.4.  IPFIX Device Diagnostics

  As an IPFIX File can be used store any collection of flows, the

Missing "to" before "store".

  format may also be used for dumping and storing various types of flow
  data for IPFIX Device diagnostics (e.g., the open flow cache of a
  Metering Process or the flow backlog of an Exporting or Collecting
  Process at the time of a process reset or crash).  File-based storage
  is preferable to remote transmission in such error-recovery
  situations.

8.2.4.  maxFlowEndMicroseconds

  Description:  The latest absolute timestamp of the last packet
      within any Flow within the scope containing this Information
      Element, rounded up to the microsecond if necessary.  This
      Information Element SHOULD be bound to its containing IPFIX
      Transport Session via IPFIX Options and the sessionScope
      Information Element.  This Information Element SHOULD be used only
      in Transport Sessions containing Flow Records with microsecond-
      precision (or better) timestamp Information Elements.

  Abstract Data Type:  dateTimeMicroeconds

Typo: dateTimeMicroseconds

8.2.8.  messageMD5Checksum

  Description:  The MD5 checksum of the IPFIX Message containing this
      record.  This Information Element SHOULD be bound to its
      containing IPFIX Message via an options record and the
      messageScope Information Element, as defined below, and SHOULD
      appear only once in a given IPFIX Message.  To calculate the value
      of this Information Element, first buffer the containing IPFIX
      Message, setting the value of this Information Element to all
      zeroes.  Then caluclate the MD5 checksum of the resulting buffer

typo: calculate

      as defined in [RFC1321], place the resulting value in this
      Information Element, and export the buffered message.

[Ballot comment]
A suggestion for editorial improvement:

Section 9.2 seems to suggest that CBC mode has "infinite error
propagation" (an error in ciphertext makes the rest of the file
unreadable). That's not the case; a single bit error in ciphertext
affects only two blocks in CBC (typically, 2*16=32 bytes), so a
single error would usually damage just a single IPFIX Message.

[Ballot discuss]
I have reviewed draft-ietf-ipfix-file-03, and I have a couple of
concerns that I'd like to discuss before recommending approval of
the document:

1) Relying on unspecified external mechanisms for authentication and
confidentiality basically means there's no interoperability between
File Writers and Readers.

It seems some sort of authentication (probably of the File Writer, not
necessarily Metering Process) would be very simple to do (using
e.g. CMS), and could be implemented either as a "detached signature"
(no modifications to the IPFIX File, signature is stored as a separate
file), by wrapping the whole IPFIX File with CMS, or including the
"detached signature" as additional information element at the end of
the file.

Encryption is probably a bit more complicated, and also something
where relying on external mechanisms (for storage and transit) is a
more reasonable assumption than for authentication. While wrapping the
whole IPFIX File with CMS can provide encryption, too, this has
implications for how the system is operated. If public-key based
encryption is used, the signer (probably File Writer) needs the public
keys of the recipients (and this might be quite difficult in many
situations). Password-based encryption for CMS (RFC 3211) would be
simpler to operate, but has its own limitations.

I think the document should specify at least an authentication
mechanism (based on public-key signatures and CMS). I'm not yet quite
sure whether specifying an encryption mechanism would be equally
important -- opinions from others would be welcome here.

2) Even if the File Writer is not signing the data when writing a
file, if the messages over TLS, it seems preserving this information
(the fact that this messages were sent by node A over TLS to node B)
would be important. I haven't worked out all the details yet, but this
could be as simple as specifying "exporterCertificates" (and possibly
"collectorCertificates" -- so we know who the exporter thought it was
sending these records to) information elements in Section 8.1.

3) Section 8.2.8 probably needs a sentence saying that MD5 is used as
a simple checksum to detect random bit errors, not as a security
mechanism or for detecting intentional modifications. (And therefore
therefore collision-resistance or algorithm agility is not absolutely
needed.)

BTW -- The authors have indicated they're going to do some changes to
address Sam Hartman's SecDir review; but updated version hasn't been
submitted yet.

[Ballot comment]
The following three issues were raised by OPS-DIR member Margaret Wassermann. They should be addressed before the publication of the document.

ISSUE #1:  Security Requirements

This document does not seem to be consistent with its own stated security requirements, nor does it specify any interoperable security mechanisms.  In particular, sections 5.6 of the document says:
"5.6. Creator Authentication and Confidentiality "Archival storage of flow data may also require assurance that no unauthorized entity can read or modify the stored data. Asymmetric- key cryptography can be applied to this problem, by signing flow data with the private key of the creator, and encrypting it with the public keys of those authorized to read it. The ideal flow storage format will support the encryption and signing of flow data. As with error correction, this problem has been addressed well at a layer below that addressed by this document. Instead of specifying a particular choice of encryption technology, we can leverage the fact Trammell, et al. 
Expires April 27, 2009 [Page 12]? Internet-Draft IPFIX Files October 20
08 that existing cryptographic technologies work quite well on data stored in files to meet this requirement. Beyond support for the use of TLS for transport over TCP or DTLS for transport over SCTP or UDP, both of which provide transient authentication and confidentiality, the IPFIX protocol does not support this requirement directly. It is assumed that this requirement will be met externally."
I agree with this section, but I cannot find any place in the document where it describes what type of file signing technology should be implemented in File Readers and Writers, what the certificates would look like and/or how they would be bound to the file data.  It also doesn't indicate what encryption technologies should be used by a File Writer to encrypt the file, what elements should/shouldn't be encrypted or how a File Reader would determine what type of decryption to perform.  Without including this information, the document is, essentially, suggesting (or nearly mandating) the implementation of non-interoperable security mechanisms.
ISSUE #2:  Option Scoping
There are options defined that are meant to apply to "the the whole IPFIX Transport Session (i.e., the IPFIX File in the common case)", such as the Export Session Details Option.  However, the document also indicates that the IPFIX file format is intended to support the free concatenation and splitting of IPFIX files at IPFIX Message boundaries.  Is it expected that session-scoped options will appear in all of the messages to which they apply?  If not, how would the session boundaries be determined?  I see that there are min and max export times in the session scoped options, but I don't see anything that prohibits (or even warns against) concatenating IPFIX sessions with overlapping times.  There is some discussion of this topic in section 7.1, where it is stated that File Readers may treat multiple files as a single Transport Session, but that they will not treat a single file as representing multiple Transport Sessions.  However, that doesn't seem to be consistent with the "free manipulability of IPFIX Files through concatenation, appending, and splitting (on IPFIX Message boundaries)" described in section 3.
ISSUE #3:  Data Compression
This issues is similar to the security issue described in ISSUE #1 
above.  The document indicates that data compression is desirable. 
Section 9.1 talks in details about issues with data compression and suggests that block compression be used.  However, the document does not mandate the implementation of any specific compression algorithm, nor does it indicate how a File Reader would determine that a file had been compressed and/or what type of compression was used.  Without this information, the document is suggesting the implementation of a non-interoperable compression mechanism.

[Ballot discuss]
I think this draft should register a file extension. (And would be happy to hand this Discuss to an apps AD). I think this should be fairly easy to go register.

[Ballot discuss]
There are several issues that I would like to raise before publication of this document.
[Note that several issues are derived from Sam Hartman's secdir review and Margaret
Wasserman's Gen-Art review.  My thanks to those reviewers!]

Section 5.5 seems internally inconsistent.  The second sentence states that an "ideal flow
storage format will support the detection and correction of encoding-level errors in the
data" but immediately follows by saying that these services are more appropriate at other
layers.

After a couple of passes, I gather that comprehensive data integrity services (including
detection and correction of encoding-level errors) are needed when storing flow data but
the flow storage format is only expected to provide for error detection and isolation.  Other
layers are expected to provide more advanced error correction if needed.

If that is what is intended, some minor edits should suffice for 5.5, but the need for
external data integrity services should be added to the security considerations.

Section 5.6 has a couple of issues.  First, encrypting bulk data directly using asymmetric
cryptography is generally inappropriate.  Data should be encrypted using an appropriate
symmetric key algorithm, then that symmetric key can be encrypted using the public key(s)
of the authorized reader(s).  Paragraph 1 of 5.6 seems to specify direct encryption using
asymmetric cryptography.

Secondly, while I agree that authentication and confidentiality can be provisioned at a lower
layer, I think a mandatory to implement security mechanism is needed for interoperability.
(CMS would be one good choice here.)

My third note on 5.6 could be satisfied here or in the security considerations.  Use of TLS
and DTLS for transient protection is mentioned in passing, but the applicability isn't clear.
If the flow is sensitive enough to protect in storage, and the messages are being transmitted
to other components prior to signing or encrypting for storage (e.g., using CMS) then the
messages should have been protected on the wire using TLS or DTLS. 

Security Considerations: 

The first paragraph ends with "The file format must therefore provide appropriate
procedures to guarantee the integrity and confidentiality of the stored information."
However, much of section 5 was devoted to explaining why the file format could not
included procedures to make that guarantee, and needed to be supplemented with
other mechanisms.

Since long term storage is envisioned by this specification, the strength of the cryptography
employed has to factor in the expected lifetime of the data.  For example,
using SHA-1 and RSA 1024 to sign a flow today is okay if the data will be retained for a
few months, but authenticating the creator in 2015 would demand SHA-256 and RSA 2048.
Similar calculus must be performed with respect to the confidentiality requirements.

Cryptographic algorithms become weaker with the passing of time and/or improvements
in cryptographic attacks.  To maintain security for a particular file, it may be necessary to
re-apply cryptographic protection.  In particular, to maintain the ability to verify the file
creator, it may be necessary to re-sign a file with stronger keys, encapsulating the old
signature and data, to extend the protection.  A reference to RFC 4810 along with a brief
explanation of the issue might be a good addition here.

Sam Hartman's followup email (4/22/09) highlighted two limitations that seem
to be inherent in the model:

>The first is that the data is not integrity protected end-to-end.  The
>exporter may transport to the collector over TLS.  Let's assume that
>the collector is digitally signing the file.  There's a break in the
>cryptographic protection at the collector: it could modify the data
>however it likes.  When I read the file years later, I have to trust
>the collector and the exporter, not just the exporter.

>If you considered that a problem and wanted to fix it, you'd need to
>add digital signatures at the exporter.  Given my understanding of
>ipfix, that would seem both expensive and unnecessary.  So, instead,
>you should document the issue and explain that the file reader needs
>to trust the file writer not to modify the contents.

>The second issue is that the collector has no way to tell the file
>reader the identity of the exporter.  If I have a TLS connection with
>badguy@evil.example.com, I might trust the data less than data from
>sensor-53@east.example.net.  The identity of the collector could be captured in a CMS
>signature on the file.
>However nothing captures the identity of the exporter--not even as an assertion from
>the collector.

These limitation need to be documented in the security considerations as well.

[Ballot discuss]
There are several issues that I would like to raise before publication of this document.
[Note that several issues are derived from Sam Hartman's secdir review and Margaret
Wasserman's Gen-Art review.  My thanks to those reviewers!]

Section 5.5 seems internally inconsistent.  The second sentence states that an "ideal flow
storage format will support the detection and correction of encoding-level errors in the
data" but immediately follows by saying that these services are more appropriate at other
layers.

After a couple of passes, I gather that comprehensive data integrity services (including
detection and correction of encoding-level errors) are needed when storing flow data but
the flow storage format is only expected to provide for error detection and isolation.  Other
layers are expected to provide more advanced error correction if needed.

If that is what is intended, some minor edits should suffice for 5.5, but the need for
external data integrity services should be added to the security considerations.

Section 5.6 has a couple of issues.  First, encrypting bulk data directly using asymmetric
cryptography is generally inappropriate.  Data should be encrypted using an appropriate
symmetric key algorithm, then that symmetric key can be encrypted using the public key(s)
of the authorized reader(s).  Paragraph 1 of 5.6 seems to specify direct encryption using
asymmetric cryptography.

Secondly, while I agree that authentication and confidentiality can be provisioned at a lower
layer, I think a mandatory to implement security mechanism is needed for interoperability.
(CMS would be one good choice here.)

My third note on 5.6 could be satisfied here or in the security considerations.  Use of TLS
and DTLS for transient protection is mentioned in passing, but the applicability isn't clear.
If the flow is sensitive enough to protect in storage, and the messages are being transmitted
to other components prior to signing or encrypting for storage (e.g., using CMS) then the messages should have been protected on the wire using TLS or DTLS. 

Security Considerations: 

The first paragraph ends with "The file format must therefore provide appropriate
procedures to guarantee the integrity and confidentiality of the stored information."
However, much of section 5 was devoted to explaining why the file format could not
included procedures to make that guarantee, and needed to be supplemented with
other mechanisms.

Since long term storage is envisioned by this specification, the strength of the cryptography
employed has to factor in the expected lifetime of the data.  For example,
using SHA-1 and RSA 1024 to sign a flow today is okay if the data will be retained for a
few months, but authenticating the creator in 2015 would demand SHA-256 and RSA 2048.
Similar calculus must be performed with respect to the confidentiality requirements.

Cryptographic algorithms become weaker with the passing of time and/or improvements
in cryptographic attacks.  To maintain security for a particular file, it may be necessary to
re-apply cryptographic protection.  In particular, to maintain the ability to verify the file
creator, it may be necessary to re-sign a file with stronger keys, encapsulating the old
signature and data, to extend the protection.  A reference to RFC 4810 along with a brief
explanation of the issue might be a good addition here.

Sam Hartman's followup email (4/22/09) highlighted two limitations that seem
to be inherent in the model:

>The first is that the data is not integrity protected end-to-end.  The
>exporter may transport to the collector over TLS.  Let's assume that
>the collector is digitally signing the file.  There's a break in the
>cryptographic protection at the collector: it could modify the data
>however it likes.  When I read the file years later, I have to trust
>the collector and the exporter, not just the exporter.

>If you considered that a problem and wanted to fix it, you'd need to
>add digital signatures at the exporter.  Given my understanding of
>ipfix, that would seem both expensive and unnecessary.  So, instead,
>you should document the issue and explain that the file reader needs
>to trust the file writer not to modify the contents.

>The second issue is that the collector has no way to tell the file
>reader the identity of the exporter.  If I have a TLS connection with
>badguy@evil.example.com, I might trust the data less than data from
>sensor-53@east.example.net.  The identity of the collector could be captured in a CMS
>signature on the file.
>However nothing captures the identity of the exporter--not even as an assertion from
>the collector.

These limitation need to be documented in the security considerations as well.

[Ballot discuss]
There are several issues that I would like to raise before publication of this document.
[Note that several issues are derived from Sam Hartman's secdir review and Margaret
Wasserman's Gen-Art review.  My thanks to those reviewers!]

Section 5.5 seems internally inconsistent.  The second sentence states that an "ideal flow
storage format will support the detection and correction of encoding-level errors in the
data" but immediately follows by saying that these services are more appropriate at other
layers.

After a couple of passes, I gather that comprehensive data integrity services (including
detection and correction of encoding-level errors) are needed when storing flow data but
the flow storage format is only expected to provide for error detection and isolation.  Other
layers are expected to provide more advanced error correction if needed.

If that is what is intended, some minor edits should suffice for 5.5, but the need for
external data integrity services should be added to the security considerations.

Section 5.6 has a couple of issues.  First, encrypting bulk data directly using asymmetric
cryptography is generally inappropriate.  Data should be encrypted using an appropriate
symmetric key algorithm, then that symmetric key can be encrypted using the public key(s)
of the authorized reader(s).  Paragraph 1 of 5.6 seems to specify direct encryption using
asymmetric cryptography.

Secondly, while I agree that authentication and confidentiality can be provisioned at a lower
layer, I think a mandatory to implement security mechanism is needed for interoperability.
(CMS would be one good choice here.)

My third note on 5.6 could be satisfied here or in the security considerations.  Use of TLS
and DTLS for transient protection is mentioned in passing, but the applicability isn't clear.
If the flow is sensitive enough to protect in storage, and the messages are being transmitted
to other components prior to signing or encrypting for storage (e.g., using CMS) then the messages should have been protected on the wire using TLS or DTLS. 

Security Considerations: 

The first paragraph ends with "The file format must therefore provide appropriate
procedures to guarantee the integrity and confidentiality of the stored information."
However, much of section 5 was devoted to explaining why the file format could not
included procedures to make that guarantee, and needed to be supplemented with
other mechanisms.

Since long term storage is envisioned by this specification, the strength of the cryptography
employed has to factor in the expected lifetime of the data.  For example,
using SHA-1 and RSA 1024 to sign a flow today is okay if the data will be retained for a
few months, but authenticating the creator in 2015 would demand SHA-256 and RSA 2048.
Similar calculus must be performed with respect to the confidentiality requirements.

Cryptographic algorithms become weaker with the passing of time and/or improvements
in cryptographic attacks.  To maintain security for a particular file, it may be necessary to
re-apply cryptographic protection.  In particular, to maintain the ability to verify the file
creator, it may be necessary to re-sign a file with stronger keys, encapsulating the old
signature and data, to extend the protection.  A reference to RFC 4810 along with a brief
explanation of the issue might be a good addition here.
addition

[Ballot discuss]
There are several issues that I would like to raise before publication of this document.
[Note that several issues are derived from Sam Hartman's secdir review and Margaret
Wasserman's Gen-Art review.  My thanks to those reviewers!]

Section 5.5 seems internally inconsistent.  The second sentence states that an "ideal flow
storage format will support the detection and correction of encoding-level errors in the
data" but immediately follows by saying that these services are more appropriate at other
layers.

After a couple of passes, I gather that comprehensive data integrity services (including
detection and correction of encoding-level errors) are needed when storing flow data but
the flow storage format is only expected to provide for error detection and isolation.  Other
layers are expected to provide more advanced error correction if needed.

If that is what is intended, some minor edits should suffice for 5.5, but the need for
external data integrity services should be added to the security considerations.

Section 5.6 has a couple of issues.  First, encrypting bulk data directly using asymmetric
cryptography is generally inappropriate.  Data should be encrypted using an appropriate
symmetric key algorithm, then that symmetric key can be encrypted using the public key(s)
of the authorized reader(s).  Paragraph 1 of 5.6 seems to specify direct encryption using
asymmetric cryptography.

Secondly, while I agree that authentication and confidentiality can be provisioned at a lower
layer, I think a mandatory to implement security mechanism is needed for interoperability.
(CMS would be one good choice here.)

My third note on 5.6 could be satisfied here or in the security considerations.  Use of TLS
and DTLS for transient protection is mentioned in passing, but the applicability isn't clear.
If the flow is sensitive enough to protect in storage, and the messages are being transmitted
to other components prior to signing or encrypting for storage (e.g., using CMS) then the messages should have been protected on the wire using TLS or DTLS. 

Security Considerations:  Since we are taking about long term storage here, strength of
cryptography employed should factor in the expected lifetime of the data.  For example,
using SHA-1 and RSA 1024 to sign a flow today is okay if the data will be retained for a
few months, but authenticating the creator in 2015 would demand SHA-256 and RSA 2048.
Similar calculus must be performed with respect to the confidentiality requirements.

[Ballot comment]
Updated (one extra typo at the end).

A well written document. Some minor editorial comments and questions.

Question: Excuse my ignorance, but what is the difference between File Time Window Options Template defined in section 8.1.2 and Export Session Details Options Template defined in section 8.1.3? The former seems to provide subset of information defined in the latter.


This document is asking for MIME type registration for the file format ;-).


6.4.  IPFIX Device Diagnostics

  As an IPFIX File can be used store any collection of flows, the

Missing "to" before "store".

  format may also be used for dumping and storing various types of flow
  data for IPFIX Device diagnostics (e.g., the open flow cache of a
  Metering Process or the flow backlog of an Exporting or Collecting
  Process at the time of a process reset or crash).  File-based storage
  is preferable to remote transmission in such error-recovery
  situations.

8.2.4.  maxFlowEndMicroseconds

  Description:  The latest absolute timestamp of the last packet
      within any Flow within the scope containing this Information
      Element, rounded up to the microsecond if necessary.  This
      Information Element SHOULD be bound to its containing IPFIX
      Transport Session via IPFIX Options and the sessionScope
      Information Element.  This Information Element SHOULD be used only
      in Transport Sessions containing Flow Records with microsecond-
      precision (or better) timestamp Information Elements.

  Abstract Data Type:  dateTimeMicroeconds

Typo: dateTimeMicroseconds

8.2.8.  messageMD5Checksum

  Description:  The MD5 checksum of the IPFIX Message containing this
      record.  This Information Element SHOULD be bound to its
      containing IPFIX Message via an options record and the
      messageScope Information Element, as defined below, and SHOULD
      appear only once in a given IPFIX Message.  To calculate the value
      of this Information Element, first buffer the containing IPFIX
      Message, setting the value of this Information Element to all
      zeroes.  Then caluclate the MD5 checksum of the resulting buffer

typo: calculate

      as defined in [RFC1321], place the resulting value in this
      Information Element, and export the buffered message.

[Ballot comment]
A well written document. Some minor editorial comments and questions.

Question: Excuse my ignorance, but what is the difference between File Time Window Options Template defined in section 8.1.2 and Export Session Details Options Template defined in section 8.1.3? The former seems to provide subset of information defined in the latter.


This document is asking for MIME type registration for the file format ;-).


6.4.  IPFIX Device Diagnostics

  As an IPFIX File can be used store any collection of flows, the

Missing "to" before "store".

  format may also be used for dumping and storing various types of flow
  data for IPFIX Device diagnostics (e.g., the open flow cache of a
  Metering Process or the flow backlog of an Exporting or Collecting
  Process at the time of a process reset or crash).  File-based storage
  is preferable to remote transmission in such error-recovery
  situations.

8.2.4.  maxFlowEndMicroseconds

  Description:  The latest absolute timestamp of the last packet
      within any Flow within the scope containing this Information
      Element, rounded up to the microsecond if necessary.  This
      Information Element SHOULD be bound to its containing IPFIX
      Transport Session via IPFIX Options and the sessionScope
      Information Element.  This Information Element SHOULD be used only
      in Transport Sessions containing Flow Records with microsecond-
      precision (or better) timestamp Information Elements.

  Abstract Data Type:  dateTimeMicroeconds

Typo: dateTimeMicroseconds

[Ballot comment]
A well written document. Some minor editorial comments and questions.

Question: Excuse my ignorance, but what is the difference between File Time Window Options Template defined in section 8.1.2 and Export Session Details Options Template defined in section 8.1.3? The former seems to provide subset of information defined in the latter.


6.4.  IPFIX Device Diagnostics

  As an IPFIX File can be used store any collection of flows, the

Missing "to" before "store".

  format may also be used for dumping and storing various types of flow
  data for IPFIX Device diagnostics (e.g., the open flow cache of a
  Metering Process or the flow backlog of an Exporting or Collecting
  Process at the time of a process reset or crash).  File-based storage
  is preferable to remote transmission in such error-recovery
  situations.

8.2.4.  maxFlowEndMicroseconds

  Description:  The latest absolute timestamp of the last packet
      within any Flow within the scope containing this Information
      Element, rounded up to the microsecond if necessary.  This
      Information Element SHOULD be bound to its containing IPFIX
      Transport Session via IPFIX Options and the sessionScope
      Information Element.  This Information Element SHOULD be used only
      in Transport Sessions containing Flow Records with microsecond-
      precision (or better) timestamp Information Elements.

  Abstract Data Type:  dateTimeMicroeconds

Typo: dateTimeMicroseconds

Gen-ART Review by Vijay K. Gurbani:

I have been selected as the General Area Review Team (Gen-ART) reviewer for this draft (for background on Gen-ART, please see http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html).

Please wait for direction from your document shepherd or AD before posting a new version of the draft.

Document: draft-ietf-ipfix-file-03.txt
Reviewer: Vijay K. Gurbani
Review Date: Apr 1, 2009

Summary: This draft is ready for publication as a Proposed Standard.

The draft has 0 major issues, 0 minor issues, and 0 Nits/Editorial comments.

IANA comments:

Upon approval of this document, IANA will make the following
assignments in the "IPFIX Information Elements" registry at
http://www.iana.org/assignments/ipfix/ipfix.xhtml

Value : TBD1
Name : collectionTimeMilliseconds
Data Type : dateTimeMilliseconds
Data Type Semantics :
Status : current
Description : The absolute timestamp at which the data within the
scope containing this Information Element was received by a
Collecting Process. This Information Element SHOULD be bound to
its containing IPFIX Message via IPFIX Options and the
messageScope Information Element, as defined below.
Units :
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD2
Name : exportSctpStreamId
Data Type : unsigned16
Data Type Semantics : identifier
Status : current
Description : The value of the SCTP Stream Identifier used by the
Exporting Process for exporting IPFIX Message data. This is
carried in the Stream Identifier field of the header of the SCTP
DATA chunk containing the IPFIX Message(s).
Units :
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD3
Name : maxExportSeconds
Data Type : dateTimeSeconds
Data Type Semantics :
Status : current
Description : The absolute Export Time of the latest IPFIX Message
within the scope containing this Information Element. This
Information Element SHOULD be bound to its containing IPFIX
Transport Session via IPFIX Options and the sessionScope
Information Element.
Units : seconds
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD4
Name : maxFlowEndMicroseconds
Data Type : dateTimeMicroeconds
Data Type Semantics :
Status : current
Description : The latest absolute timestamp of the last packet
within any Flow within the scope containing this Information
Element, rounded up to the microsecond if necessary. This
Information Element SHOULD be bound to its containing IPFIX
Transport Session via IPFIX Options and the sessionScope
Information Element. This Information Element SHOULD be used only
in Transport Sessions containing Flow Records with microsecond-
precision (or better) timestamp Information Elements.
Units : microseconds
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD5
Name : maxFlowEndMilliseconds
Data Type : dateTimeMilliseconds
Data Type Semantics :
Status : current
Description : The latest absolute timestamp of the last packet
within any Flow within the scope containing this Information
Element, rounded up to the millisecond if necessary. This
Information Element SHOULD be bound to its containing IPFIX
Transport Session via IPFIX Options and the sessionScope
Information Element. This Information Element SHOULD be used only
in Transport Sessions containing Flow Records with millisecond-
precision (or better) timestamp Information Elements.
Units : milliseconds
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD6
Name : maxFlowEndNanoseconds
Data Type : dateTimeNanoseconds
Data Type Semantics :
Status : current
Description : The latest absolute timestamp of the last packet
within any Flow within the scope containing this Information
Element. This Information Element SHOULD be bound to its
containing IPFIX Transport Session via IPFIX Options and the
sessionScope Information Element. This Information Element SHOULD
be used only in Transport Sessions containing Flow Records with
nanosecond-precision timestamp Information Elements.
Units : nanoseconds
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD7
Name : maxFlowEndSeconds
Data Type : dateTimeSeconds
Data Type Semantics :
Status : current
Description : The latest absolute timestamp of the last packet
within any Flow within the scope containing this Information
Element, rounded up to the second if necessary. This Information
Element SHOULD be bound to its containing IPFIX Transport Session
via IPFIX Options and the sessionScope Information Element.
Units : seconds
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD8
Name : messageMD5Checksum
Data Type : octetArray (16 bytes)
Data Type Semantics :
Status : current
Description : The MD5 checksum of the IPFIX Message containing this
record. This Information Element SHOULD be bound to its
containing IPFIX Message via an options record and the
messageScope Information Element, as defined below, and SHOULD
appear only once in a given IPFIX Message. To calculate the value
of this Information Element, first buffer the containing IPFIX
Message, setting the value of this Information Element to all
zeroes. Then caluclate the MD5 checksum of the resulting buffer
as defined in [RFC1321], place the resulting value in this
Information Element, and export the buffered message.
Units :
Range :
References : RFC 1321, The MD5 Message-Digest Algorithm [RFC1321]
Requester : [RFC-ipfix-file-03]


Value : TBD9
Name : messageScope
Data Type : octet
Data Type Semantics :
Status : current
Description : The presence of this Information Element as scope in
an Options Template signifies that the options described by the
Template apply to the IPFIX Message that contains them. It is
defined for general purpose message scoping of options, and
proposed specifically to allow the attachment a checksum to a
message via IPFIX Options. The value of this Information Element
MUST be written as 0 by the File Writer or Exporting Process. The
value of this Information Element MUST be ignored by the File
Reader or the Collecting Process.
Units :
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD10
Name : minExportSeconds
Data Type : dateTimeSeconds
Data Type Semantics :
Status : current
Description : The absolute Export Time of the earliest IPFIX Message
within the scope containing this Information Element. This
Information Element SHOULD be bound to its containing IPFIX
Transport Session via an options record and the sessionScope
Information Element.
Units : seconds
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD11
Name : minFlowStartMicroseconds
Data Type : dateTimeMicroseconds
Data Type Semantics :
Status : current
Description : The earliest absolute timestamp of the first packet
within any Flow within the scope containing this Information
Element, rounded down to the microsecond if necessary. This
Information Element SHOULD be bound to its containing IPFIX
Transport Session via an options record and the sessionScope
Information Element. This Information Element SHOULD be used only
in Transport Sessions containing Flow Records with microsecond-
precision (or better) timestamp Information Elements.
Units : microseconds
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD12
Name : minFlowStartMilliseconds
Data Type : dateTimeMilliseconds
Data Type Semantics :
Status : current
Description : The earliest absolute timestamp of the first packet
within any Flow within the scope containing this Information
Element, rounded down to the millisecond if necessary. This
Information Element SHOULD be bound to its containing IPFIX
Transport Session via an options record and the sessionScope
Information Element. This Information Element SHOULD be used only
in Transport Sessions containing Flow Records with millisecond-
precision (or better) timestamp Information Elements.
Units : milliseconds
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD13
Name : minFlowStartNanoseconds
Data Type : dateTimeNanoseconds
Data Type Semantics :
Status : current
Description : The earliest absolute timestamp of the first packet
within any Flow within the scope containing this Information
Element. This Information Element SHOULD be bound to its
containing IPFIX Transport Session via an options record and the
sessionScope Information Element. This Information Element SHOULD
be used only in Transport Sessions containing Flow Records with
nanosecond-precision timestamp Information Elements.
Units : nanoseconds
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD14
Name : minFlowStartSeconds
Data Type : dateTimeSeconds
Data Type Semantics :
Status : current
Description : The earliest absolute timestamp of the first packet
within any Flow within the scope containing this Information
Element, rounded down to the second if necessary. This
Information Element SHOULD be bound to its containing IPFIX
Transport Session via an options record and the sessionScope
Information Element.
Units : seconds
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD15
Name : opaqueOctets
Data Type : octet
Data Type Semantics :
Status : current
Description : This Information Element is used to encapsulate non-
IPFIX data into an IPFIX Message stream, for the purpose of
allowing a non-IPFIX data processor to store a data stream inline
within an IPFIX File. A Collecting Process or File Writer MUST
NOT try to interpret this binary data. This Information Element
differs from paddingOctets as its contents are meaningful in some
non-IPFIX context, while the contents of paddingOctets MUST be
0x00 and are intended only for Information Element alignment.
Units :
Range :
References :
Requester : [RFC-ipfix-file-03]


Value : TBD16
Name : sessionScope
Data Type : octet
Data Type Semantics :
Status : current
Description : The presence of this Information Element as scope in
an Options Template signifies that the options described by the
Template apply to the IPFIX Transport Session that contains them.
Note that as all options are implicitly scoped to Transport
Session and Observation Domain, this Information Element is
equivalent to a "null" scope. It is defined for general purpose
session scoping of options, and proposed specifically to allow the
attachment of time window to an IPFIX File via IPFIX Options. The
value of this Information Element MUST be written as 0 by the File
Writer or Exporting Process. The value of this Information
Element MUST be ignored by the File Reader or the Collecting
Process.
Units :
Range :
References :
Requester : [RFC-ipfix-file-03]


We understand the above to be the only IANA Action for this document.

Document:  draft-ietf-ipfix-file-03.txt
Title:    Specification of the IPFIX File Format
Editors:  B. Trammell, E. Boschi, L. Mark, T. Zseby, A. Wagner
Intended status:  Standards Track

As required by RFC-to-be draft-ietf-proto-wgchair-doc-shepherding,
this is the current template for the Document Shepherd Write-Up.
Changes are expected over time. This version is dated September 17, 2008.

(1.a) Who is the Document Shepherd for this document? Has the
      Document Shepherd personally reviewed this version of the
      document and, in particular, does he or she believe this
      version is ready for forwarding to the IESG for publication?

        Nevil Brownlee.  I have reviewed this draft, I believe
        it is ready to forward to the IESG for publication.

(1.b) Has the document had adequate review both from key WG members
      and from key non-WG members? Does the Document Shepherd have
      any concerns about the depth or breadth of the reviews that
      have been performed?

        This is version 3 of the draft.  It has had extensive review
        by the WG members, and by others from the PSAMP WG.
        These reviews seem sufficiently thorough to me.

(1.c) Does the Document Shepherd have concerns that the document
      needs more review from a particular or broader perspective,
      e.g., security, operational complexity, someone familiar with
      AAA, internationalization or XML?

        No.  The draft is concerned only with specifying a
        way to store IPFIX Information in files so as to preserve
        it for later use, e.g. for analysis by application programs,
        for testing IPFIX colletcors, etc.

(1.d) Does the Document Shepherd have any specific concerns or
      issues with this document that the Responsible Area Director
      and/or the IESG should be aware of? For example, perhaps he
      or she is uncomfortable with certain parts of the document, or
      has concerns whether there really is a need for it. In any
      event, if the WG has discussed those issues and has indicated
      that it still wishes to advance the document, detail those
      concerns here. Has an IPR disclosure related to this document
      been filed? If so, please include a reference to the
      disclosure and summarize the WG discussion and conclusion on
      this issue.

        There are no specific concerns for this document.

(1.e) How solid is the WG consensus behind this document? Does it
      represent the strong concurrence of a few individuals, with
      others being silent, or does the WG as a whole understand and
      agree with it?

        This document has strong consensus within the IPFIX
        Working Group.

(1.f) Has anyone threatened an appeal or otherwise indicated extreme
      discontent? If so, please summarise the areas of conflict in
      separate email messages to the Responsible Area Director. (It
      should be in a separate email because this questionnaire is
      entered into the ID Tracker.)

        No.

(1.g) Has the Document Shepherd personally verified that the
      document satisfies all ID nits? (See
      http://www.ietf.org/ID-Checklist.html and
      http://tools.ietf.org/tools/idnits/). Boilerplate checks are
      not enough; this check needs to be thorough. Has the document
      met all formal review criteria it needs to, such as the MIB
      Doctor, media type and URI type reviews?

        idnits 2.10.02 says
          a. Change boilerplate to RFC 4748-style
              16 Dec 08 is the 'required from' date
          b. Possible downref to ' IPFIX Exporting Types' draft
              This draft is now in RFC Editor queue

        b. Exporting Types is Standards Track, no problem.
        a. could be fixed in RFC Editor 48hr

        The draft is IPFIX-centric, it needs no other formal checks.

(1.h) Has the document split its references into normative and
      informative? Are there normative references to documents that
      are not ready for advancement or are otherwise in an unclear
      state? If such normative references exist, what is the
      strategy for their completion? Are there normative references
      that are downward references, as described in [RFC3967]? If
      so, list these downward references to support the Area
      Director in the Last Call procedure for them [RFC3967].

        Yes, the references are split.
        No, the only dangling reference is to the IPFIX Exporting
          Types draft; I'm submitting that together with this one.
        No, there are no downward references.

(1.i) Has the Document Shepherd verified that the document IANA
      consideration section exists and is consistent with the body
      of the document? If the document specifies protocol
      extensions, are reservations requested in appropriate IANA
      registries? Are the IANA registries clearly identified? If
      the document creates a new registry, does it define the
      proposed initial contents of the registry and an allocation
      procedure for future registrations? Does it suggest a
      reasonable name for the new registry? See [RFC5226]. If the
      document describes an Expert Review process has Shepherd
      conferred with the Responsible Area Director so that the IESG
      can appoint the needed Expert during the IESG Evaluation?

        The document has an IANA Considerations section that
        clearly explains what IANA should do.  It asks for
        16 new Information Element numbers in the IPFIX IE Registry.
        Now that the IFIX IE Registry is in production, this is
        a simple addition to it.

(1.j) Has the Document Shepherd verified that sections of the
      document that are written in a formal language, such as XML
      code, BNF rules, MIB definitions, etc., validate correctly in
      an automated checker?

        The document has no parts written in a formal language.

(1.k) The IESG approval announcement includes a Document
      Announcement Write-Up. Please provide such a Document
      Announcement Write-Up? Recent examples can be found in the
      "Action" announcements for approved documents. The approval
      announcement contains the following sections:

Technical Summary

This document describes a file format for the storage of flow data
based upon the IPFIX Protocol.  It proposes a set of requirements for
flat-file, binary flow data file formats, then specifies the IPFIX
File format to meet these requirements based upon IPFIX Messages.
This IPFIX File format is designed to facilitate interoperability and
reusability among a wide variety of flow storage, processing, and
analysis tools.

Working Group Summary

Work started on this document in 2006, and has gone through several
revisions in response to mailing list discussion.  Originally it
included the material about 'exporting type information,' but the
WG split it out into a seperate RFC to make it more easily accessible
for other uses.  The WG has reached a clear consensus on this draft.

Document Quality

This document has been reviwed by Benoit Claise, Paul Aitken, Andrew
Johnson, and Gerhard Muenz, as well as by the WG chairs.  It specifies
a file format for IPFIX data, making the point that this is simply
another data transport for IPFIX.  It does not make any changes to the
IPFIX protocol.