Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information
RFC 7011

Note: This ballot was opened for revision 07 and is now closed.

(Joel Jaeggli) Yes

Comment (2013-07-05 for -08)
No email
send info
The opsdir review of the latest draft is extensive...

Thanks sue for doing it


Appendix A.  Operations and Management Review Checklist
Well written.  Only a few technical questions/considerations:

a) What happens when different observation points of an observation domain have different levels of timing (nano-second vs. second)? Does the second timing data get put in a different queue than nano-second data? 
b) Is there key rotation with DLTS using 2-way/3-way/ or 4-way handshake?
c) What happens if the transports switch rapidly (UDP to TCP to STCP and rotate (UDP to TCP to UDP)? Are these denial of service attacks? 

Editorial: 
page 8 Flow key – could use an example. 
           
page 25 – 4.1 end of sentence change from: ; see (IPFIX-IANA] for their definitions”  to: defined in [IPFIX-IANA]:
p. 36 Section 8.0 last paragraph – could use an example.
p. 50 section 11.1 – Last paragraph the offset by commas that starts “,i.e., a true and ends (and/or TCP), this” is confusing.  Please reword 

p. 51  11.2 para. 1 “[RFC5246] and [5347],” to [RFC5246] and [5347]” otherwise sentence does not have strong sense of while.’

p. 52  section 11.4, para. 2 change “or scope information, a large amount” to “or scope information, or a large amount” 

p. 53 “section 11.4 para 5 change “a Collecting process; if provided, the” to “a Collecting process; and if provided, the”

 
          
A.1.  Operational Considerations
1. Has deployment been discussed: 
a. Does the document include a description of how this protocol or technology is going to be deployed and managed?

Actual Deployments are not discussed in depth. Management and operational conditions for multiple transport, DOS, and scaling issues are discussed. 


 
b. Is the proposed specification deployable?  If not, how could it be improved? 

This solution has description of scaling and management concerns. Any solution that monitors at per interface or more refined will run into issues with scaling. 

Improving the deployment would take another approach to the problem of store/forward or trigger updates. However, this document is about updating an existing process. For this approach, the authors have done a good job at working through possible refinements. If IETF wants 

c. Does the solution scale well from the operational and management perspective? Does the proposed approach have any scaling issues that could affect usability for large-scale operation?

Most issues have been mentioned. A few things that should be considered: 1) fluctuations between times and queuing for different observation points or different data and 2) fluctuations if the transport failures cause the implementation to rate between transports. 

d. Are there any coexistence issues? 

Coexistence issues in transports or IDs are clearly stated. The document specifies it is interoperable with previous version of protocol [RFC101]

   2.  Has installation and initial setup been discussed?   
* Is the solution sufficiently configurable? Are configuration parameters clearly defined? Are configuration parameters normalized?  Does the configuration have a reasonable default value? 

Solution appears to be configurable, defined, and normalized, with default values, but that configuration is dealt with elsewhere, and it is not the topic of this draft.  
o Configuration: RFC6615 (MIB), RFC627 (NETCONF), 
o Implementation guidelines [RFC5153], testing [RFC5471] 
 
* Will configuration be pushed to a device by a configuration
          manager, or pulled by a device from a configuration server?

The MIB and NETCONF options allow either configuration manager to push the information. It is not clear if the IPFix device, exporter, collecting process or collector can pull global configuration.  Since the TLVs in a sense self-configure, this is in a sense pulling the configuration.   


* How will the devices and managers find and authenticate each other? 
    
The devices and managers appear to find each other in an a priori manner via configuration. Group functions exist for observation point to observation domain, collecting processes to collecting hosts, and exporter processes to exporters.  However, it does not appear that these grouping processes self-detect. The data within these monitoring flows self-detect but the identity does not appear to find each outside of configuration. If I have missed this, you might consider that I re-read RFC3917, RFC5470, RFC6615(MIB), RFC6728(netconf), RFC6727 (MIB), and RFC5102bis (MIB).

RFC6728 indicates that the following trees/substrees contain sensitive information and write operations can have an impact:
a) /ipfix/Observation 
b) ipfix/cache
c) ipfix/exportingProcess
d) ipfix/collectingProcess

Due to these statements and section 11 that states an IPFIX collecting process (or processes) must prevent collection from an unauthorized Exporting Process or the export of data to an unauthorized collected process.    

This implies that the “discover” each method would need additional security mechanisms. 

3. Has the migration path been discussed?  Are there any backward compatibility issues?

The document states the protocol is interoperable with RFC5101. The self-defining mechanism seems to provide this. 

4. Have the Requirements on other protocols and functional
       components been discussed? 

In quick summary – Yes. The smorgasbord of options and parameters has been described, 

The impact of the ipfix process on the underlying carrying protocols (UDP, TCP, STCP), the device/node in the being sampled, the exporters impact, the collectors impact have been carefully specified. New protocol functions are either specified or reference by RFC. The NETCONF/YANG or MIB models are given as Management. 



   5.  Has the impact on network operation been discussed?   

	This document does discuss:
o how ipfix increases traffic load and choice the network operator has in managing, 
o how ipfix will increase traffic load on existing networks and how to manage it,
o how the protocol use impacts exporting processes and exporter, collecting processes and collector, 
o how the device impacts the behaviors of other protocols by sending traffic load and how to manage it,
o how the device requires security.

Since the IDs are self-defining, it does not appear that the DNS service has a critical impact on these. The DNS-ID is mutual authentication process which requires the DNS-ID as specified in section 6.4 of RFC6125. The security requires that the Common-Name (CN-IDs) not be placed in identifiers.   

6. Have suggestions for verifying correct operation been discussed?

The Testing operations exists RFC5471 and RFC5153. Since the protocols are interoperable only the differences need to be examine. 

The only differences that section 1.1 notes that impact implementation and testing are the timer encodings link to epochs or rollover, and template management relaxation. 

While section 5 and section 8 cover this information to one skilled in the art, it may be worthwhile to update both RFC5471 and RFC5153 with the appropriate information. 


7. Has management interoperability been discusses? 

In short, yes.  This protocol is a model protocol for information models, data models, and choices across platforms. 

8. Are there fault or threshold conditions that should be reported

This management information has timing implications and ties to a timing utility.  The information is reporting on a constant stream. There does not appear to be any notifications defined in MIBs. This is not(!!) a polling application for event-drive. 
Overload alarms/notifications might be useful, except they might further provide a DOS attack.  

State saving, caching, template definition keeping are described carefully and thoroughly.





   9.  Is configuration discussed?  See Section 3.4.

       *  Are configuration defaults and default modes of operation
          considered? Defaults are described in MIB. Default process 
          in this document. The discussion seems sufficient. 

       *  Is there discussion of what information should be preserved
          across reboots of the device or the management system?  Can
          devices realistically preserve this information through hard
          reboots where physical configuration might change (e.g., 
	   cards might be swapped while a chassis is powered down)?

Topics regarding reboot are not covered in this document. The word restart is only used regarding restarting the UDP. The word reboot is not used. 

A.2.  Management Considerations


   Do you anticipate any manageability issues with the specification?

1. Is management interoperability discussed?   

The management may be centralized or distribution or a combination. The collector processes are considered to be remote from the exporting processes. However, the exportation/collection can be in the same virtual environment. 

No textual or graphical user interfaces are required. The format of the protocol utilizes TLVs. The internal data may be considered “sensitive” and may be encrypted due to its sensitivity whether it is textual or binary. 

   2.  Is management information discussed?  See Section 3.2.

* What is the minimal set of management (configuration, faults,
          performance monitoring) objects that need to be instrumented
          in order to manage the new protocol?

	This is not discussed in this document.  
	

   3.  Is fault management discussed?  See Section 3.3.

* Is Liveness Detection and Monitoring discussed? For the liveness indications for the transport protocols (STCP, UDP, TCP) and the exterior needs for liveness/monitoring neded when UDP is issued.   

* Does the solution have failure modes that are difficult to diagnose or correct? Faults and alarms are logged. However, if errors reoccur the mere keeping of the faults may overload. An overload mechanisms on exporting was discussed in 4.2 (Metering process)and 4.3 exporting process.

4. Is configuration management discussed? 

* Is protocol state information exposed to the user?  MIBS and YANG/netconf model expose state and tables to user. 

* Significant failures of transport, stopping metering (4.2), stopping export (4.3) are discussed. 
          


5. Is accounting management discussed?  
Only in terms of the link between performance data and accounting.

   6.  Is performance management discussed?  See Section 3.6.

* Protocol impacts performance of nodes (exporter load, collector load) and the network data passage (due to amount of data sent. This is discussed. 
* Protocol performance information is exposed when the information is dropped (metering 4.2, exporting, 4.3).  The performance issues are discussed, but the performance metrics 

       *  Is protocol performance information exposed to the user?

7. Is security management discussed?  
Security features required to support IPFIX are discussed in section 11. This section discussed the security features the protocol requires, the applicability of TLS and DTLS, the usage of a unique secure channel, the mutual authentication, protection against DOS, and security issues with UDP only, requirement to log IPFIX attack, how to secure the collector, and the need to secure the data collected.

A.3.  Documentation
* Is an operational considerations and/or manageability section part of the document?  Security section is. The document weaves the operational considerations and management within the whole document since this is a protocol that impacts monitoring. The actual management and monitoring of the protocol doing the flow monitoring is part of this discussion. The configuration and data models are (thankfully) elsewhere described. 

* Does the proposed protocol have a significant operational impact on the Internet?  Yes, yes, and yes!

* Is there proof of implementation and/or operational experience? Significant lessons learned are causing the update.  No specific implementation or operational experience is noted.

(Jari Arkko) No Objection

(Richard Barnes) No Objection

(Gonzalo Camarillo) No Objection

(Spencer Dawkins) (was Discuss) No Objection

Comment (2013-06-11 for -07)
No email
send info
Brian provided explanations and text proposals (following), and these proposals address my DISCUSS (cleared) and all but one of my comments. 

On the #3 text proposal below ... we're talking about an Observation Domain using a single TCP connection, is that correct?

If an OD sends a TCP stream that looks like this (each observation is transmitted as a separate IP packet)

- Observation 1
- Observation 2 (assume this one gets lost in the network)
- Observation 3
- Observation 4
- Observation 5

TCP guarantees in-order delivery, so the sending TCP will retransmit Observation 2 tenaciously, and the receiving TCP holds everything it receives after Observation 2 until it can deliver the retransmitted Observation 2. The Collecting Process won't see Observation 3 until after Observation 2 is successfully transmitted.

If two Observation Domains, "a" and "b", are sharing a single TCP connection, and the interleaved TCP stream looks like this:

- Observation a-1
- Observation a-2 (assume this one gets lost in the network)
- Observation a-3
- Observation b-1
- Observation b-2
- Observation b-3
- Observation a-4
- Observation b-4
- Observation b-5
- Observation a-5

the in-order delivery guarantee spans Observation Domains, so not only does the receiving TCP hold everything for Observation Domain "a" until it receives Observation a-2, it holds everything for Observation Domain "b", because all of these observations are being blocked by the same head of line problem with the same TCP connection.

So in this proposed text:

   Exporting Processes may choose IPFIX Message lengths lower than the 
   maximum in order to avoid head-of-line blocking and/or to ensure 
   timely export of Data Records.

I think you can minimize head-of-line blocking, but I don't think you can avoid it. 

Maybe minimizing head-of-line blocking is OK (the Collecting Process is, after all, not a human, so I'm not sure why it would care about head-of-line blocking, as long as it sees reliable in-order delivery in the fullness of time), so I don't want to be difficult, but I did want to try to explain better what I was trying to say.
--

From Brian:

To summarize, I suggest the following changes to the document to address this DISCUSS and COMMENT:

(1) NEW definition for Sequence Number in Section 3.1 (s/SHOULD/can/):

 Sequence Number 

      Incremental sequence counter modulo 2^32 of all IPFIX Data Records
      sent in the current stream from the current Observation Domain by
      the Exporting Process. Each SCTP Stream counts sequence numbers
      separately, while all messages in a TCP connection or UDP
      transport session are considered to be part of the same stream.
      This value can be used by the Collecting Process to identify
      whether any IPFIX Data Records have been missed. Template and
      Options Template Records do not increase the Sequence Number.

(2) NEW Section 10.2.5. Failover (in 10.2 SCTP) (clarify failure detect by lower-layer timeout):

   If the Collecting Process does not acknowledge an attempt by the
   Exporting Process to establish an association, SCTP will automatically
   retry association establishment using exponential backoff. The
   Exporter MAY log an alarm if the underlying SCTP association establishment
   times out; this timeout should be configurable on the Exporter.

   The Exporting Process MAY open a backup SCTP association to a Collecting
   Process in advance, if it supports Collecting Process failover.

(3) NEW Section 10.4.3. MTU paragraph 2 (remove confusing head of line blocking language):

   Exporting Processes may choose IPFIX Message lengths lower than the 
   maximum in order to avoid head-of-line blocking and/or to ensure 
   timely export of Data Records.

(4) NEW Section 10.4.4. Connection Establishment and Shutdown first two paragraphs:

   The IPFIX Exporting Process initiates a TCP connection to the
   Collecting Process. An Exporting Process MAY support more than one
   active connection to different Collecting Processes (including the
   case of different Collecting Processes on the same host). An Exporting
   Process MAY support more than one active connection to the same 
   Collecting Process to avoid head of line blocking across Observation Domains.

   The Exporter MAY log an alarm if the underlying TCP connection establishment
   times out; this timeout should be configurable on the Exporter.

(5) NEW Section 10.4.5. Failover (in 10.4 TCP) (clarify failure detect by lower-layer timeout):

   If the Collecting Process does not acknowledge an attempt by the
   Exporting Process to establish a connection, TCP will automatically
   retry connection establishment using exponential backoff. The
   Exporter MAY log an alarm if the underlying TCP connection establishment
   times out; this timeout should be configurable on the Exporter.

   The Exporting Process MAY open a backup TCP connection to a Collecting
   Process in advance, if it supports Collecting Process failover.

(Adrian Farrel) No Objection

Comment (2013-07-07 for -08)
No email
send info
I hate the idea of tit-for-tat Discusses and will not raise one.
However, having recently seen a number of Discusses entered on documents
saying that insufficient attention had been paid to RFC 5706 and the
manageability issues and impacts for protcols, I must observe that this
document is considerably lacking in that department. I looked for, but
could not find an existing RFC providing a management framework for
IPFIX although I do notice that a MIB module exists. Maybe RFC 5153 is
supposed to cover this, but on a brief skim it does not seem to give 
much information about how IPFIX is managed.

Please note that just because IPFIX is, itself, a management protocol
does not mean that the impact that the impact on te network of its use
should not be considered. Indeed, there are considerable concerns about
how IPFIX could impact a live network. Furthermore, IPFIX needs to be
managed, operated, and diagnosed in its own right, and that bears 
description.

I leave it to the sponsoring AD and document editors to examine why they 
are comfortable to progress this document without this material.

---

Section 8.4

   Default settings for these values are deployment- and
   application-specific.

I.e., there are no default settings?

(Stephen Farrell) No Objection

Comment (2013-06-10 for -07)
No email
send info
- I mostly used the diff to 5101 for this review.

- In section 8, you have new text saying that the CP MUST
store all template record information for the duration of
each transport section. I wondered if that creates any
potential for a DoS?

- Is all the naming stuff in 11.3 fully worked out and likely
to be, or actually, implemented? E.g. it says that use of
DNS-ID is a SHOULD be implementaton of CN based lists is a
MUST - does that make sense really?

- You say: SSL/TLS before TLS1.1 is a MUST NOT, TLS1.1 is a
MUST and TLS1.2 is a SHOULD. Would it not be better to just
say TLS1.2 is a MUST? Perhaps TLS1.2 support is (soon) more
likely to be widely available for this kind of application -
did you check recently? (I didn't, and [1] is a year old
now.)

  [1] http://www.imperialviolet.org/2012/06/08/tlsversions.html 

- I wondered if its possible to analyse network timing to try
to discover what kind of IPFIX collecting is going on, for
example, via timing analysis. Since I don't know, I'm not
asking that you add something to section 11, but I do
wonder:-)

(Brian Haberman) No Objection

Barry Leiba No Objection

(Pete Resnick) No Objection

(Martin Stiemerling) No Objection

Comment (2013-06-11 for -07)
No email
send info
In support of Spencer's DISCUSS point

(Sean Turner) No Objection

Comment (2013-06-10 for -07)
No email
send info
Should probably say something about the version of DTLS you want supported in s11.3 or is that covered in s11.1?

Do you need to up the SHOULD in RFC 6347 about the cookie exchange to a MUST?

(Ted Lemon) (was Discuss) Abstain

Comment (2013-07-11 for -09)
No email
send info
I'm abstaining because I think the IANA registry thing is a bad idea, but nobody else does.   In general I think the document is a good idea, so don't take this too seriously...

FORMER DISCUSS:

I see a lot of cases of definitions for attributes of option templates where the text defining the attribute has been removed since RFC5101, and [IANA-IPFIX] is given as a reference where the definitions can be found.   For example in section 4.3, the definitions for notSentFlowTotalCount, notSentPacketTotalCount and notSentOctetTotalCount.

This seems really weird to me.   If the text is identical in the document and in the registry, it's certainly fair to say that this is redundant.   But the way I would be inclined to address the redundancy would be to keep the text in the IETF standard, and abbreviate the text in the registry, pointing the registry back at the document.  The IANA registry is not a standard—it's a living document, which can change over time, so the explanatory text in it can likewise change over time.   It probably _won't_, but I think relying on that is a bad idea.

To clear this DISCUSS, you would need to give a satisfactory explanation as to why it makes sense to move normative text out of this document into the IANA registry, or you could restore the text to this document, or you could point out another IETF standard that also contains the definitions (and perhaps change the reference in this document to point to that one, rather than to the IANA registry).   Or you could get the other ADs to gang up on me and tell me I'm making a fuss over nothing.   :)

The authors went with the last option above.   :)

COMMENTS:

The document uses the term "treatment," I think as a term of art, but it results in some constructs that are hard to parse, like this one in section 2, Page 8, about halfway down:

  As sampling is  a packet treatment, this definition includes packets selected by a	sampling mechanism.

It might help to add a terminology section on "packet treatment."   However, this depends on the target readership, so I'm by no means demanding that such a section be added; simply suggesting it based on the fact that I stumbled over the term when reading the document.

It seems unnecessary to repeat "network byte order (also known as big-endian byte order)" throughout the document.   It might be better to just say "network byte order (also known as big-endian byte order)" the first time, and just "network byte order" subsequently.

In 6.1.3 and 6.1.4, the floating point format is specified as IEEE 754 in network byte order.   IEEE 754 doesn't specify a network byte ordering for floating point numbers.   I think the encoding is fairly straightforward, but I am not convinced that a reader of this document who did not already know what IEEE 754 encoding in network byte order means would be able to figure it out.   It would help to have a reference here to a document that defines what "network byte order" means here, or just a quick statement about how the bytes are arranged.

In 8.2, there's a paragraph that's supposed to clarify the applicability of templates to data records that's got such a big parse stack it left my head spinning.   I propose the following change:

OLD:
   Put another way, a Template describes Data Records contained in IPFIX
   Messages with an Export Time between the Export Time of the IPFIX
   Message containing the Template Record and either the Export Time of
   the IPFIX Message containing the Template Withdrawal withdrawing it
   or the end of the Transport Session, inclusive.
NEW:
   Put another way, a Template describes Data Records contained in IPFIX
   messages when the export time of such messages is between a
   specific start and end time.   The start time is the Export Time of the IPFIX
   Message containing the Template record.   The end time is one of two
   times.   If the template is withdrawn during the session, then it is the
   Export Time of the IPFIX message containing the Template Withdrawal
   for the template.   Otherwise, it is the end of the Transport Session.

In the last paragraph of the first part of section 9, the text does not mention the handling of a malformed message in the case of TCP transport.   I think in this case there is no way to resynchronize, because it's impossible to characterize the error that caused the malformed message.   That being the case, doesn't the TCP connection have to be dropped when a malformed message is encountered?   Shouldn't this be mentioned either in Section 9 or Section 10.4?

Overall this update to RFC 5101 looks like a significant improvement.   Thanks for doing it!

(Stewart Bryant) Recuse

(Benoît Claise) Recuse