Traffic Management Benchmarking
draft-ietf-bmwg-traffic-management-06

[Ballot comment]
Thanks for your work on this draft.  I just have one question that was raised in the SecDir review:

Was there any thought to how the tcpinc efforts might affect the ability to gather measurements?
http://www.ietf.org/mail-archive/web/secdir/current/msg05726.html

It doesn't have an impact on this draft yet, but could in the future.

[Ballot comment]
Section 6.2.1 Queue/Scheduler Individual Tests Overview
I understand that is not finished work, but do we need to say a few words about codel and pie
http://datatracker.ietf.org/doc/draft-ietf-aqm-pie/ and http://datatracker.ietf.org/doc/draft-ietf-aqm-codel/

- More of a transport AD question:
"To this end, the stateful tests will use TCP test patterns to emulate applications."
Do we need to emulate any applications on SCTP test patterns?
Asking the transport ADs about which deployed protocols use SCTP, their answer was: WebRTC, SS7 signalling in 3gpp mobile networks.


Editorial:

- Section 1:

  Specifically, this framework defines the methods to characterize
  the capacity of the following traffic management features in network
  devices; classification, policing, queuing / scheduling, and
  traffic shaping.

Section 1.2

  This testing framework describes the tests and metrics for each of
  the following traffic management functions:
  - Policing
  - Queuing / Scheduling
  - Shaping

I would add classification to that list.

-  Section 1.2
    "Note the NDE SHOULD be used in "full pipe" delay mode."

I don't know what "full pipe" delay mode is

- OLD:
It is not within scope of this of this framework to specify
NEW:
It is not within scope of this framework to specify



- General observation. This is one of these documents where too many acronyms make it difficult to read (and I have some QoS background)
Ex: "The tests will verify that the network device can handle the CIR with CBS and the EIR with EBS
Note: I don't expect any action on this remark

- Section 6:

  Each test SHOULD compare the network device's internal statistics
  (available via command line management interface, SNMP, etc.) to the
  measured metrics defined in section 4.

FYI. Most MIB interface/QoS counters are not updated real-time in routers. 10 sec for interface stats is common.
So no real-time monitoring is possible.

- two instances of tester*: not sure what they mean
    configure the tester* to generate a profile of emulated of an application traffic mixture

IESG/Authors/WG Chairs:

IANA has reviewed draft-ietf-bmwg-traffic-management-04, which is currently in Last Call, and has the following comments:

We understand that, upon approval of this document, there are no IANA Actions that need completion.

While it is helpful for the IANA Considerations section of the document to remain in place upon publication, if the authors prefer to remove it, IANA doesn't object.

If this assessment is not accurate, please respond as soon as possible.

This is the Publication Request and document shepherd write-up for   

Traffic Management Benchmarking
  https://tools.ietf.org/html/draft-ietf-bmwg-traffic-management-03

This version is dated 24 February 2012.

Al Morton is the Document Shepherd, and prepared this form: March 2015.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational, as indicated on the title page.
All BMWG RFCs are traditionally Informational,
in part because they do not define protocols and
the traditional conditions for Stds track advancement
did not apply.  However, they are specifications and
the RFC 2119 terms are applicable to identify the
level of requirements.

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

This memo specifies the methodology for benchmarking the
traffic management capabilities of networking devices (i.e. policing,
shaping, etc.). One challenge has been to provide a repeatable test method that
objectively compares performance of the device's traffic management
capabilities and to specify the means to benchmark traffic management
with representative traffic, including stateless and statefull traffic.
The consensus is that the challenge has been met, and its time to
gain more experience with the procedures as specified.

Working Group Summary:

There has been fairly continuous progress during development. The most lively
discussions were prompted by presentation of actual test results using
the draft methods, which require significant time investment but are well-
worth the result. These drafts serve a useful purpose for the industry.


Document Quality:

Are there existing implementations of the protocol? Have a significant number of vendors indicated their plan to implement the specification? Are there any reviewers that merit special mention as having done a thorough review, e.g., one that resulted in important changes or a conclusion that the document had no substantive issues? If there was a MIB Doctor, Media Type or other expert review, what was its course (briefly)? In the case of a Media Type review, on what date was the request posted?

As far as expert reviews, there was substantial review from several industry professionals,
most notably one a CCNP, one a professional telecom industry consultant,
and several from a vendor (Brocade).

Personnel:

Who is the Document Shepherd? Who is the Responsible Area Director?

Al Morton is Shepherd, Joel Jaeggli is AD.

(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

I've reviewed this draft at each of three WGLCs. Nits check is now clean.
AFAICT, all my comments and those of the WG have been addressed.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

There was cross-area review with the AQM WG, good comments were addressed.
ALl WGLC have been cross posted to AQM WG.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

None.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

There is no IPR claimed, according to the authors.


(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No IPR disclosed.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Although the comments and review intensity started strong,
it now appears that the WG is satisfied.
The third WGLC on -02 closed on 19 Feb 2015 with a small number
of comments.
The second WGLC (on -01) closed on 6 January, 2015 with substantial
comments.
The first WGLC (on -00) closed October 21 2014, with substantial
comments.


(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

No nits

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, media type, and URI type reviews.

NA

(13) Have all references within this document been identified as either normative or informative?

All references are Normative, and marked as such.
There appear to be some missing references, to MEF-14 MEF-19 and MEF-37.

This will be fixed in next rev (04).

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

No

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

No

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 5226).

There are no IANA actions required,
although the wording of the IANA section is
somewhat unusual.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, etc.

NA.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (Traffic Management Benchmarking) to Informational RFC


The IESG has received a request from the Benchmarking Methodology WG
(bmwg) to consider the following document:
- 'Traffic Management Benchmarking'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2015-04-20. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This framework describes a practical methodology for benchmarking the
  traffic management capabilities of networking devices (i.e. policing,
  shaping, etc.). The goal is to provide a repeatable test method that
  objectively compares performance of the device's traffic management
  capabilities and to specify the means to benchmark traffic management
  with representative application traffic.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-traffic-management/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-traffic-management/ballot/


No IPR declarations have been submitted directly on this I-D.

This is the Publication Request and document shepherd write-up for   

Traffic Management Benchmarking
  https://tools.ietf.org/html/draft-ietf-bmwg-traffic-management-03

This version is dated 24 February 2012.

Al Morton is the Document Shepherd, and prepared this form: March 2015.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational, as indicated on the title page.
All BMWG RFCs are traditionally Informational,
in part because they do not define protocols and
the traditional conditions for Stds track advancement
did not apply.  However, they are specifications and
the RFC 2119 terms are applicable to identify the
level of requirements.

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

Relevant content can frequently be found in the abstract and/or introduction of the document. If not, this may be an indication that there are deficiencies in the abstract or introduction.

Working Group Summary:

There has been fairly continuous progress during development. The most lively
discussions were prompted by presentation of actual test results using
the draft methods, which require significant time investment but are well-
worth the result. These drafts serve a useful purpose for the industry.


Document Quality:

Are there existing implementations of the protocol? Have a significant number of vendors indicated their plan to implement the specification? Are there any reviewers that merit special mention as having done a thorough review, e.g., one that resulted in important changes or a conclusion that the document had no substantive issues? If there was a MIB Doctor, Media Type or other expert review, what was its course (briefly)? In the case of a Media Type review, on what date was the request posted?

As far as expert reviews, there was substantial review from several industry professionals,
most notably one a CCNP, one a professional telecom industry consultant,
and several from a vendor (Brocade).

Personnel:

Who is the Document Shepherd? Who is the Responsible Area Director?

Al Morton is Shepherd, Joel Jaeggli is AD.

(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

I've reviewed this draft at each of three WGLCs. Nits check is now clean.
AFAICT, all my comments and those of the WG have been addressed.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

There was cross-area review with the AQM WG, good comments were addressed.
ALl WGLC have been cross posted to AQM WG.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

None.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

There is no IPR claimed, according to the authors.


(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No IPR disclosed.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Although the comments and review intensity started strong,
it now appears that the WG is satisfied.
The third WGLC on -02 closed on 19 Feb 2015 with a small number
of comments.
The second WGLC (on -01) closed on 6 January, 2015 with substantial
comments.
The first WGLC (on -00) closed October 21 2014, with substantial
comments.


(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

No nits

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, media type, and URI type reviews.

NA

(13) Have all references within this document been identified as either normative or informative?

All references are Normative, and marked as such.
There appear to be some missing references, to MEF-14 MEF-19 and MEF-37.

This will be fixed in next rev (04).

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

No

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

No

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 5226).

There are no IANA actions required,
although the wording of the IANA section is
somewhat unusual.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, etc.

NA.

All BMWG RFCs have been Informational due to lack of applicable process on the standards track. However, these drafts contain requirements as testing specifications, and make use of RFC2119 language (which originated in BMWG).