Updates for the Back-to-Back Frame Benchmark in RFC 2544
draft-ietf-bmwg-b2b-frame-04

[Ballot comment]
Hi Al,

Thanks for this document.  I mostly found it easy to read with just a few minor comments:

I agree with Murray's comment that there seems more text capitalized than is potentially helpful.  Possible words/phrases to check: Obsolete, "Back-to-back Frames" except when used for "Back-to-back Frame Benchmark", "Data Center", "Throughput", "Ingress", "Egress", "Frames", "Buffer"

Section 2, para 3:

However,
  conditions simultaneously sending multiple frame sizes, such as those
  described in [RFC6985], MUST NOT be used in Back-to-back Frame
  testing.
 
I found this sentence slightly hard to parse, rephrasing it might aid readability.

In section 6, it states "The number of tests Averaged for the Benchmark, N, MUST be reported."  Should that be illustrated in the "Back-to-Back Frame Results" table, either as part of the table, or as extra information alongside it?

Regards,
Rob

[Ballot comment]
Thank you for staging changes to resolve the tsvart and secdir
reviewers' comments.

I assume that it goes without saying that the computed StdDev are sample
standard deviations, not population standard deviations.

Section 6

What kind of DUT is only going to have 4 microseconds of buffer?

Section 10.1

It is surprising to see a normative reference being made to the Obsolete
RFC 1944.

Likewise, some of the references are only tenuously referenced and would
seem better characterized as informative (e.g., RFC 6201, RFC 6985).

Section 10.2

On the other hand, if the methods of RFC 8239 SHOULD be used in some
cases, that suggests that it should more properly be characterized as a
normative reference.

Similarly, the TST009 binary search is one of the two options from which
a selection MUST be made, which seems to make it normative as well.

[Ballot comment]
Thank you for discussing the feedback from the SECDIR review and thank you to Mališa Vučinić for this review.  The candidate text proposed at https://mailarchive.ietf.org/arch/msg/secdir/2CTnqKT7kwGLLSKMnEUTP5HC8wc/ is helpful (fully recognizing that this is addressing a issue not specified to this particular benchmark update).

[Ballot comment]
There are lots of terms throughout this document that are capitalized when I'm not certain they should be (e.g., "Buffer", "Burst", "Data Center", etc.), and I ended up looking around for specific definitions of them.  You might want to give this a pass to check on all of that, because leaving it to the RFC Editor to sort out will probably slow things down.

[Ballot discuss]
Thank you for engaging with the TSVART review. Despite the wordsmithing that has gone on, I am not sure that we have captured the correct text.

The proposed change is:
> I clarified:
> The duration of the trial MUST include least 2 seconds in addition to the time
> required to send and receive each burst of frames, to ensure that DUT buffers to
> deplete.
>
> and I'll add:
> The upper search limit for the time to send each burst MUST be configurable as
> high as 30 seconds (buffer time results reported at the configured upper limit are
> likely invalid, and the test MUST be repeated with a higher search limit).

But IIUC it's the additional time that needs to scale up. A layman's reading of the document, IMO, suggests that the burst length has a binary search but the 2 seconds of waiting can be fixed.

(Via drafts-lastcall-comment@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has reviewed draft-ietf-bmwg-b2b-frame-03, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any registry actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object.

If this assessment is not accurate, please respond as soon as possible.

Thank you,

Michelle Cotton
Protocol Parameters Engagement Sr. Manager
IANA Services

The following Last Call announcement was sent out (ends 2020-12-04):

From: The IESG
To: IETF-Announce
CC: warren@kumari.net, sbanks@encrypted.net, bmwg-chairs@ietf.org, draft-ietf-bmwg-b2b-frame@ietf.org, bmwg@ietf.org, Sarah Banks
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (Updates for the Back-to-back Frame Benchmark in RFC 2544) to Informational RFC


The IESG has received a request from the Benchmarking Methodology WG (bmwg)
to consider the following document: - 'Updates for the Back-to-back Frame
Benchmark in RFC 2544'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2020-12-04. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  Fundamental Benchmarking Methodologies for Network Interconnect
  Devices of interest to the IETF are defined in RFC 2544.  This memo
  updates the procedures of the test to measure the Back-to-back frames
  Benchmark of RFC 2544, based on further experience.

  This memo updates Section 26.4 of RFC 2544.





The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-bmwg-b2b-frame/



No IPR declarations have been submitted directly on this I-D.

... I just noticed that I hopped straight from PubReq to IESG eval, skipping IETF LC... while this might make life easier, someone will complain :-)
This'll larn me to try progress documents while on BKK timezone and listening to other sessions...

As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated 1 November 2019.

(1) What type of RFC is being requested: Informational

(2) Document Announcement

Technical Summary: Fundamental Benchmarking Methodologies for Network Interconnect Devices of interest to the IETF are defined in RFC 2544.  This memo updates the procedures of the test to measure the Back-to-back frames Benchmark of RFC 2544, based on further experience.

Working Group Summary:

The draft had ample input and review from the Working Group, and Consensus was clear. We did not experience dissent or major disagreement; the process was fairly smooth.

Document Quality:

This is a testing methodology; there's no issue of running code or existing implementation.  The document is ready for publication

Personnel:

Sarah Banks is the Document Shepherd? Warren Kumari is the Responsible Area Director?

(3) As the WG Co-Chair, I read and review most drafts coming into the working group, and ALL of the drafts written by my co-chair, as is this draft. In addition, the draft has received numerous reviews and comments from WG members. The document is well written, has clean nits (1 issue for date, which is fine), and is ready for publication in the view of the WG.

(4) I do not have any concerns about the depth or breadth of the reviews that have been performed.

(5) This work was well contained and updates RFC 2544 based on our further experience, an RFC that originated in this same WG. There is no need for external WG input in my opinion, and we did not solicit any extra review outside of the WG.

(6) I have no specific concerns or issues that the  the Responsible Area Director and/or the IESG should be aware of.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed? Yes.

(8) No IPR disclosures have been made.

(9) How solid is the WG consensus behind this document? We have strong consensus for this document.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? No threats of appeals or otherwise have been made or are expected.

(11) Identify any ID nits the Document Shepherd has found in this document. Folks can run nits, however, the main issue raised was the date (May 2020), which is fine.

(12) No other formal review required.

(13) Have all references within this document been identified as either normative or informative? Yes.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? Nothing is in an unclear state.

(15) Are there downward normative references references (see RFC 3967)? No.

(16) Will publication of this document change the status of any existing RFCs? This draft updates RFC2544.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. This is NA/ there are no protocol changes required here.

(18) List any new IANA registries that require Expert Review for future allocations. N/A

(19) This is a testing methodology draft, not a protocol proposal or likewise.

(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools? N/A

As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated 1 November 2019.

(1) What type of RFC is being requested: Informational

(2) Document Announcement

Technical Summary: Fundamental Benchmarking Methodologies for Network Interconnect Devices of interest to the IETF are defined in RFC 2544.  This memo updates the procedures of the test to measure the Back-to-back frames Benchmark of RFC 2544, based on further experience.

Working Group Summary:

The draft had ample input and review from the Working Group, and Consensus was clear. We did not experience dissent or major disagreement; the process was fairly smooth.

Document Quality:

This is a testing methodology; there's no issue of running code or existing implementation.  The document is ready for publication

Personnel:

Sarah Banks is the Document Shepherd? Warren Kumari is the Responsible Area Director?

(3) As the WG Co-Chair, I read and review most drafts coming into the working group, and ALL of the drafts written by my co-chair, as is this draft. In addition, the draft has received numerous reviews and comments from WG members. The document is well written, has clean nits (1 issue for date, which is fine), and is ready for publication in the view of the WG.

(4) I do not have any concerns about the depth or breadth of the reviews that have been performed.

(5) This work was well contained and updates RFC 2544 based on our further experience, an RFC that originated in this same WG. There is no need for external WG input in my opinion, and we did not solicit any extra review outside of the WG.

(6) I have no specific concerns or issues that the  the Responsible Area Director and/or the IESG should be aware of.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed? Yes.

(8) No IPR disclosures have been made.

(9) How solid is the WG consensus behind this document? We have strong consensus for this document.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? No threats of appeals or otherwise have been made or are expected.

(11) Identify any ID nits the Document Shepherd has found in this document. Folks can run nits, however, the main issue raised was the date (May 2020), which is fine.

(12) No other formal review required.

(13) Have all references within this document been identified as either normative or informative? Yes.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? Nothing is in an unclear state.

(15) Are there downward normative references references (see RFC 3967)? No.

(16) Will publication of this document change the status of any existing RFCs? This draft updates RFC2544.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. This is NA/ there are no protocol changes required here.

(18) List any new IANA registries that require Expert Review for future allocations. N/A

(19) This is a testing methodology draft, not a protocol proposal or likewise.

(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools? N/A