Skip to main content

Benchmarking Methodology for EVPN and PBB-EVPN
draft-ietf-bmwg-evpntest-11

Revision differences

Document history

Date Rev. By Action
2024-02-18
11 (System) Document has expired
2024-01-26
11 Gunter Van de Velde Request closed, assignment withdrawn: Jon Mitchell Last Call OPSDIR review
2024-01-26
11 Gunter Van de Velde Closed request for Last Call review by OPSDIR with state 'Overtaken by Events': Cleaning up stale OPSDIR queue
2023-08-17
11 sudhin jacob New version available: draft-ietf-bmwg-evpntest-11.txt
2023-08-17
11 (System) New version approved
2023-08-17
11 (System) Request for posting confirmation emailed to previous authors: Kishore Tiruveedhula , sudhin jacob
2023-08-17
11 sudhin jacob Uploaded new revision
2023-08-16
10 Kishore Tiruveedhula New version available: draft-ietf-bmwg-evpntest-10.txt
2023-08-16
10 (System) New version approved
2023-08-16
10 (System) Request for posting confirmation emailed to previous authors: Kishore Tiruveedhula , sudhin jacob
2023-08-16
10 Kishore Tiruveedhula Uploaded new revision
2022-02-15
09 (System) Document has expired
2022-02-14
09 Warren Kumari
Unable to reach authors, and requests for volunteers produced none.

More background: https://mailarchive.ietf.org/arch/msg/bmwg/q_QpkXNK2YcNxMqix3EhrY-3O0w/


Minutes - IETF112:
---
- Benchmarking Methodology for EVPN and PBB-EVPN
  …
Unable to reach authors, and requests for volunteers produced none.

More background: https://mailarchive.ietf.org/arch/msg/bmwg/q_QpkXNK2YcNxMqix3EhrY-3O0w/


Minutes - IETF112:
---
- Benchmarking Methodology for EVPN and PBB-EVPN
  https://datatracker.ietf.org/doc/draft-ietf-bmwg-evpntest/
  status: IESG Review on July 1, 2021; Resolving DISCUSS and Comment
ballots
  AD Advisor has imposed a *Dead*-line (Oct 31)

  >>> Question whether anyone else in the group will help with this
DISCCUSS, And AUTH48???
  >>> All attempts to reach authors have failed.
  >>> Sarah will reach out to the co-authors again.
----

Video: https://www.youtube.com/watch?v=mUZKhuqa5BY&t=411s
2022-02-14
09 (System) Removed all action holders (IESG state changed)
2022-02-14
09 Warren Kumari IESG state changed to Dead from IESG Evaluation::Revised I-D Needed
2022-01-28
09 Warren Kumari
Document discussed at IETF112:
---

- Benchmarking Methodology for EVPN and PBB-EVPN
  https://datatracker.ietf.org/doc/draft-ietf-bmwg-evpntest/
  status: IESG Review on July 1, 2021; Resolving DISCUSS and …
Document discussed at IETF112:
---

- Benchmarking Methodology for EVPN and PBB-EVPN
  https://datatracker.ietf.org/doc/draft-ietf-bmwg-evpntest/
  status: IESG Review on July 1, 2021; Resolving DISCUSS and Comment ballots
  AD Advisor has imposed a *Dead*-line (Oct 31)
 
  >>> Question whether anyone else in the group will help with this DISCCUSS, And AUTH48???
  >>> All attempts to reach authors have failed.
  >>> Sarah will reach out to the co-authors again.

---

Asking for new authors by 2022-02-04, otherwise it is dead.
2021-09-15
09 Warren Kumari 2021-09-15 - Another reminder sent to authors.
2021-08-26
09 Warren Kumari Changed action holders to Kishore Tiruveedhula, sudhin jacob (Better tracking of who holds the ball - waiting on the authors)
2021-08-24
09 Warren Kumari 2021-08-24 -- Sent reminder email to authors.
2021-07-01
09 Cindy Morgan Changed consensus to Yes from Unknown
2021-07-01
09 (System) Changed action holders to Warren Kumari, Kishore Tiruveedhula, sudhin jacob (IESG state changed)
2021-07-01
09 Cindy Morgan IESG state changed to IESG Evaluation::Revised I-D Needed from IESG Evaluation
2021-07-01
09 Zaheduzzaman Sarker
[Ballot discuss]

Thanks for the attempt here, defining test cases are tricky.

I think to use the test cases in this document effectively, followings need …
[Ballot discuss]

Thanks for the attempt here, defining test cases are tricky.

I think to use the test cases in this document effectively, followings need to correctly specified/addressed -

* All the tests are said to be run for N times and collect N samples. Are those N same? What should be the default value for the N samples? How do we decide that there are sufficient samples? I find it a bit odd that there is no recommendation provided here. There should be a min and max range of values the N can take so that we know we have done enough tests to conclude on the results. Is there are specific waiting time required between running those tests N times? This feel like very under specified to me.

* Section 3.10: what is the exit criteria of this test? how do we know the test was successful?

* Section 3.12: I am missing a recommended duration of running the test here. I am supposed to take hourly reading but for how long time I should do that to get to an conclusion?

* May be I missed it but I haven't found any mention of non-test traffic if they are allowed during the test, if yes what kind of considerations should be  taken when interpreting the results. If they are not allowed during the test it should be written as well.

I also support Martin and Eric's discuss.
2021-07-01
09 Zaheduzzaman Sarker
[Ballot comment]

Some non-blocking comments -

* Section 2:
  ** please expand abbreviated words in their first occurrence if not listed in well known …
[Ballot comment]

Some non-blocking comments -

* Section 2:
  ** please expand abbreviated words in their first occurrence if not listed in well known terminologies. See the list : https://www.rfc-editor.org/materials/abbrev.expansion.txt
  ** why is this called topology 1? are expected to have more typologies? Where are they? and it is actually not clear to me what is topology 1.

* Section 3.8: what is PPS?

* Section 3.9: how do you measure packet loss? is this loss measured for the whole run or with with a time window?

* Section 4.9 : "Then increment the scale of N by 5% of N till the limit is reached." Please clarify if the increment steps always 5% of initial value or N or N = N+5% of N.

Nits :

* Section 2 : Figure 2 - s/SHPFE3/SHPE3 or define SHPFE3.
2021-07-01
09 Zaheduzzaman Sarker [Ballot Position Update] New position, Discuss, has been recorded for Zaheduzzaman Sarker
2021-07-01
09 Robert Wilton
[Ballot comment]
Hi,

I would like to thank the authors for the work that they have put into this document, and also Sarah for the …
[Ballot comment]
Hi,

I would like to thank the authors for the work that they have put into this document, and also Sarah for the doc shepherd work that she did on improving the text.

A few nits that I spotted:

Sometimes this doc refers to the just "MAC" and sometimes "MAC address".  The prose would probably flow better if these were always "MAC address".

The requirements boilerplate text is section 1.1, doesn't look quite right, and should probably be:

  The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
  "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
  "OPTIONAL" in this document are to be interpreted as described in BCP
  14
[RFC2119] [RFC8174] when, and only when, they appear in all
  capitals, as shown here.

In section 1.2:
  IRB: Integrated routing and bridging interface
Normally, I would expect IRB to just be "Integrated Routing and Bridging".  Given this term is only referenced once I would suggest changing the term and moving the "interface" to where the term is used (if needed).

In Section 3.2, it refers to the "data plane", but that should be "control plane".

Regards,
Rob
2021-07-01
09 Robert Wilton [Ballot Position Update] New position, No Objection, has been recorded for Robert Wilton
2021-07-01
09 Martin Vigoureux
[Ballot comment]
It's not clear to me what this document is trying to achieve.
It is clear that it defines things to measure, but
- …
[Ballot comment]
It's not clear to me what this document is trying to achieve.
It is clear that it defines things to measure, but
- why these and not something else?
- what do the values reflect?
- what can be concluded using these measurements?
- what might affect the results obtained?
Also, all these measurements seems to be done in an idealistic situation where the systems are not under any other stress than the one needed to do the measure.



NITS
  All-Active Redundancy Mode: When all PEs attached to an Ethernet
  segment are allowed to forward known unicast traffic to/from that
  Ethernet segment for a given VLAN, then the Ethernet segment is
  defined to be operating in All-Active redundancy mode.

  AA: All Active mode
This doesn't seem to be used

  RT: TrafficGenerator.
This is never used and should be removed


VLAN should be capitalized


s/measured.The/measured. The/
s/support,Interior/support, Interior/
s/parameters.It/parameters. It/
s/standrard/standard/
s/sample.The/sample. The/
s/this documents/this document/
s/routes.But/routes.But/
s/process,CPU/process,CPU/
2021-07-01
09 Martin Vigoureux [Ballot Position Update] New position, No Objection, has been recorded for Martin Vigoureux
2021-06-30
09 Murray Kucherawy
[Ballot discuss]
[This is largely a repeat of Eric's DISCUSS position, which I noticed only after I wrote mine up.  I'll leave it as-is for …
[Ballot discuss]
[This is largely a repeat of Eric's DISCUSS position, which I noticed only after I wrote mine up.  I'll leave it as-is for the moment.]

The tests in Sections 3.3, 3.4, 4.3, and 4.4 each define two distinct things to be measured, but they appear to refer to themselves using a common name.  Feels like some text copy-paste happened here from previous sections, but wasn't fully developed.  For instance, Section 3.3 says:

  Measure the time taken for flushing these X MAC addresses.  Measure
  the time taken to relearn these X MAC in DUT.  The test is repeated
  for N times and the values are collected.  The flush and the
  relearning time is calculated by averaging the values obtained by N
  samples.  N is an arbitrary number to get a sufficient sample.  The
  time measured for each sample is denoted by T1,T2...Tn.  The
  measurement is carried out using external server which polls the DUT
  using automated scripts which measures the MAC counter value with
  timestamp.

  Flush rate = (T1+T2+..Tn)/N

  Relearning rate = (T1+T2+..Tn)/N

So you finish the definition appearing to define two distinct things using the same arithmetic over a single time series.  In other words, you call two different time series by the same name, and then give an identical formal definition for what are really two different measurements.

It seems to me what you actually meant was for these to be, maybe, "F1+F2+..+Fn" for the flush rate and "R1+R2+..+Rn" for the relearning rate, which makes it clear they are distinct quantities.
2021-06-30
09 Murray Kucherawy Ballot discuss text updated for Murray Kucherawy
2021-06-30
09 Murray Kucherawy
[Ballot discuss]
The tests in Sections 3.3, 3.4, 4.3, and 4.4 each define two distinct things to be measured, but they appear to refer to …
[Ballot discuss]
The tests in Sections 3.3, 3.4, 4.3, and 4.4 each define two distinct things to be measured, but they appear to refer to themselves using a common name.  Feels like some text copy-paste happened here from previous sections, but wasn't fully developed.  For instance, Section 3.3 says:

  Measure the time taken for flushing these X MAC addresses.  Measure
  the time taken to relearn these X MAC in DUT.  The test is repeated
  for N times and the values are collected.  The flush and the
  relearning time is calculated by averaging the values obtained by N
  samples.  N is an arbitrary number to get a sufficient sample.  The
  time measured for each sample is denoted by T1,T2...Tn.  The
  measurement is carried out using external server which polls the DUT
  using automated scripts which measures the MAC counter value with
  timestamp.

  Flush rate = (T1+T2+..Tn)/N

  Relearning rate = (T1+T2+..Tn)/N

So you finish the definition appearing to define two distinct things using the same arithmetic over a single time series.  In other words, you call two different time series by the same name, and then give an identical formal definition for what are really two different measurements.

It seems to me what you actually meant was for these to be, maybe, "F1+F2+..+Fn" for the flush rate and "R1+R2+..+Rn" for the relearning rate, which makes it clear they are distinct quantities.
2021-06-30
09 Murray Kucherawy
[Ballot comment]
I support Martin and Eric's DISCUSS positions about how some of the tests are characterized.

Like Eric, I found the number of typos, …
[Ballot comment]
I support Martin and Eric's DISCUSS positions about how some of the tests are characterized.

Like Eric, I found the number of typos, capitalization mistakes, and sentence fragments to be distracting.

Though its coverage of the WG process was good, the shepherd writeup declares the document status but avoided answering the other questions about its status (e.g., "Why is this the proper type of RFC?").  This seems to be a common omission, and I'm starting to wonder if we should just drop it from the template.

I think RFC 7432 and RFC 7623 should be normative, insofar as this document doesn't really make sense unless one understands those first.

Please use the correct boilerplate for BCP 14.  (See RFC 8174, Section 2.)  However, I also note that the only place the key words are used is Section 7, so it occurs to me you could avoid using BCP 14 altogether and basically say the same thing with the same force.

Section 1.2 defines "All-Active Redundancy Mode", but it's not used anywhere in this document other than its own definition.  Similarly, "AA", "ES", "Ethernet Tag", "RT", and "Sub interface" don't appear anywhere else.  "Single-Active Redundancy Mode" could be folded into the definition of "SA", since the latter is actually used elsewhere.
2021-06-30
09 Murray Kucherawy [Ballot Position Update] New position, Discuss, has been recorded for Murray Kucherawy
2021-06-30
09 Amanda Baber IANA Review state changed to IANA OK - No Actions Needed from Version Changed - Review Needed
2021-06-30
09 Roman Danyliw
[Ballot comment]
Thank you to Robert Sparks for the SECDIR review.

** Section 3.* and Section 4.*  Are there recommended values for X, N, or …
[Ballot comment]
Thank you to Robert Sparks for the SECDIR review.

** Section 3.* and Section 4.*  Are there recommended values for X, N, or F?  If this document is intended to help others benchmark EVPNs, it would useful to describe how to calculate the number of times to run the test or how many MACs to generate to get useful results.

** Section 3 and 4.  If all the metrics use Topology 1, and this is the only topology provided in the document, why repeat it each time in the description of each metric?

** Section 7.  Thanks for adding the text “Security features mentioned in the RFC 7432 will affect the test results” in response to the SECDIR review.  Is it possible to clarify this text?  Which RFC7432 security mechanisms are assumed to be in Topology 1?  Could a summary cross-walk of the RFC7432 reference security mechanisms against their impact on the metrics be easily documented?

** idnits returns:

== The document seems to lack the recommended RFC 2119 boilerplate, even if
    it appears to use RFC 2119 keywords -- however, there's a paragraph with
    a matching beginning. Boilerplate error?

  == Missing Reference: 'RFC7632' is mentioned on line 111, but not defined

  == Unused Reference: 'RFC2544' is defined on line 1248, but no explicit
    reference was found in the text

  == Unused Reference: 'RFC2899' is defined on line 1253, but no explicit
    reference was found in the text

  == Unused Reference: 'RFC7623' is defined on line 1268, but no explicit
    reference was found in the text

** Editorial nits:
-- Whole document.  Editorial.  Choose either “VLAN” or “vlan” and use it consistently in the document.

-- Section 2.  Typo. s/support,Interior/support, Interior/

-- Section 2.  Typo. s/parameters.It/parameters. It/

-- Section 3.*. Typo in a few places. s/standrard/standard/g

-- Section 4.11. Typo s/process,CPU/process, CPU/
2021-06-30
09 Roman Danyliw Ballot comment text updated for Roman Danyliw
2021-06-30
09 Roman Danyliw
[Ballot comment]
Thank you to Robert Sparks for the SECDIR review.

** Section 3.* and Section 4.*  Are there a recommend values for X, N, …
[Ballot comment]
Thank you to Robert Sparks for the SECDIR review.

** Section 3.* and Section 4.*  Are there a recommend values for X, N, or F?  If this document is intended to help others benchmark EVPNs, it would useful to describe how to calculate the number of times to run the test or how many MACs to generate to get useful results.

** Section 3 and 4.  If all the metrics use Topology 1, and this is the only topology provided in the document, why repeat it each time in the description of each metric?

** Section 7.  Thanks for adding the text “Security features mentioned in the RFC 7432 will affect the test results” in response to the SECDIR review.  Is it possible to clarify this text?  Which RFC7432 security mechanisms are assumed to be in Topology 1?  Could a summary cross-walk of the RFC7432 reference security mechanisms against their impact on the metrics be easily documented?

** idnits returns:

== The document seems to lack the recommended RFC 2119 boilerplate, even if
    it appears to use RFC 2119 keywords -- however, there's a paragraph with
    a matching beginning. Boilerplate error?

  == Missing Reference: 'RFC7632' is mentioned on line 111, but not defined

  == Unused Reference: 'RFC2544' is defined on line 1248, but no explicit
    reference was found in the text

  == Unused Reference: 'RFC2899' is defined on line 1253, but no explicit
    reference was found in the text

  == Unused Reference: 'RFC7623' is defined on line 1268, but no explicit
    reference was found in the text

** Editorial nits:
-- Whole document.  Editorial.  Choose either “VLAN” or “vlan” and use it consistently in the document.

-- Section 2.  Typo. s/support,Interior/support, Interior/

-- Section 2.  Typo. s/parameters.It/parameters. It/

-- Section 3.*. Typo in a few places. s/standrard/standard/g

-- Section 4.11. Typo s/process,CPU/process, CPU/
2021-06-30
09 Roman Danyliw [Ballot Position Update] New position, No Objection, has been recorded for Roman Danyliw
2021-06-30
09 Éric Vyncke
[Ballot discuss]
Thank you for the work put into this document. I have some regrets about the amount of typos, bad capitalizations, ...

Special thanks …
[Ballot discuss]
Thank you for the work put into this document. I have some regrets about the amount of typos, bad capitalizations, ...

Special thanks for Sarah Banks' shepherd write-up about the WG process / consensus.

Please find below 3 blocking DISCUSS points (but should be easy to address), some non-blocking COMMENT points (but replies would be appreciated), and some nits.

I hope that this helps to improve the document,

Regards,

-éric


== DISCUSS ==

Most of the tests are labelled 'rate' but what they measure is not 'rate' but 'time'. What did I fail to understand ?

-- Sections 3.3, 3.4, and 4.3 --
Perhaps did I fail to understand the purpose of this test but how can it be that "Flush Rate" is equal to "Relearning Rate" ? Should there be different T1 for flush / relearn ?

-- Section 3.9 --
Please s/ip/IPv4/
2021-06-30
09 Éric Vyncke
[Ballot comment]
== COMMENTS ==

As most of the test in section 4 are identical to those in section 3, is there a reason for …
[Ballot comment]
== COMMENTS ==

As most of the test in section 4 are identical to those in section 3, is there a reason for having such repetition ? I.e., rather than having a single set of tests and mentioning the applicability of those tests to the PBB-EVPN scenarios.

-- Abstract --
Suggest to expand PBB as https://www.rfc-editor.org/materials/abbrev.expansion.txt does not have the compound PBB-EVPN. Same applies in section 1 where the expansion is used but not defined.


-- Section 2 --
Suggest to mention early that SH = single home and MH = multi-home

Strange use of 'traffic generator' for a box that also receives traffic. Is there a better word in BMWG ?

Unclear from figure 1 whether the link between SHPE3 and traffic generator acts as a "PE-CE link" as this link is the only unqualified one.

-- Section 3.1 --
In "X different source and destination MAC address for one vlan" (beside the singular form for "address") aren't X different source addresses enough to trigger the MAC learning ? I.e., no need to vary the destination MAC addresses.

Using external scripts to count learned MAC addresses appears very rudimentary and not very accurate...

-- Section 3.2 --
Suggest to use the same text or similar presentation for the measurement part as in section 3.1

-- Section 3.9 --
As it can be expected that most of the EVPN are dual-stack, I wonder whether the 2 single-stack measurements are useful as there could be an interaction between IPv4 and IPv6 learning. Unsure how to test it though (perhaps 50% IPv4 and 50 IPv6 ?).

-- Section 3.12 --
If "SOAK" is an acronym (based on the all uppercase), then please expand it, else use "Soak"

== NITS ==

Please always use the all uppercase character for VLAN same for 'ipv6', 'arp', 'ip', ...

Also check punctuations as they should be followed by a space character.

-- Section 5 --
You may want to add a last name to Al in "to thank Al"
2021-06-30
09 Éric Vyncke [Ballot Position Update] New position, Discuss, has been recorded for Éric Vyncke
2021-06-29
09 Erik Kline
[Ballot discuss]
[S3.9]

* I think this should probably be easy to clear up, likely by clarifying
  the scope of the test.  The general …
[Ballot discuss]
[S3.9]

* I think this should probably be easy to clear up, likely by clarifying
  the scope of the test.  The general point I'd like to discuss is that ND
  machinery has many more test cases than ARP REPLYs do, and it would be
  good to clarify what is to be tested.  For that matter, ARP has fun cases
  too.

  The test in S3.9 seems to be for L2 unicast traffic ("[s]end ... to the
  target IRB address"), but should there be tests for learning in other
  scenarios?

    (1) L2 broadcast Gratuitous ARP REQUESTs (GARP)?

    (2) L3 Multicast NAs to all-nodes (Solicited flag = 0)

    (3) L3 Multicast NAs to all-routers (Solicited flag = 0)

  My guess is it's simplest to craft text to just say the test covers
  unicast ARP REPLYs and unicast ND NAs, but the option exists to expand
  the test matrix as well, I suppose.
2021-06-29
09 Erik Kline
[Ballot comment]
[S3.3] [question]

* "Fail the DUT-CE link" means to cause a link failure that can be detected
  by the DUT (e.g., a …
[Ballot comment]
[S3.3] [question]

* "Fail the DUT-CE link" means to cause a link failure that can be detected
  by the DUT (e.g., a lost of PHY signal or some means to detect a
  directly-connected cable was removed)?

[S3.9] [question]

* What does "Send X arp/neighbour discovery(ND)" mean, exactly?

  If these are meant to be ARP REPLY/Neighbor Advertisements (NAs) then
  being explicit about that seems helpful.

  Or are these ARP REQUESTs and NS Neighbor Solicitations for the IRB address?

* "Send ... to the target IRB address configured in EVPN instance" seems
  like the ARP REPLY/ND NAs are to be sent unicast.  If that's true, I
  think it would improve clarity to say they're unicast.
2021-06-29
09 Erik Kline [Ballot Position Update] New position, Discuss, has been recorded for Erik Kline
2021-06-28
09 Martin Duke
[Ballot discuss]
(3.8) (4.8) Why is packet loss measured in time? How is learning 2X MAC addresses relevant to the packet loss measurement at the …
[Ballot discuss]
(3.8) (4.8) Why is packet loss measured in time? How is learning 2X MAC addresses relevant to the packet loss measurement at the traffic generator?
How long does the traffic generator have to wait to conclude that the packet is lost?

(3.9) Is a single failure to learn an address sufficient to determine that the device has reached capacity? Or could packet loss or some other phenomenon lose some addresses? In other words, be more precise on how polling reveals the capacity.

Is there some lower bound on the time between sending ARP/ND packets and querying the DUT?

(3.11, 3.12, 4.10, 4.11) Does the traffic generator send F frames in total or F ffs? The spec says both. Are there any constraints on F, perhaps an integer multiple of X?
2021-06-28
09 Martin Duke
[Ballot comment]
(3.10, 4.9) Again, is there a minimum time between sending the traffic and querying the result?

(3.12, 4.11) I don’t believe you’ve adequately …
[Ballot comment]
(3.10, 4.9) Again, is there a minimum time between sending the traffic and querying the result?

(3.12, 4.11) I don’t believe you’ve adequately addressed Al’s TSVART review. What does “100% compared to the average usage” mean? Is that double? Shouldn’t there be a formula to compute average usage?

As Al asks, what is the threshold over which an increase in memory usage will fail the test?
2021-06-28
09 Martin Duke [Ballot Position Update] New position, Discuss, has been recorded for Martin Duke
2021-06-28
09 Alvaro Retana [Ballot comment]
The datatracker should be updated to indicate that this document replaces draft-kishjac-bmwg-evpntest.

Please reply to the rtg-dir review: https://mailarchive.ietf.org/arch/msg/rtg-dir/6cK9FKqvIIKEd_f6Ov1bKGfw_3w/#
2021-06-28
09 Alvaro Retana [Ballot Position Update] New position, No Objection, has been recorded for Alvaro Retana
2021-06-18
09 (System) IANA Review state changed to Version Changed - Review Needed from IANA OK - No Actions Needed
2021-06-18
09 sudhin jacob New version available: draft-ietf-bmwg-evpntest-09.txt
2021-06-18
09 (System) New version approved
2021-06-18
09 (System) Request for posting confirmation emailed to previous authors: Kishore Tiruveedhula , sudhin jacob
2021-06-18
09 sudhin jacob Uploaded new revision
2021-06-16
08 Joel Halpern Request for Last Call review by RTGDIR Completed: Has Issues. Reviewer: Joel Halpern. Sent review to list.
2021-06-10
08 Amanda Baber IANA Review state changed to IANA OK - No Actions Needed from Version Changed - Review Needed
2021-06-08
08 Luc André Burdet Request for Last Call review by RTGDIR is assigned to Joel Halpern
2021-06-08
08 Luc André Burdet Request for Last Call review by RTGDIR is assigned to Joel Halpern
2021-06-08
08 Luc André Burdet Assignment of request for Last Call review by RTGDIR to Eric Gray was withdrawn
2021-06-07
08 Amy Vezza Placed on agenda for telechat - 2021-07-01
2021-06-07
08 Warren Kumari Ballot has been issued
2021-06-07
08 Warren Kumari [Ballot Position Update] New position, Yes, has been recorded for Warren Kumari
2021-06-07
08 Warren Kumari Created "Approve" ballot
2021-06-07
08 Warren Kumari IESG state changed to IESG Evaluation from Waiting for AD Go-Ahead
2021-06-03
08 Luc André Burdet Request for Last Call review by RTGDIR is assigned to Eric Gray
2021-06-03
08 Luc André Burdet Request for Last Call review by RTGDIR is assigned to Eric Gray
2021-06-03
08 Luc André Burdet Assignment of request for Last Call review by RTGDIR to Lou Berger was withdrawn
2021-05-30
08 (System) IANA Review state changed to Version Changed - Review Needed from IANA OK - No Actions Needed
2021-05-30
08 sudhin jacob New version available: draft-ietf-bmwg-evpntest-08.txt
2021-05-30
08 (System) New version approved
2021-05-29
08 (System) Request for posting confirmation emailed to previous authors: Kishore Tiruveedhula , bmwg-chairs@ietf.org, sudhin jacob
2021-05-29
08 sudhin jacob Uploaded new revision
2021-05-24
07 Jana Iyengar Request for Last Call review by TSVART Completed: Ready with Issues. Reviewer: Jana Iyengar. Sent review to list.
2021-05-24
07 Ines Robles Request for Last Call review by GENART Completed: Ready. Reviewer: Ines Robles. Sent review to list.
2021-05-24
07 (System) IESG state changed to Waiting for AD Go-Ahead from In Last Call
2021-05-22
07 Robert Sparks Request for Last Call review by SECDIR Completed: Has Nits. Reviewer: Robert Sparks. Sent review to list.
2021-05-19
07 Min Ye Assignment of request for Last Call review by RTGDIR to Manav Bhatia was rejected
2021-05-19
07 (System) IANA Review state changed to IANA OK - No Actions Needed from IANA - Review Needed
2021-05-19
07 Sabrina Tanamal
(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has reviewed draft-ietf-bmwg-evpntest-07, which is currently in Last Call, and has the following comments:

We …
(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has reviewed draft-ietf-bmwg-evpntest-07, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any registry actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object.

If this assessment is not accurate, please respond as soon as possible.

Thank you,

Sabrina Tanamal
Senior IANA Services Specialist
2021-05-14
07 Min Ye Request for Last Call review by RTGDIR is assigned to Lou Berger
2021-05-14
07 Min Ye Request for Last Call review by RTGDIR is assigned to Lou Berger
2021-05-14
07 Min Ye Request for Last Call review by RTGDIR is assigned to Manav Bhatia
2021-05-14
07 Min Ye Request for Last Call review by RTGDIR is assigned to Manav Bhatia
2021-05-13
07 Tero Kivinen Request for Last Call review by SECDIR is assigned to Robert Sparks
2021-05-13
07 Tero Kivinen Request for Last Call review by SECDIR is assigned to Robert Sparks
2021-05-13
07 Jean Mahoney Request for Last Call review by GENART is assigned to Ines Robles
2021-05-13
07 Jean Mahoney Request for Last Call review by GENART is assigned to Ines Robles
2021-05-12
07 Gunter Van de Velde Request for Last Call review by OPSDIR is assigned to Jon Mitchell
2021-05-12
07 Gunter Van de Velde Request for Last Call review by OPSDIR is assigned to Jon Mitchell
2021-05-12
07 Magnus Westerlund Request for Last Call review by TSVART is assigned to Jana Iyengar
2021-05-12
07 Magnus Westerlund Request for Last Call review by TSVART is assigned to Jana Iyengar
2021-05-10
07 Alvaro Retana Requested Last Call review by RTGDIR
2021-05-10
07 Cindy Morgan IANA Review state changed to IANA - Review Needed
2021-05-10
07 Cindy Morgan
The following Last Call announcement was sent out (ends 2021-05-24):

From: The IESG
To: IETF-Announce
CC: Sarah Banks , bmwg-chairs@ietf.org, bmwg@ietf.org, draft-ietf-bmwg-evpntest@ietf.org, …
The following Last Call announcement was sent out (ends 2021-05-24):

From: The IESG
To: IETF-Announce
CC: Sarah Banks , bmwg-chairs@ietf.org, bmwg@ietf.org, draft-ietf-bmwg-evpntest@ietf.org, sbanks@encrypted.net, warren@kumari.net
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (Benchmarking Methodology for EVPN and PBB-EVPN) to Informational RFC


The IESG has received a request from the Benchmarking Methodology WG (bmwg)
to consider the following document: - 'Benchmarking Methodology for EVPN and
PBB-EVPN'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2021-05-24. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  This document defines methodologies for benchmarking EVPN and PBB-
  EVPN performance.  EVPN is defined in RFC 7432, and is being deployed
  in Service Provider networks.  Specifically, this document defines
  the methodologies for benchmarking EVPN/PBB-EVPN convergence, data
  plane performance, and control plane performance.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-bmwg-evpntest/



No IPR declarations have been submitted directly on this I-D.




2021-05-10
07 Cindy Morgan IESG state changed to In Last Call from Last Call Requested
2021-05-10
07 Warren Kumari Last call was requested
2021-05-10
07 Warren Kumari Last call announcement was generated
2021-05-10
07 Warren Kumari Ballot approval text was generated
2021-05-10
07 Warren Kumari IESG state changed to Last Call Requested from Publication Requested
2021-05-10
07 Warren Kumari Ballot writeup was changed
2021-05-05
07 Sarah Banks
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated …
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated 1 November 2019.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

The document defines methodologies for benchmarking EVPN and PBB-EVPN performance.  EVPN is defined in RFC 7432, and is being deployed in Service Provider networks.  Specifically this document defines the methodologies for benchmarking EVPN/PBB-EVPN convergence, data plane performance, and control plane performance.

Working Group Summary:

The doc went through WGLC 4 times. We sent the document out to BESS for review, to get SME eyeballs on the draft. At one point, the Doc Shepherd had concerns that we had too many EVPN drafts in the WG, asked the authors to work together to sort that out, which they did. We've also seen additional editing by the authors after feedback from Warren, and additional reviews from Brian on Sections 3 and 4.

Document Quality:

This is a "how to benchmark" draft - yes there are implementations of the protocol, but those implementations specifically are out of scope for this draft per se; we are not a conformance WG.

Personnel:

Sarah Banks is the Document Shepherd
Warren Kumari is the responsible Area Director


(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

This doc shepherd performed a number of reviews of the document, including wearing an initial "Editor" hat, to assist with language, format, and flow. The document is ready for publication; it's been reviewed by BMWG, reviewed by BESS, had SMEs present on it, and underwent 4 WGLC, additional editing and reviews.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

No - because they were done with BESS ahead of this step in the process. Our sincere thanks to the BESS WG for their time.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

None.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

No IPR has been filed and the authors have not indicated they plan to file any.

(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Solid consensus.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

Nits is generally clean. Readers can review Nits output on datatracker. There are some instances of mentions of RFCs without explicit mention.

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

This is not that kind of document. No MIB/YANG/URI concerns here.

(13) Have all references within this document been identified as either normative or informative?

yes.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

No.

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

Yes (from Nits tool):
  ** Downref: Normative reference to an Informational RFC: RFC 2544
  ** Downref: Normative reference to an Informational RFC: RFC 2899

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 8126).

No IANA considerations here.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, YANG modules, etc.

None - NA

(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools (https://trac.ietf.org/trac/ops/wiki/yang-review-tools) for syntax and formatting validation? If there are any resulting errors or warnings, what is the justification for not fixing them at this time? Does the YANG module comply with the Network Management Datastore Architecture (NMDA) as specified in RFC8342?

NA
2021-05-05
07 Sarah Banks IETF WG state changed to Submitted to IESG for Publication from In WG Last Call
2021-05-05
07 Sarah Banks IESG state changed to Publication Requested from I-D Exists
2021-05-05
07 (System) Changed action holders to Warren Kumari (IESG state changed)
2021-05-05
07 Sarah Banks IESG process started in state Publication Requested
2021-05-05
07 Sarah Banks
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated …
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated 1 November 2019.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

The document defines methodologies for benchmarking EVPN and PBB-EVPN performance.  EVPN is defined in RFC 7432, and is being deployed in Service Provider networks.  Specifically this document defines the methodologies for benchmarking EVPN/PBB-EVPN convergence, data plane performance, and control plane performance.

Working Group Summary:

The doc went through WGLC 4 times. We sent the document out to BESS for review, to get SME eyeballs on the draft. At one point, the Doc Shepherd had concerns that we had too many EVPN drafts in the WG, asked the authors to work together to sort that out, which they did. We've also seen additional editing by the authors after feedback from Warren, and additional reviews from Brian on Sections 3 and 4.

Document Quality:

This is a "how to benchmark" draft - yes there are implementations of the protocol, but those implementations specifically are out of scope for this draft per se; we are not a conformance WG.

Personnel:

Sarah Banks is the Document Shepherd
Warren Kumari is the responsible Area Director


(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

This doc shepherd performed a number of reviews of the document, including wearing an initial "Editor" hat, to assist with language, format, and flow. The document is ready for publication; it's been reviewed by BMWG, reviewed by BESS, had SMEs present on it, and underwent 4 WGLC, additional editing and reviews.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

No - because they were done with BESS ahead of this step in the process. Our sincere thanks to the BESS WG for their time.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

None.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

No IPR has been filed and the authors have not indicated they plan to file any.

(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Solid consensus.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

Nits is generally clean. Readers can review Nits output on datatracker. There are some instances of mentions of RFCs without explicit mention.

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

This is not that kind of document. No MIB/YANG/URI concerns here.

(13) Have all references within this document been identified as either normative or informative?

yes.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

No.

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

Yes (from Nits tool):
  ** Downref: Normative reference to an Informational RFC: RFC 2544
  ** Downref: Normative reference to an Informational RFC: RFC 2899

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 8126).

No IANA considerations here.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, YANG modules, etc.

None - NA

(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools (https://trac.ietf.org/trac/ops/wiki/yang-review-tools) for syntax and formatting validation? If there are any resulting errors or warnings, what is the justification for not fixing them at this time? Does the YANG module comply with the Network Management Datastore Architecture (NMDA) as specified in RFC8342?

NA
2021-03-09
07 Sarah Banks Sending the document through a 4th and hopefully final WGLC. The document has seen extensive editing and reviewing post feedback from Warren.
2021-03-09
07 Sarah Banks IETF WG state changed to In WG Last Call from WG Document
2021-03-09
07 Sarah Banks
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated …
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated 1 November 2019.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

The document defines methodologies for benchmarking EVPN and PBB-EVPN performance.  EVPN is defined in RFC 7432, and is being deployed in Service Provider networks.  Specifically this document defines the methodologies for benchmarking EVPN/PBB-EVPN convergence, data plane performance, and control plane performance.

Working Group Summary:

The doc went through WGLC 4 times. We sent the document out to BESS for review, to get SME eyeballs on the draft. At one point, the Doc Shepherd had concerns that we had too many EVPN drafts in the WG, asked the authors to work together to sort that out, which they did. We've also seen additional editing by the authors after feedback from Warren, and additional reviews from Brian on Sections 3 and 4.

Document Quality:

This is a "how to benchmark" draft - yes there are implementations of the protocol, but those implementations specifically are out of scope for this draft per se; we are not a conformance WG.

Personnel:

Sarah Banks is the Document Shepherd
Warren Kumari is the responsible Area Director


(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

This doc shepherd performed a number of reviews of the document, including wearing an initial "Editor" hat, to assist with language, format, and flow. The document is ready for publication; it's been reviewed by BMWG, reviewed by BESS, had SMEs present on it, and underwent 4 WGLC, additional editing and reviews.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

No - because they were done with BESS ahead of this step in the process. Our sincere thanks to the BESS WG for their time.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

None.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

No IPR has been filed and the authors have not indicated they plan to file any.

(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Solid consensus.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

Nits is generally clean. Readers can review Nits output on datatracker. There are some instances of mentions of RFCs without explicit mention.

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

This is not that kind of document. No MIB/YANG/URI concerns here.

(13) Have all references within this document been identified as either normative or informative?

yes.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

No.

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

Yes (from Nits tool):
  ** Downref: Normative reference to an Informational RFC: RFC 2544
  ** Downref: Normative reference to an Informational RFC: RFC 2899

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 8126).

No IANA considerations here.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, YANG modules, etc.

None - NA

(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools (https://trac.ietf.org/trac/ops/wiki/yang-review-tools) for syntax and formatting validation? If there are any resulting errors or warnings, what is the justification for not fixing them at this time? Does the YANG module comply with the Network Management Datastore Architecture (NMDA) as specified in RFC8342?

NA
2021-02-02
07 sudhin jacob New version available: draft-ietf-bmwg-evpntest-07.txt
2021-02-02
07 (System) New version approved
2021-02-02
07 (System) Request for posting confirmation emailed to previous authors: Kishore Tiruveedhula , sudhin jacob
2021-02-02
07 sudhin jacob Uploaded new revision
2020-08-07
06 sudhin jacob New version available: draft-ietf-bmwg-evpntest-06.txt
2020-08-07
06 (System) New version approved
2020-08-07
06 (System) Request for posting confirmation emailed to previous authors: Kishore Tiruveedhula , sudhin jacob
2020-08-07
06 sudhin jacob Uploaded new revision
2020-05-15
05 Warren Kumari IETF WG state changed to WG Document from Submitted to IESG for Publication
2020-05-15
05 Warren Kumari
I'm returning this to the WG. The document has gotten stuck, and this needs additional eyes.

I'm hoping for thorough review be the WG, and …
I'm returning this to the WG. The document has gotten stuck, and this needs additional eyes.

I'm hoping for thorough review be the WG, and hope to see it come back soon...
2020-05-15
05 Warren Kumari IESG state changed to I-D Exists from Publication Requested
2020-03-11
05 sudhin jacob New version available: draft-ietf-bmwg-evpntest-05.txt
2020-03-11
05 (System) New version approved
2020-03-11
05 (System) Request for posting confirmation emailed to previous authors: Kishore Tiruveedhula , sudhin jacob
2020-03-11
05 sudhin jacob Uploaded new revision
2019-12-18
04 sudhin jacob New version available: draft-ietf-bmwg-evpntest-04.txt
2019-12-18
04 (System) New version approved
2019-12-18
04 (System) Request for posting confirmation emailed to previous authors: Kishore Tiruveedhula , sudhin jacob
2019-12-18
04 sudhin jacob Uploaded new revision
2019-11-12
03 Sarah Banks
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated …
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated 1 November 2019.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

The document defines methodologies for benchmarking EVPN and PBB-EVPN performance.  EVPN is defined in RFC 7432, and is being deployed in Service Provider networks.  Specifically this document defines the methodologies for benchmarking EVPN/PBB-EVPN convergence, data plane performance, and control plane performance.

Working Group Summary:

The doc went through WGLC 3 times. We sent the document out to BESS for review, to get SME eyeballs on the draft. At one point, the Doc Shepherd had concerns that we had too many EVPN drafts in the WG, asked the authors to work together to sort that out, which they did. No additional concerns have been raised since.

Document Quality:

This is a "how to benchmark" draft - yes there are implementations of the protocol, but those implementations specifically are out of scope for this draft per se; we are not a conformance WG.

Personnel:

Sarah Banks is the Document Shepherd
Warren Kumari is the responsible Area Director


(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

This doc shepherd performed a number of reviews of the document, including wearing an initial "Editor" hat, to assist with language, format, and flow. The document is ready for publication; it's been reviewed by BMWG, reviewed by BESS, had SMEs present on it, and underwent 3 WGLC.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

No - because they were done with BESS ahead of this step in the process. Our sincere thanks to the BESS WG for their time.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

None.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

No IPR has been filed and the authors have not indicated they plan to file any.

(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Solid consensus.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

Nits is generally clean. Readers can review Nits output on datatracker. There are some instances of mentions of RFCs without explicit mention.

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

This is not that kind of document. No MIB/YANG/URI concerns here.

(13) Have all references within this document been identified as either normative or informative?

yes.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

No.

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

Yes (from Nits tool):
  ** Downref: Normative reference to an Informational RFC: RFC 2544
  ** Downref: Normative reference to an Informational RFC: RFC 2899

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 8126).

No IANA considerations here.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, YANG modules, etc.

None - NA

(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools (https://trac.ietf.org/trac/ops/wiki/yang-review-tools) for syntax and formatting validation? If there are any resulting errors or warnings, what is the justification for not fixing them at this time? Does the YANG module comply with the Network Management Datastore Architecture (NMDA) as specified in RFC8342?

NA
2019-11-12
03 Sarah Banks Responsible AD changed to Warren Kumari
2019-11-12
03 Sarah Banks IETF WG state changed to Submitted to IESG for Publication from WG Consensus: Waiting for Write-Up
2019-11-12
03 Sarah Banks IESG state changed to Publication Requested from I-D Exists
2019-11-12
03 Sarah Banks IESG process started in state Publication Requested
2019-11-12
03 Sarah Banks Intended Status changed to Informational from None
2019-11-12
03 Sarah Banks
The doc shows as a WG doc, but it was in LC and I am late to approve that. Moving to write up to move …
The doc shows as a WG doc, but it was in LC and I am late to approve that. Moving to write up to move this along.
2019-11-12
03 Sarah Banks IETF WG state changed to WG Consensus: Waiting for Write-Up from WG Document
2019-11-12
03 Sarah Banks
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated …
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated 1 November 2019.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

The document defines methodologies for benchmarking EVPN and PBB-EVPN performance.  EVPN is defined in RFC 7432, and is being deployed in Service Provider networks.  Specifically this document defines the methodologies for benchmarking EVPN/PBB-EVPN convergence, data plane performance, and control plane performance.

Working Group Summary:

The doc went through WGLC 3 times. We sent the document out to BESS for review, to get SME eyeballs on the draft. At one point, the Doc Shepherd had concerns that we had too many EVPN drafts in the WG, asked the authors to work together to sort that out, which they did. No additional concerns have been raised since.

Document Quality:

This is a "how to benchmark" draft - yes there are implementations of the protocol, but those implementations specifically are out of scope for this draft per se; we are not a conformance WG.

Personnel:

Sarah Banks is the Document Shepherd
Warren Kumari is the responsible Area Director


(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

This doc shepherd performed a number of reviews of the document, including wearing an initial "Editor" hat, to assist with language, format, and flow. The document is ready for publication; it's been reviewed by BMWG, reviewed by BESS, had SMEs present on it, and underwent 3 WGLC.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

No - because they were done with BESS ahead of this step in the process. Our sincere thanks to the BESS WG for their time.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

None.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

No IPR has been filed and the authors have not indicated they plan to file any.

(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Solid consensus.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

Nits is generally clean. Readers can review Nits output on datatracker. There are some instances of mentions of RFCs without explicit mention.

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

This is not that kind of document. No MIB/YANG/URI concerns here.

(13) Have all references within this document been identified as either normative or informative?

yes.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

No.

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

Yes (from Nits tool):
  ** Downref: Normative reference to an Informational RFC: RFC 2544
  ** Downref: Normative reference to an Informational RFC: RFC 2899

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 8126).

No IANA considerations here.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, YANG modules, etc.

None - NA

(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools (https://trac.ietf.org/trac/ops/wiki/yang-review-tools) for syntax and formatting validation? If there are any resulting errors or warnings, what is the justification for not fixing them at this time? Does the YANG module comply with the Network Management Datastore Architecture (NMDA) as specified in RFC8342?

NA
2019-08-21
03 sudhin jacob New version available: draft-ietf-bmwg-evpntest-03.txt
2019-08-21
03 (System) New version approved
2019-08-21
03 (System) Request for posting confirmation emailed to previous authors: Kishore Tiruveedhula , sudhin jacob
2019-08-21
03 sudhin jacob Uploaded new revision
2019-05-02
02 sudhin jacob New version available: draft-ietf-bmwg-evpntest-02.txt
2019-05-02
02 (System) New version approved
2019-05-02
02 (System) Request for posting confirmation emailed to previous authors: Kishore Tiruveedhula , sudhin jacob
2019-05-02
02 sudhin jacob Uploaded new revision
2019-03-18
01 Al Morton Notification list changed to Sarah Banks <sbanks@encrypted.net>
2019-03-18
01 Al Morton Document shepherd changed to Sarah Banks
2018-11-26
01 sudhin jacob New version available: draft-ietf-bmwg-evpntest-01.txt
2018-11-26
01 (System) New version approved
2018-11-26
01 (System) Request for posting confirmation emailed to previous authors: Kishore Tiruveedhula , sudhin jacob
2018-11-26
01 sudhin jacob Uploaded new revision
2018-11-05
00 Al Morton Added to session: IETF-103: bmwg  Thu-0900
2018-09-17
00 sudhin jacob New version available: draft-ietf-bmwg-evpntest-00.txt
2018-09-17
00 (System) WG -00 approved
2018-09-12
00 sudhin jacob Set submitter to "SUDHIN JACOB ", replaces to (none) and sent approval email to group chairs: bmwg-chairs@ietf.org
2018-09-12
00 sudhin jacob Uploaded new revision