Benchmarking Methodology for EVPN and PBB-EVPN
draft-ietf-bmwg-evpntest-11

[S3.3] [question]

* "Fail the DUT-CE link" means to cause a link failure that can be detected
  by the DUT (e.g., a lost of PHY signal or some means to detect a
  directly-connected cable was removed)?

[S3.9] [question]

* What does "Send X arp/neighbour discovery(ND)" mean, exactly?

  If these are meant to be ARP REPLY/Neighbor Advertisements (NAs) then
  being explicit about that seems helpful.

  Or are these ARP REQUESTs and NS Neighbor Solicitations for the IRB address?

* "Send ... to the target IRB address configured in EVPN instance" seems
  like the ARP REPLY/ND NAs are to be sent unicast.  If that's true, I
  think it would improve clarity to say they're unicast.

I support Martin and Eric's DISCUSS positions about how some of the tests are characterized.

Like Eric, I found the number of typos, capitalization mistakes, and sentence fragments to be distracting.

Though its coverage of the WG process was good, the shepherd writeup declares the document status but avoided answering the other questions about its status (e.g., "Why is this the proper type of RFC?").  This seems to be a common omission, and I'm starting to wonder if we should just drop it from the template.

I think RFC 7432 and RFC 7623 should be normative, insofar as this document doesn't really make sense unless one understands those first.

Please use the correct boilerplate for BCP 14.  (See RFC 8174, Section 2.)  However, I also note that the only place the key words are used is Section 7, so it occurs to me you could avoid using BCP 14 altogether and basically say the same thing with the same force.

Section 1.2 defines "All-Active Redundancy Mode", but it's not used anywhere in this document other than its own definition.  Similarly, "AA", "ES", "Ethernet Tag", "RT", and "Sub interface" don't appear anywhere else.  "Single-Active Redundancy Mode" could be folded into the definition of "SA", since the latter is actually used elsewhere.

Some non-blocking comments -

* Section 2: 
   ** please expand abbreviated words in their first occurrence if not listed in well known terminologies. See the list : https://www.rfc-editor.org/materials/abbrev.expansion.txt 
   ** why is this called topology 1? are expected to have more typologies? Where are they? and it is actually not clear to me what is topology 1.

* Section 3.8: what is PPS?

* Section 3.9: how do you measure packet loss? is this loss measured for the whole run or with with a time window?

* Section 4.9 : "Then increment the scale of N by 5% of N till the limit is reached." Please clarify if the increment steps always 5% of initial value or N or N = N+5% of N.

Nits :

* Section 2 : Figure 2 - s/SHPFE3/SHPE3 or define SHPFE3.

== COMMENTS ==

As most of the test in section 4 are identical to those in section 3, is there a reason for having such repetition ? I.e., rather than having a single set of tests and mentioning the applicability of those tests to the PBB-EVPN scenarios.

-- Abstract --
Suggest to expand PBB as https://www.rfc-editor.org/materials/abbrev.expansion.txt does not have the compound PBB-EVPN. Same applies in section 1 where the expansion is used but not defined.


-- Section 2 --
Suggest to mention early that SH = single home and MH = multi-home

Strange use of 'traffic generator' for a box that also receives traffic. Is there a better word in BMWG ?

Unclear from figure 1 whether the link between SHPE3 and traffic generator acts as a "PE-CE link" as this link is the only unqualified one.

-- Section 3.1 --
In "X different source and destination MAC address for one vlan" (beside the singular form for "address") aren't X different source addresses enough to trigger the MAC learning ? I.e., no need to vary the destination MAC addresses.

Using external scripts to count learned MAC addresses appears very rudimentary and not very accurate...

-- Section 3.2 --
Suggest to use the same text or similar presentation for the measurement part as in section 3.1

-- Section 3.9 --
As it can be expected that most of the EVPN are dual-stack, I wonder whether the 2 single-stack measurements are useful as there could be an interaction between IPv4 and IPv6 learning. Unsure how to test it though (perhaps 50% IPv4 and 50 IPv6 ?).

-- Section 3.12 --
If "SOAK" is an acronym (based on the all uppercase), then please expand it, else use "Soak" 

== NITS ==

Please always use the all uppercase character for VLAN same for 'ipv6', 'arp', 'ip', ...

Also check punctuations as they should be followed by a space character.

-- Section 5 --
You may want to add a last name to Al in "to thank Al"

(3.8) (4.8) Why is packet loss measured in time? How is learning 2X MAC addresses relevant to the packet loss measurement at the traffic generator?
How long does the traffic generator have to wait to conclude that the packet is lost?

(3.9) Is a single failure to learn an address sufficient to determine that the device has reached capacity? Or could packet loss or some other phenomenon lose some addresses? In other words, be more precise on how polling reveals the capacity.

Is there some lower bound on the time between sending ARP/ND packets and querying the DUT?

(3.11, 3.12, 4.10, 4.11) Does the traffic generator send F frames in total or F ffs? The spec says both. Are there any constraints on F, perhaps an integer multiple of X?