Improving Awareness of Running Code: The Implementation Status Section
draft-sheffer-running-code-06

[Ballot comment]
Thanks for the new draft version.

- I could live with your proposed change addressing my DISCUSS:

  As a result,
  we do not envisage changes to this section after approval of the
  document for publication, e.g. the RFC Errata process does not apply.

However, based on my recollection of the IESG telechat discussion, I had it a stronger message in mind:
  As a result, changes to this section after approval of the
  document for publication should not be permitted, e.g. the
  RFC Errata process does not apply.

-
OLD:
  Working group chairs are requested to ensure that this section is not
  used as a marketing venue for specific implementations

NEW:
  Document shepherd and working group chairs are requested to ensure that this section is not
  used as a marketing venue for specific implementations

Justification: after the document left the WG, the document shepherd is responsible for the document, not the WG chairs ... even if we all know that the two functions overlap in most cases.

[Ballot discuss]
I would like to see this RFC published.
Before doing so, we should solve a specific problem: What about errata regarding the "Implementation Status" section?
I can foresee an explosion of "me too, I have an implementation" errata.
That's a difficult topic though. Errata on existing text might be OK, while new implementations are not OK.
Where do we draw the line?

The following would fix my concern.
OLD:
  We
  recommend that the Implementation Status section should be removed
  from Internet Drafts before they are published as RFCs.
NEW:
  The Implementation Status section must be removed
  from Internet Drafts before they are published as RFCs.


Alternatively, clear guidelines are required, so that we don't redo the discussion every time.

[Ballot comment]
- I was first thinking that an extra bullet in section 2 would be useful
  o  Implementation experience: Any useful information the implementers want to share with the community
However, since that section will not be published, that information will be lost.
On the other side, if it's kept on a WIKI.

- An extra benefit of this section is for the reviewer.
For example, in case of a MIB doctor review, having multiple implementations is a good argument for not insisting on a I-would-have-done-it-differently argument.

[Ballot comment]
This is a very good proposal and want to see it adopted. Thank you for writing it. (I share the opinion in comments from Stewart that a pointer to the wiki would be a better option though.)

[Ballot discuss]
This is a very good proposal and want to see it adopted. Thank you for writing it. (I share the opinion in comments from Stewart that a pointer to the wiki would be a better option though.)

But I do have one issue that I would like to discuss before final approval of this proposal. After discussion I will ballot Yes for this document. Here's the issue:

I believe section 2 would fit the situation in various IETF efforts better, if it clearly said that incomplete information is acceptable. For instance, I have implemented multiple IETF protocols in recent year or two in my day job, but I'm not sure I would have been comfortable with passing all the listed items about my implementation in all cases.

[Ballot discuss]
This is a very good proposal and want to see it adopted. Thank you for writing it. (I share the opinion in comments from Stewart that a pointer to the wiki would be a better option though.)

But I do have one issue that I would like to discuss before final approval of this proposal. I believe section 2 would fit the situation in various IETF efforts better, if it clearly said that incomplete information is acceptable. For instance, I have implemented multiple IETF protocols in recent year or two in my day job, but I'm not sure I would have been comfortable with passing all the listed items about my implementation in all cases.

[Ballot comment]
EDIT: It has been pointed out to me that the document actually says to remove the Implementation Status section before going to RFC (end of Section 2).  Alright then, carry on!  Nonetheless, it might be helpful to note that this sort of information can also be useful after an RFC is published, so authors should consider moving the Implementation Status section to some other place (e.g., a web page or wiki) before it disappears at publication.

(Was: I think this is a fine experiment.  But I also think that an RFC is completely the wrong place for this sort of information.  Partly because of the points Stewart raises, but mainly because implementation status can be very dynamic information.  So if you put it in an RFC, it's virtually guaranteed to become stale, especially if the RFC is successful!  It would be better to figure out a way to express this stuff in a more dynamic medium (e.g., a wiki) that could be referenced from an RFC in a standard way.)

[Ballot comment]
I think this is a fine experiment.  But I also think that an RFC is completely the wrong place for this sort of information.  Partly because of the points Stewart raises, but mainly because implementation status can be very dynamic information.  So if you put it in an RFC, it's virtually guaranteed to become stale, especially if the RFC is successful!  It would be better to figure out a way to express this stuff in a more dynamic medium (e.g., a wiki) that could be referenced from an RFC in a standard way.

[Ballot discuss]
I have very mixed feeling about this.

I  understand why this would be a good thing to do.

On the other hand I have concerns about the commercial factors that may creep in regarding the completeness and correctness of the statements, the pressures on the implementers to include a statement or remain silent, the order of the companies in the list, and the legal issues that may creep into it and impact the IETF. For example this may be considered an advertisement by an Advertizing Standards Agency with conformance issues , or by an equipment customer leading compliance liability issues which impact the IETF.

I think that we need to run this past Jorge and likely have a suitable legal disclaimer at the top of the section and we may even need a vetting process for the text.

I think that we should probably insist that the section be removed before it goes to the RFCEditor queue lest we have commercial interests require that we add them whilst in the queue or risk being sued for disadvantaging their implementation. In that regard the Wiki approach is far superior to the list in the document approach.

Now I know that it has been done in some documents already, but this is institutionalizing the process and we need to proceed carefully.

[Ballot comment]
I see no reason why this would not apply to process, and thus contra to the statement in section 6, the evidence seems to be that  it is in implementation.

[Ballot discuss]
I have very mixed feeling about this.

I  understand why this would be a good thing to do.

On the other hand I have concerns about the commercial factors that may creep in regarding the completeness and correctness of the statements, the pressures on the implementers to include a statement or remain silent, the order of the companies in the list, and the legal issues that may creep into it and impact the IETF. For example this may be considered an advertisement by an Advertizing Standards Agency with conformance issues , or by an equipment customer leading conformance liability issues which impact the IETF.

I think that we need to run this past Jorge and likely have a suitable legal disclaimer at the top of the section and we may even need a vetting process for the text.

I think that we should probably insist that the section be removed before it goes to the RFCEditor queue lest we have commercial interests require that we add them whilst in the queue or risk being sued for disadvantaging their implementation. In that regard the Wiki approach is far superior to the list in the document approach.

Now I know that it has been done in some documents already, but this is institutionalizing the process and we need to proceed carefully.

[Ballot comment]

I see no reason why this would not apply to process, and thus contra to the statement in section 6, the evidence seems to be that  it is in implementation.

[Ballot comment]
I'm fine with putting this into practice.

I have one non-blocking request - Is it possible to call this "something besides an experiment" in the draft? (A "proposal"? An "invitation"? an "exploration"?)

I'm sorry for the late surprise. I sent e-mail multiple times about the RFC 3933-ness of Stephen's Fast-Track draft, but don't see that I was equally vocal about this draft.

My point in sending those e-mails was to say "if you don't need to change BCP text, don't do this as an RFC 3933 process experiment, because RFC 3933 process experiments are still somewhat heavyweight, so you only do one when you need to change BCP text".

That got heard. This draft isn't an RFC 3933 process experiment, and it says so.

My concern is that the proposal is still described as some variation of "an experiment" about 15 times, RFC 3933 is even listed as a reference, and the draft doesn't say "and by the way, this isn't an *RFC 3933 process* experiment" until about the ninth time the proposal called an experiment.

If it's too late to do anything else, perhaps this point could be moved earlier in the draft.

I am hoping that (with Martin's agreement) we do some actual RFC 3933 process experiments in TSV during the shorter of (the next two years | Spencer being recalled), and I'd like for the community to understand what that means when we put one into place. It is difficult enough to do process change without confusing the community about how that change will be carried out.

I recognize that "an experiment" != "an RFC 3933 process experiment", but if I tried to publish a "proposal for a standardized protocol" that mentioned about halfway through the spec that I didn't mean "an RFC 6410 proposed standard", I hope somebody would object!

[Ballot comment]
Not sure that there's any paritcular reason to propose a time limit given the non-mandatory status. if it falls into disuse or is wildly successful then I suppose something can be done about either of those cases.

[Ballot comment]
Is it worth noting that if this section is in the main body of the specification that the section should be marked as informational?  Likely that the preamble that's there is enough, but maybe worth a thought.

IESG/Authors/WG Chairs:

IANA has reviewed draft-sheffer-running-code-04, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any IANA actions.

If this assessment is not accurate, please respond as soon as possible.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
Reply-To: ietf@ietf.org
Subject: Last Call:  (Improving Awareness of Running Code: the Implementation Status Section) to Experimental RFC


The IESG has received a request from an individual submitter to consider
the following document:
- 'Improving Awareness of Running Code: the Implementation Status
Section'
  as Experimental RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2013-05-10. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This document describes a simple process that allows authors of
  Internet-Drafts to record the status of known implementations by
  including an Implementation Status section.  This will allow
  reviewers and working groups to assign due consideration to documents
  that have the benefit of running code, by considering the running
  code as evidence of valuable experimentation and feedback that has
  made the implemented protocols more mature.

  The process in this document is offered as an experiment.  Authors of
  Internet-Drafts are encouraged to consider using the process for
  their documents, and working groups are invited to think about
  applying the process to all of their protocol specifications.  The
  authors of this document intend to collate experiences with this
  experiment and to report them to the community.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-sheffer-running-code/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-sheffer-running-code/ballot/


No IPR declarations have been submitted directly on this I-D.

Got this fine writeup from Adrian:

Shepherd write-up for draft-sheffer-running-code

Document Shepherd: Adrian farrel (adrian@olddog.co.uk)

(1) What type of RFC is being requested (BCP, Proposed Standard,
    Internet Standard, Informational, Experimental, or Historic)? Why
    is this the proper type of RFC? Is this type of RFC indicated in
    the title page header?

    This document is proposed as Experimental.  It describes an
    experiment that document authors can perform, and also describes
    how the authors of this document will collect results and
    report back to the community.

    This document was considered as Informational (by just suggesting
    an approach), but the authors believe it would be better to conduct
    an experiment and report back.

    This document is not a process experiment (RFC3933) because it does
    not suggest any change to or varioation of documented IETF process.

    The title page header indicates Experimental.

(2) The IESG approval announcement includes a Document Announcement
    Write-Up. Please provide such a Document Announcement Write-Up.
    Recent examples can be found in the "Action" announcements for
    approved documents. The approval announcement contains the
    following sections:

    Technical Summary

      This document describes a simple process that allows authors of
      Internet-Drafts to record the status of known implementations by
      including an Implementation Status section.  This will allow
      reviewers and working groups to assign due consideration to
      documents that have the benefit of running code and potentially
      treat the documents with implementations preferentially.

      The process in this document is offered as an experiment.  Authors
      of Internet-Drafts are encouraged to consider using the process
      for their documents, and working groups are invited to think about
      applying the process to all of their protocol specifications.  The
      authors of this document intend to collate experiences with this
      experiment and to report them to the community.

    Working Group Summary

      This document was not considered in any WG as there is none
      chartered to consider this topic, or anything remotely similar.

    Document Quality

      This document does not describe a protocol, and so there can be no
      implementations, per se.  However, a number of document authors
      have already decided to engage with the experiment as noted in
      section 6.1.  A few more authors have also indicated that they
      will think about participating, and several WG chairs have said
      that they will encourage their authors to look into it.

      The document has been circulated on the main IETF list and on the
      WG chairs list.  Both lists generated a small amount of traffic
      that has led to minor improvments in the document.

      There is no formal language in the draft tht requires review or
      testing.

    Personnel

      Adrian Farrel (adrian@olddog.co.uk) is the Document Shepherd.
      Stephen Farrell (stephen.farrell@cs.tcd.ie) is the Responsible AD.

(3) Briefly describe the review of this document that was performed by
    the Document Shepherd. If this version of the document is not ready
    for publication, please explain why the document is being forwarded
    to the IESG.

    The document shepherd is a co-author of this document and has read
    every word multple times, and mainly in the order they are written.
    The document shepherd has also examined how the process described
    might be applied and applied it to this document.

(4) Does the document Shepherd have any concerns about the depth or
    breadth of the reviews that have been performed?

    No.  It will, however, also be good to get IESG review because that
    group of people has a deep understanding of IETF process.

(5) Do portions of the document need review from a particular or from
    broader perspective, e.g., security, operational complexity, AAA,
    DNS, DHCP, XML, or internationalization? If so, describe the review
    that took place.

    No, but see (4).

(6) Describe any specific concerns or issues that the Document Shepherd
    has with this document that the Responsible Area Director and/or
    the IESG should be aware of? For example, perhaps he or she is
    uncomfortable with certain parts of the document, or has concerns
    whether there really is a need for it. In any event, if the
    interested community has discussed those issues and has indicated
    that it still wishes to advance the document, detail those concerns
    here.

    The only concerns the document shepherd has relate to the potnetial
    for large amounts of work when following up the experiment at some
    point in the future.

(7) Has each author confirmed that any and all appropriate IPR
    disclosures required for full conformance with the provisions of
    BCP 78 and BCP 79 have already been filed. If not, explain why.

    Yes, both authors have sent such confirmations to the responsible
    AD.

(8) Has an IPR disclosure been filed that references this document?
    If so, summarize any discussion and conclusion regarding the IPR
    disclosures.

    No IPR disclosures filed that reference this document.

(9) How solid is the consensus of the interested community behind this
    document? Does it represent the strong concurrence of a few
    individuals, with others being silent, or does the interested
    community as a whole understand and agree with it?

    The discussion on the IETF list and WG chairs list indicated a fair
    number of people who throught that the idea would merit
    experimentation.  A significantly large number of people hold to the
    view that running code is an important element in testing a
    specification document form completeness, correctness, and lack of
    ambiguity, and those people believe that knowing about
    implementations helps make the assessment of whether a draft is
    ready to advance for publication as an RFC.

    There has been no oposition to this document or its ideas although
    some have raised a voice that it does not go far enough and should
    either mandate implementation reports or guarantee favoured
    publication status for documents that have been implemented.  Those
    views are a small minority.

(10) Has anyone threatened an appeal or otherwise indicated extreme
    discontent? If so, please summarise the areas of conflict in
    separate email messages to the Responsible Area Director. (It
    should be in a separate email because this questionnaire is
    publicly available.)

    No appeals threatened.  No discontent indicated, extreme or
    otherwise.

(11) Identify any ID nits the Document Shepherd has found in this
    document. (See http://www.ietf.org/tools/idnits/ and the
    Internet-Drafts Checklist). Boilerplate checks are not enough;
    this check needs to be thorough.

    idnits is clean

(12) Describe how the document meets any required formal review
    criteria, such as the MIB Doctor, media type, and URI type reviews.

    No formal review criteria apply to this document.

(13) Have all references within this document been identified as either
    normative or informative?

    Yes.

(14) Are there normative references to documents that are not ready for
    advancement or are otherwise in an unclear state? If such normative
    references exist, what is the plan for their completion?

    No normative references are to documents that are not ready for
    advancement or are otherwise in an unclear state.

(15) Are there downward normative references references (see RFC 3967)?
    If so, list these downward references to support the Area Director
    in the Last Call procedure.

    No downrefs.

(16) Will publication of this document change the status of any existing
    RFCs? Are those RFCs listed on the title page header, listed in the
    abstract, and discussed in the introduction? If the RFCs are not
    listed in the Abstract and Introduction, explain why, and point to
    the part of the document where the relationship of this document to
    the other RFCs is discussed. If this information is not in the
    document, explain why the interested community considers it
    unnecessary.

    This document does not propose to change the state of any existing
    RFCs.

(17) Describe the Document Shepherd's review of the IANA considerations
    section, especially with regard to its consistency with the body of
    the document. Confirm that all protocol extensions that the document
    makes are associated with the appropriate reservations in IANA
    registries. Confirm that any referenced IANA registries have been
    clearly identified. Confirm that newly created IANA registries
    include a detailed specification of the initial contents for the
    registry, that allocations procedures for future registrations are
    defined, and a reasonable name for the new registry has been
    suggested (see RFC 5226).

    This document makes no requests for IANA action, and includes an
    appropriate Null IANA Section.

(18) List any new IANA registries that require Expert Review for future
    allocations. Provide any public guidance that the IESG would find
    useful in selecting the IANA Experts for these new registries.

    None such.

(19) Describe reviews and automated checks performed by to validate
    sections of the document written in a formal language, such as XML
    code, BNF rules, MIB definitions, etc.

    No formal language is used.