Tracking Reviews of Documents
draft-sparks-genarea-review-tracker-03

[Ballot comment]
This draft is really clear. I haven't been on a review team since 2010, but it sure looks good from the outside.

Just a couple of nits and minor thoughts ...

"all the of documents" is probably "all of the documents".

In this text,

  o  The tool must make it easy for the secretary to see the result of
      previous reviews from this team for a given document.  In Secdir,
      for example, if the request is for a revision that has only minor
      differences, and the previous review result was "Ready", a new
      assignment will not be made.
                ^^^^^^^^^^^^^^^^
I would have parsed this more easily if the text had been "the secretary will not make a new assignment". Darned past tense.

In this text,

  o  A secretary must be able to configure the tool to remind them to
      followup when actions are due.  (For instance, a secretary could
      receive email when a review is about to become overdue).
     
I've never been a secretary, but is it obvious whether this would be per-secretary or per-document (possibly inherited from a per-secretary setting)? If this was a calendar, there would be a default alert interval, but I'd be able to say "remind me again on Tuesday". Is that relevant here?

Thank you for capturing Appendix B.  Suggested features deferred for future work. That stuff would likely be great, but not worth holding everything else up for.

[Ballot comment]
The shepherd write-up says Brian C's comments have still
to be handled - is that still true? I guess they have been
handled, given the dates.

[Ballot comment]
-- 3, bullet starting with "Reviewers must be able to indicate they are transitioning..."

Is this different than a "soft hiatus" followed by a "hard hiatus"?

[Ballot comment]
Thank you for your work summarizing the tool requirements, it will be good to have reviewers tied to the tracker in the future.  It would be nice to see the connection from a draft to the review without the need for the secretariat to create that match.

I see the last bullet of section3 is out of scope for this effort and I have one other item that would be nice to have, but could also be a future request similar in nature.  This would be the ability for the AD to send a thank you note from the interface when they are notified that their review was used.

[Ballot comment]
I agree with Alvaro that there is no need to publish this fine document as an RFC. Let it exist as a living document somewhere.

[Ballot comment]
This is a nice document, and the expected enhancements will fill a needed gap. 

However, I really don’t see the need to publish a list of tool requirements as an RFC.  Not only will the requirements eventually be satisfied, but others will come up — for example the author/editor-focused requirements briefly mentioned in the appendix.  I think it would be better if this document was kept in the GEN Area wiki.

Having said that, I won’t stand in the way of publication.

(Via drafts-lastcall@iana.org): IESG/Authors:

IANA has reviewed draft-sparks-genarea-review-tracker-02, which is currently in Last Call, and has the following comments:

We understand that, upon approval of this document, there are no IANA Actions that need completion.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, IANA does not object.

If this assessment is not accurate, please respond as soon as possible.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (Tracking Reviews of Documents) to Informational RFC


The IESG has received a request from an individual submitter to consider
the following document:
- 'Tracking Reviews of Documents'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2015-08-04. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  Several review teams ensure specific types of review are performed on
  Internet-Drafts as they progress towards becoming RFCs.  The tools
  used by these teams to assign and track reviews would benefit from
  tighter integration to the Datatracker.  This document discusses
  requirements for improving those tools without disrupting current
  work flows.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-sparks-genarea-review-tracker/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-sparks-genarea-review-tracker/ballot/


No IPR declarations have been submitted directly on this I-D.

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?

  This document is Informational, as indicated in both the document
  itself and the tracker. It is certainly not protocol, and therefore
  neither Standards Track nor Experimental are appropriate. It is also
  not changing any IETF procedure, so BCP is also out. Informational
  seems correct.

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  Several review teams ensure specific types of review are performed on
  Internet-Drafts as they progress towards becoming RFCs.  The tools
  used by these teams to assign and track reviews would benefit from
  tighter integration to the Datatracker.  This document discusses
  requirements for improving those tools without disrupting current
  work flows.

Working Group Summary

  Several of the current review teams have looked at this document
  including gen-art, secdir, and tsv-art. Some short discussion took
  place on dir-coord and tools-discuss; comments were addressed, either
  in discussion or in the document. There were no objections (and a few
  kudos).

Document Quality

  People seem satisfied with the document's quality. There have been
  some editorial comments on the tools-discuss list that have yet
  to be addressed, but no showstoppers. The shepherd has no concerns
  regarding consensus for the contents of this document.

Personnel

  Pete Resnick  is the shepherd.
  Jari Arkko is the responsible AD.

(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

  The shepherd gave the document a read through. Not being a particular
  expert on review teams, the document seems reasonable. The shepherd
  also read through the comments in the archive. Seems ready.

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

  Reviews seemed diligent. Brian Carpenter's comments at:

 

  still need to be addressed, but none is a showstopper.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

  The directorate reviews that took place seem sufficient.

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

  The shepherd has no additional concerns or issues.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

  Both authors have said that they have no IPR to disclose. Tero pointed
  out that he holds copyright for parts of the review tool code, but
  this doesn't directly impact the document itself. The shepherd agrees,
  though neither of us is a lawyer.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

  No IPR disclosures have been made.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

  A small group of folks involved in review teams gave review
  and seem to support it.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

  No objections seen, let alone appeal threats.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

  The 2606 warning is bogus. The shepherd has no opinion on whether
  Appendix A needs / around it.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

  None applicable.

(13) Have all references within this document been identified as
either normative or informative?

  All references are Informative, and that seems right.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

  No normative references.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

  No downward references.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

  No other RFCs impacted.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

  No IANA actions, and that is appropriate.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

  No IANA actions, and that is appropriate.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

  No such checks, other than ID-Nits.

[Draft - Writeup In Progress]

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?

  This document is Informational, as indicated in both the document
  itself and the tracker. It is certainly not protocol, and therefore
  neither Standards Track nor Experimental are appropriate. It is also
  not changing any IETF procedure, so BCP is also out. Informational
  seems correct.

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  Several review teams ensure specific types of review are performed on
  Internet-Drafts as they progress towards becoming RFCs.  The tools
  used by these teams to assign and track reviews would benefit from
  tighter integration to the Datatracker.  This document discusses
  requirements for improving those tools without disrupting current
  work flows.

Working Group Summary

  Several of the current review teams have looked at this document
  including gen-art, secdir, and tsv-art. Some short discussion took
  place on dir-coord and tools-discuss; comments were addressed, either
  in discussion or in the document. There were no objections (and a few
  kudos).

Document Quality

  People seem satisfied with the document's quality. Brian Carpenter
  made some editorial comments on the tools-discuss list that have yet
  to be addressed, but no showstoppers. The shepherd has no concerns
  regarding consensus for the contents of this document.

Personnel

  Pete Resnick  is the shepherd.
  Jari Arkko is the responsible AD.

(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

  The shepherd gave the document a read through. Not being a particular
  expert on review teams, the document seems reasonable. The shepherd
  also read through the comments in the archive. Seems ready.

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

  Reviews seemed diligent.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

  The directorate reviews that took place seem sufficient.

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

  The shepherd has no additional concerns or issues.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

  Not yet. Waiting for response.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

  No IPR disclosures have been made.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

  A small group of folks involved in review teams gave review
  and seem to support it.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

  No objections seen, let alone appeal threats.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

  The 2606 warning is bogus. The shepherd has no opinion on whether
  Appendix A needs / around it.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

  None applicable.

(13) Have all references within this document been identified as
either normative or informative?

  All references are Informative, and that seems right.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

  No normative references.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

  No downward references.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

  No other RFCs impacted.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

  No IANA actions, and that is appropriate.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

  No IANA actions, and that is appropriate.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

  No such checks, other than ID-Nits.

Looks good to me. I plan to ask for last call once you give me a shepherd and
shepherd writeup for this document. (It isn’t strictly speaking necessary, but
Robert indicated that he might be able to have one. And it would be better
to have one if it is possible. Let me know if you can’t make it happen.)

One minor issue:

Note, however, that having to
  wait for the document to appear in the archive to know the link to
  paste into the tool is a significant enough impedance that this link
  is often not provided by the reviewer.

s/document/review mail/? would be easier to understand at least for me.