Requirements for IETF Technical Publication Service
draft-mankin-pub-req-11

[Ballot discuss]
This document creates a confusion between the term Shepherd as used by the Proto process and some other definition in this draft. I know certain people wish these were the same thing but they are not. When I first read this draft, I put in a comment that this should be changed - after it was clear that this confusion had IESG members talking past each other, I moved it to a discuss. The Protos work may only be a draft but it is a process we are using, have been using for a long time, and I would not want to change the terminology use there.

Reading the definition of 2.1 - I am totally unclear is this Shepherd is the Responsible AD or the PROTO Shepherd. I imagine that both would qualify as "Shepherd" under this definition. It is true that POSTCORR-2 might clarify which of two it is but this is pretty obtuse. I am not a fan of confusing things or changing the process by hiding it in obtuse changes to documents. I will point out that POSTEDIT-2 contradicts this by implying that the document shepherd is *not* the AD when it writes "(an Area Director or document shepherd."

So here is my concrete proposal to fix all this - I can imagine many other ways to fix it to but here is one. It is based on the assumption that we want this to document the current practice and not some possible future change. At a very high level, it seems to me that for changes to something the IESG approved, the AD as a member of the the IESG needs to make a reasonable call on if a change needs to come back to the the IESG or is OK as is. I think this draft should reflect that while at the same time allow the PROTOS process (which I am a big fan of) and make it so the use of the term Shepherd here is consistent with the common usage of it at IETF which is more or less the PROTOS use. I think we could do this with very small edits in a few places in the document.

In section 2.1 where we have a sentence that says
  The document shepherd (shown in the diagram as "Shepherd" is
  an individual designated by the IESG to shepherd a document through
  the reviews and the publication process.
Add a few words to end of this to make clear this is not the AD 
  The document shepherd (shown in the diagram as "Shepherd" is
  an individual designated by the IESG to shepherd a document through
  the reviews and the publication process and is often not an AD.

In Figure 1 in third column where we have IESG, Shepherd, Editor, WG, change this to add AD to the list

In Req-POSTEDIT-2 we change
      For the IETF standards
      process stream this includes the authors and an individual
      authorized to approve changes (an Area Director or document
      shepherd.
to
      For the IETF standards
      process stream this includes the authors and the Area Director.


In Req-POSTCORR-2 we change:
      party. In the IETF standards process stream this
      includes the individual designated by the IESG (sometimes referred
      to as the document shepherd and currently an AD).
to
      parties. For the IETF standards
      process stream this includes the authors and the Area Director.


In Req-POSTCORR-3 change
      For the IETF standards process stream, that
      authority is appointed by the IESG and is usually the document
      shepherd.
to
      For the IETF standards process stream, that
      authority is the Area Director.

[Ballot discuss]
The term shepherd is too confusing when combined with the PROTOs work. Current IESG members don't even all have the same idea of what we are talking about here. Please make it so no one is confused. I suggest changing to use  Responsible AD - if this changes in the future, we can change it. Or if you really want the de-refence, make up some new term and clearly define it.

[Ballot comment]
Section 3.15 is an area where improvement is really needed.  First,
  errata need to be published in a more timely manner.  Second, errata
  need to be much easier to locate.

  Section 4 should include a discussion of timely errata publication.

  Section 7 should also talk about integrity of the index.

IANA Last Call Comment:

As described in the IANA Considerations section, there are no specific
registrations or assignments to be made as a result of approval of this
document. The IANA recognizes that any future requirements that result from
changes in technical publication processes will need to be reviewed by IANA and
the IETF to understand to what extent, if any, the work flow of the documents
through IANA are affected.

[Ballot discuss]
I would like to see some discussion and perhaps rewriting of the sections on pre-approval editing and errata.

The pre-approval editing section reads as an advertisement for that process. While I understand that a requirement "to be able to do" is not the same as a requirement "to do", I feel this section will look pretty wierd if we don't do pre-approval editing or do it with another mechanism (e.g. volunteers, a different editing organization).  There are all kinds of things we could ask the RFC Editor to do in the future, not just limited to pre-approval editing, and this one is given special treatment. 

(Aside: My personal opinion is that pre-approval editing will delay documents, be much too expensive, and not substantially improve quality.  Having had a book copy-edited, I can staunchly claim that any difficulty understanding my book is my fault, and that the copy-editor had little chance of fixing it up significantly without themselves being a technical expert.)

The section on errata needs to be clearer on who is to maintain the errata and how.  Is it sufficient to just point to an external website, e.g. a Web page maintained by a volunteer or a group-edited Wiki page?  Or must the RFC be able to maintain Web pages and accept edits (diffs?) from approved errata maintainers?

[Ballot discuss]
I would like to see some discussion and perhaps rewriting of the sections on pre-approval editing and errata.

The pre-approval editing section reads as an advertisement for that process. While I understand that a requirement "to be able to do" is not the same as a requirement "to do", I feel this section will look pretty wierd if we don't do pre-approval editing or do it with another mechanism (e.g. volunteers, a different editing organization).  There are all kinds of things we could ask the RFC Editor to do in the future, not just limited to pre-approval editing, and this one is given special treatment. 

(Aside: My personal opinion is that pre-approval editing will delay documents, be much too expensive, and not substantially improve quality.  Having had a book copy-edited, I can staunchly claim that any difficulty understanding my book is my fault, and that the copy-editor had little chance of fixing it up significantly without themselves being a technical expert.)

The section on errata needs to be clearer on who is to maintain the errata and how.  Is it sufficient to just point to an external website, e.g. a Web page maintained by a volunteer or a group-edited Wiki page?  Or must the RFC be able to maintain Web pages and accept edits (diffs?) from approved errata maintainers?

[Ballot discuss]
I would like to see some discussion and perhaps rewriting of the sections on pre-approval editing and errata.

The pre-approval editing section reads as an advertisement for that process. While I understand that a requirement "to be able to do" is not the same as a requirement "to do", I feel this section will look pretty wierd if we don't do pre-approval editing or do it with another mechanism (e.g. volunteers, a different editing organization).  There are all kinds of things we could ask the RFC Editor to do in the future, not just limited to pre-approval editing, and this one is given special treatment. 

(Aside: My personal opinion is that pre-approval editing will delay documents, be much too expensive, and not substantially improve quality.  Having had a book copy-edited, I can staunchly claim that any difficulty understanding my book is my fault, and that the copy-editor had little chance of fixing it up significantly without themselves being a technical expert.)

The section on errata needs to be clearer on who is to maintain the errata and how.  Is it sufficient to just point to an external website, e.g. a Web page maintained by a volunteer or a group-edited Wiki page?  Or must the RFC be able to maintain Web pages and accept edits (diffs?) from approved errata maintainers?

[Ballot comment]
PreEdit-1

I think the requirement for pre-approval editing is probably stated wrong. We have not decided we are going to use this and would not want any contract pricing to be based on this. It might be better to phrase it more along the lines of, if the IETF decided to use pre-approval editing, it should be possible for the editor to provide .....

PostEdit-2

I find be confused on what a "document shepherd" is in this case. (I have same problem other places this term is used in this document). Would be nice to clarify this term.

RefVal-1

On the expected remain available part, I think this is only needed for normative references.


FormalVal-1

I find this very difficult to do for topics I am an expert in. I suspect it will be very difficult for the technical editor will be able check a files is a valid X.509 cert, or check that a file is a valid GML file that include definitions from multiple namespaces. I guess it depends on the level of checking we are thinking will happen. If we could clarify what sort of level of checking we are thinking of that would be better. It seems this might be better done by the WG. 


DocConvert-2

This requirement does not make sense to me. The editor should accept these files and do what with them?

Expedite-1

When one stream expedites some document, it slows down other streams. Do we want any other stream to be able to slow down IESG stream?

Stats-3 and Stats-4

This seems lacking definition on what we want and very expensive to track. I think we should carefully consider what we would be willing to pay to get this?

PubHelp-1 and PubHelp-2

I like the idea of this but it is not clear to me that it should be the publisher doing this.

TimeFrames-2

Given we have consensus to even do pre approval editing, I think saying that we agree it should be done 10 days might be a bit strong. It is not clear if you mean the average is under 10 days or it is always under 10 days. Having it this sort sounds fairly unrealistic to me. Given the burst nature of when documents get done at IETF, if we have enough resources to turn this around in 10 days most the time, they resource will be idle a large percentage of the year.

[Ballot discuss]
In Section 2.1, Figure 1, the document gives the actors involved in specific steps of a document's life cycle.  It does not include the "drafts publisher" actor or actions currently provided by the IETF Secretariat.  Req-STATUSTRK-2 later requires integration of state information provided by the Technical publisher with "IETF tools".  In practice, the principle integration would be with the "drafts publisher" tools; explicitly noting the existence of this actor and those tools seems to be a good idea.
 
The document says:

Req-POSTCORR-2 - The IETF technical publisher should only allow
      post-approval technical changes which have been approved by the
      appropriate party.  In the IETF standards process stream this
      includes the individual designated as the document shepherd, or
      sometimes, by referral, the IESG.

I believe the current policy is actually to seek approval from the "Responsible AD" as distinct from the document shepherd (commonly WG chair). Have I got that wrong, or is this meant to update that policy?  The same issue seems to arise in Req-POSTCORR-3.

In 3.9, the document says:

Req-DOCCONVERT-1 - The IETF technical publisher should accept as
      input ascii text files and publish documents as ascii text files,
      ps files, and pdf files.

  o  Req-DOCCONVERT-2 - The technical publisher should accept
      supplemental source files that may contain information such as:
      code, formal descriptions (XML, ASN.1, etc.) graphics, data
      files, etc.

Our current procedures allow a document editor/author to provide a secondary postscript or PDF file accompanying the ascii input file.  The author remains responsible for updating those through the editing process, and the RFC Editor is responsible for declaring that the updated files are equivalent.  Once that is done, the RFC Editor publishes them as secondary formats (with pdf distinct from the .txt.pdf files).  Req-DOCCONVERT-2 appears to change that process so that the author is no longer responsible for updating those formats, and the technical publisher is.  I do not believe that is clear, however, as  the task "accept supplemental source files" is not clear as to whether these are published as-is or as maintaining equivalence with the ascii text.  Clarifying this requirement would be useful.

[Ballot comment]
In 3.2, the document says:

If publication can take more than 6 months, it
  may be necessary to take measures to ensure the draft version remains
  available.

It might be useful to clarify that the actor taking these measure is not necessarily the technical publisher.

In 3.5, the document says:

  Discussion: The RFC Editor validates sections of a document
  containing MIBs, Augmented Backus Naur Form (ABNF), eXtensible Markup
  Language (XML), Abstract Syntax Notation One (ASN.1) and possibly
  other formal languages.

Req-FORMALVAL-1 - The IETF technical publisher should validate
      sections of documents containing formal languages.  In particular
      ASN.1, ABNF, and XML should be verified using appropriate tools.

It may be useful to clarify that "verification" for XML is for well-formed XML, rather than validation against a DTD; validation often has the second meaning for XML.  The text above the formal requirement makes this a bit ambiguous.

In 3.6, the document uses both the term "insert" and "populate" for assigned protocol parameters.  I personally found populate less confusing, as "insert" sounded more like IANA's registry action. YMMV.

In 3.9, the document says:

Discussion: Currently, the RFC Editor accepts input as an ascii text
  file (supplemented by xml if available)

I believe the RFC Editor also still accepts nroff.

Nit:

Post-publication Corrections: Appropriate processes must be
      defined with the IETF to ensure that errata is appropriately
      vetted and authorized.

"errata" is plural.

[Ballot comment]
The document needs to expand acronyms when first used. Most, such as "RFC", "IPR", and "BCP" the reader is very likely to know. Some like "IANA", "ETSI", etc most readers will know, but some might not. At least one "ISDs" I couldn't figure out.

[Ballot discuss]
Quoting 3.1:
>#  Discussion: Pre-approval review is not part of the normal process
>  flow with the IETF but this concept has been explored in the early
>  copy editing experiment.  Early feedback from the experiment has been
>  promising, however, the effectiveness of early copy-editing is still
>  being evaluated.


While I agree that early review was promising, I think that all the
later feedback--certainly all that I've seen--was very negative.  So I
question whether it is reasonable to recommend that the publisher
should do pre-review editing.  I think the first step in resolving
this discuss is to actually ask Bert and the rfc-editor for the
current state of the early copy editing experiment.

[Ballot comment]
Section 1: Missing Reference for what ISDs are.

Section 3.1: Missing reference for the early copy editing experiment.

Section 3.2:
  Discussion: This is not required.  A final approved version is
  available as a draft.  If publication can take more than 6 months, it
  may be necessary to take measures to ensure the draft version remains
  available.

I think that the formulation in the above is a bit strange considering that we do have a mechanism in place that don't kill of draft if the process takes more than 6 month.

[Ballot comment]
Section 1., paragraph 3:

>    The intention of this document is to clarify the IETF's consensus on
>    its requirements for its technical publication service.  This
>    document is not a discussion of how well the RFC Editor fulfils those
>    requirements.

  Nit: s/the RFC Editor/the RFC Edidor, who is the current IETF
  technical publisher, /


Section 2., paragraph 2:

>    This draft specifically addresses those documents published by the
>    IETF technical standards process.  In all cases, the requirements
>    have been written in generic terms, so that they may be used to
>    express the requirements of other RFC streams, elsewhere.

  s/RFC streams/streams of technical specifications/ - since that's what
  the preceding paragraphs claim?


Section 2.1., paragraph 5:

>                Figure 1: Stages of a Working Group Document

  Nit: Figure 1 breaks between pages.


Section 3.3., paragraph 15:

>    o  Req-POSTEDIT-3 - The IETF technical publisher should exercise
>      restraint in making stylistic changes that introduce a substantial
>      review load but only provides incremental increase in the clarity
>      of the specification.  Specific guidelines on the types of changes
>      allowed may be further specified, but ultimately restraint in
>      editing must be imposed by the IETF technical publisher.  Changes
>      for stylistic consistency should be done only when there are major
>      problems with the quality of the document.

  It may make sense for the IETF technical publisher to provide some
  style and grammar recommendations to the author community to minimize
  the need for such stylistic changes. (Oh, I see you have this below;
  nevermind.)


Section 3.11., paragraph 4:

>    o  Req-STATUSTRK-1 - The IETF technical publisher should make state
>      information publicly available for each document in the
>      publication process.

  Would be good if the the IETF technical publisher made this state
  (also) available through a documented interface, for tools
  development.


Section 3.16., paragraph 3:

>    o  Req-INDEX-1 - The IETF technical publisher should maintain the
>      index of all IETF published documents.

  Would be good if the the IETF technical publisher made this index
  (also) available through a documented interface, for tools
  development.


Section 4.1., paragraph 4:

>    o  Goal-TIMEFRAMES-1 - The consensus of the IETF community is that
>      an average publication time of under a month is desirable.  It is
>      understood that in some cases there will be delays outside of the
>      publisher's control. The actual performance targets and metrics
>      are expected to be determined as part of the contract negotiation
>      process.

  Can we maybe phrase these performance expectations a bit more
  precisely in terms of O(pages)?

PROOTO-like writeup from Leslie Daigle:

  (1.a)  Who is the Document Shepherd for this document?  Has the
          Document Shepherd personally reviewed this version of the
          document and, in particular, does he or she believe this
          version is ready for forwarding to the IESG for publication?

Leslie Daigle is acting as pseudo-shepherd for this document.  She has
personally reviewed this version and believes it is ready to
publish.

  (1.b)  Has the document had adequate review both from key WG members
          and from key non-WG members?  Does the Document Shepherd have
          any concerns about the depth or breadth of the reviews that
          have been performed?

This document was discussed on the techspec@ietf.org mailing list
and in 2 bof sessions at successive IETF meetings.  The IAB reviewed it
and supports its publication.  The shepherding AD put the -08 version
out for an IETF-wide last call.  In response to that, the authors
provided a revised version (-09) which is submitted for publication.
I believe the document has been well-reviewed.


  (1.c)  Does the Document Shepherd have concerns that the document
          needs more review from a particular or broader perspective,
          e.g., security, operational complexity, someone familiar with
          AAA, internationalization or XML?

No such concerns.


  (1.d)  Does the Document Shepherd have any specific concerns or
          issues with this document that the Responsible Area Director
          and/or the IESG should be aware of?  For example, perhaps you
          he or she is uncomfortable with certain parts of the document,
          or has concerns whether there really is a need for it.  In any
          event, if those issues have been discussed in the WG and the
          WG has indicated that it still wishes to advance the document,
          detail those concerns here.

No such concerns.

  (1.e)  How solid is the WG consensus behind this document?  Does it
          represent the strong concurrence of a few individuals, with
          others being silent, or does the WG as a whole understand and
          agree with it?

There is no working group here.  The document was shaped by input
gathered as described in (1.b).  The following key issues were brought
up during IETF last call (on the -08 document), which have been
addressed in -09:

i)  Timeframes/numbers expressed in the text:  -09 has been clarified
to capture timeframes or numbers that are viewed as requirements to
the IETF technical specification process -- from which service level
type numbers might be derived elsewhere (e.g., in a contract written
by the IASA).  The earlier concern was whether this document was trying
to provide the service level agreement or unrealistically
the technical publisher activity.  That should no longer be the case.

ii) Use of the term "IETF family"/scope of the document:  -09 makes
very clear that this document is for publication of IETF (standards
body) documents, and that argument is now moot.

iii)  Reusability:  though this was developed for the IETF stream
of publications, it is clearly desirable that any actual technical
publisher be presented with a reasonably unified set of requirements/
service expectations for all documents published under the larger
umbrella.  The -09 version has been rewritten so that the text
of the requirements is generic to document development and
publication, with the IETF-specific points provided as illustration.


  (1.f)  Has anyone threatened an appeal or otherwise indicated extreme
          discontent?  If so, please summarise the areas of conflict in
          separate email messages to the Responsible Area Director.  (It
          should be in a separate email because this questionnaire will
          be entered into the ID Tracker.)

Define "extreme discontent" in the IETF?!  Does everyone love it?  No.
Threats of appeal? Not since the second bof (and I'd have to check the
minutes to remind myself what John Klensin's specific issue was, but
the direction we took in the discussion caused him to withdraw the
proposal of appeal).


  (1.g)  Has the Document Shepherd verified that the document satisfies
          all ID nits?  (See http://www.ietf.org/ID-Checklist.html and
          http://tools.ietf.org/tools/idnits/).  Boilerplate checks are
          not enough; this check needs to be thorough.

Truthfully: no.


  (1.h)  Has the document split its references into normative and
          informative?  Are there normative references to documents that
          are not ready for advancement or are otherwise in an unclear
          state?  If such normative references exist, what is the
          strategy for their completion?  Are there normative references
          that are downward references, as described in [RFC3967]?  If
          so, list these downward references to support the Area
          Director in the Last Call procedure for them [RFC3967].

N/A


  (1.i)  For Standards Track and BCP documents, the IESG approval
          announcement includes a write-up.  Please provide such a
          write-up.  Recent examples can be found in the "Action"
          announcements for approved documents.  The approval
          announcement contains the following sections:


N/A

(from Stephen Hayes)

draft-mankin-pub-req-09 incorporates comments from the IETF LC.

The changes from version 08 are:

1. Although the scope of the document addresses only the IETF standards process stream, the requirements have been stated in general terms.  Where required additional specificity is added to the requirement for the case of the IETF standards process stream.  The scope (Section 2) has been updated to reflect this approach.

No distinction was made between "generic" and "standards specific" requirements as such a categorization would lead to contention and is ultimately useless since it is up to other organizations to decide which requirements to reuse.

2. To correct the mixing of performance metrics and performance targets in section 4, the proposed metrics were moved to section 3.19 (statistics).  The performance targets were recast as goals.

3. changed "refrain from" to "exercise restraint" in the post-edit requirements

4. section 3.4 - extended requirement REFVAL-1 to require that references are not only currently available, but are expected to be available in the future.

5. section 3.17 - extended requirement PUBACCESS-1 to include not only finding, but also retrieving documents.

6. The figure in 2.1 was updated so that actors were truly actors.

7. A requirement was added to section 3.21 to provide a help desk

8. Added a mention of "editoral management" in section 2.

9. Added a sentence to section 2 indicating that for the IETF standards process stream, the post-approval processing is initiated by the IESG after technical approval.

10. Indicated in requirements req-PREEDIT-1 and POSTEDIT-1 that the technical publisher should not do a technical review.

11. modified post-corr-3 to cover both suspect and unreasonable changes (with examples)

12. Various editoral corrections.