Network Working Group Alex Zinin
Internet Draft Alcatel
Expiration Date: November 2005 May 2005
File name: draft-zinin-early-review-01.txt
Area Review Teams for
Early Cross-functional Reviews
draft-zinin-early-review-00.txt
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet Drafts are working documents of the Internet Engineering
Task Force (IETF), its Areas, and its Working Groups. Note that other
groups may also distribute working documents as Internet Drafts.
Internet Drafts are draft documents valid for a maximum of six
months. Internet Drafts may be updated, replaced, or obsoleted by
other documents at any time. It is not appropriate to use Internet
Drafts as reference material or to cite them other than as a "working
draft" or "work in progress".
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Abstract
This document contains a proposal for cross-functional IETF review
process that can be initiated at early stages of a document life
cycle. The approach is based on existing experience with area
directorates and other expert groups within the IETF.
Please send any comments on this document to the IESG (iesg@ietf.org)
1. Introduction
Zinin [Page 1]
INTERNET DRAFT Early Document Review May 2005
Cross-functional (intra-area and cross-area) review are the
properties of the IETF that are supposed to ensure high quality of
the produced technologies, their scalability and safety for the
Internet as a whole. It has been widely acknowledged that these
properties are among the core values of the IETF and have to be
preserved in order for the IETF to continue being a successful
engineering and standards organization.
Currently, the AD review and IESG review processes are the only
formal ways within the IETF to ensure cross-functional, and in
particular cross-area reviews. Since these reviews happen only when
the documents are submitted for final approval, issues are brought up
late in the process, often when contributors have invested a lot of
cycles into the document, the technology has possibly been
implemented, and changes in direction or certain technical details
are painful and frustrating. If interim cross-functional or cross-
area review is needed, it is currently done informally, and this
process is not necessarily coordinated with the following review by
the IESG members.
Creating a more formal and better described mechanism for cross-area
review that would be coordinated with the IESG review process, and
could be involved at any point within a life cycle of a document
should substantially improve the quality of the documents submitted
to the IESG and minimize the number of substantial issues identified
late in the process. A straight-forward way to do this could be to
request interim IESG review before the document is completed and
submitted for final approval. However, if applied to a considerable
number of documents (and we do want more documents to benefit from
cross-area review), this would put additional load on the IESG and
could impact document processing time.
This document proposes a cross-area review process that should have
better scaling characteristics. The method is based on delegation of
the document review function from ADs to the area review teams,
currently known as "area directorates" in some areas, or "doctor
groups" in others. Note that the described process does not obviate
the need for the cross-functional peer review performed by regular
IETF participants (members of the review teams or not). On the other
hand, the peer review process can be improved by directly (and
informally) soliciting comments from the review teams.
The proposed method is based on the experience several IETF Area
Directors have with involving groups of experts in the process of
early document review and during the IESG review cycle.
More specifically:
Zinin [Page 2]
INTERNET DRAFT Early Document Review May 2005
o the Routing Area Directors have been using the Routing Area
Directorate group (rtg-dir) for review of the documents coming
out of the WGs within the Routing Area before they are submit-
ted for final approval to the IESG. The directorate is also
asked to review certain documents appearing on the IESG agenda
from time to time.
o Operations directorate (ops-dir) has also been used for early
document review and during the IESG review period.
o The MIB-Doctors group is consistently involved in preparation
of MIB documents before they are brought to the IESG
<comment> Should add info on the Transport-Doctors group here
</comment>
At this point the process of document review by these expert groups
is not described and has no formal standing with the IETF. Not all
areas have similar expert groups and/or they are not publicly known
to the wide community. There is also no described way for WG chairs
to request review by experts groups in other areas and expect a guar-
anteed follow-up.
This proposal formalizes the notion of such expert review teams,
describes how they are used for cross-functional review, the relation
of this process to the final IESG review process, and how the cross-
functional review can be requested and should be followed up on.
2. Proposal
2.1 Overview
Briefly, the cross-functional review process may be described as fol-
lows.
Each area has an area review team (ART) which ADs delegate the
interim document review function to. When necessary (early in the
process, or during the WG Last call, or both), the WG chairs request
the review for a document by sending an e-mail to all required ARTs
(at a minimum the ART of the area the WG belongs to). The IETF-wide
Last Call announcement is cc'ed to all ARTs. Provided feedback is
taken to the WG for discussion. Consistency with the later IESG
review process is ensured through training of the reviewers by the
ADs and reviewers communicating the recommendations wrt the documents
to the ADs.
The following sections provide more details regarding the proposed
approach.
Zinin [Page 3]
INTERNET DRAFT Early Document Review May 2005
2.2 Assumptions
The described proposal is predicated on certain assumptions that are
worth spelling out:
1. The IESG remains the body responsible for final approval of
the IETF documents.
The implication of this assumption is that the documents will
finally have to go through the IESG review process with indi-
vidual IESG members checking the document. This implies that
interim reviews performed before that stage should be coordi-
nated with later IESG reviews. Otherwise it is possible that
the issues discussed previously will be brought up again. The
coordination is also need to usefully off-load (part of) the
document review function from individual ADs.
2. The ADs remain to be trusted by the community (through the
NomCom process) to function as final document reviewers and
are responsible for the document quality.
The implication here is that since the ADs are held responsi-
ble for the results of the document review, they need to feel
comfortable delegating this review function to the ART mem-
bers, which will have bearings on the method of ART member
selection.
The reasoning behind using these assumptions is to use the existing
mechanisms and tools (known running code) as much as possible and
avoid extreme changes to the document approval process that may
result in a DoS on it.
2.3 Area Review Teams
Each area creates an Area Review Team (ART) that is addressable
through a well-known mailing list address (e.g. "<area-
name>-review@ietf.org"). At a minimum, the group includes the ADs,
however it is expected that it will include technical experts, will-
ing to contribute their time to reviewing the IETF documents from the
same and other areas and providing consultation to the area direc-
tors. The ADs will delegate the review function for some (or all)
documents to ART members.
Selection of the ART members is done personally by the ADs. Possible
variations, however, may include open call for nominations, followed
up by ADs interviewing the candidates and personally approving them.
<comment>
Zinin [Page 4]
INTERNET DRAFT Early Document Review May 2005
See assumption 2 in section 2.2 for the reasons
</comment>
When a document needs to be reviewed by ART, the AD assigns two ART
members as "token holders". All ART members are encouraged to review
the document, however, the token holders are held responsible for
providing comments within a 2-week time frame and following up on
them with the document authors and/or the hosting WG. The token hold-
ers will also provide the ADs with their recommendation including the
summary of the discussion, the list of issues and how they have been
addressed.
2.4 Cross-Area Review Process
This section describes the actual cross-functional review process
that can be initiated at any point within the life cycle of a docu-
ment. This process is automatically initiated whenever an IETF- wide
Last Call is started for a document.
1. The set of area review teams engaged is determined by the ini-
tiator of the review process in consultation with the respon-
sible AD. At a minimum, this set includes the ART of the host-
ing area. For the IETF Last Call, this set includes all ARTs.
<comment>
the stimulus behind choosing the right set of review teams is
to get the review comments that are likely to be brought up
during the final IESG review earlier in the process
</comment>
The set of review teams may also include the IAB review team
(assuming this is a subset of IAB members, or the IAB itself
otherwise).
2. An e-mail message with the review request is sent to the mail-
ing lists of all ARTs that need to be engaged.
3. ADs for each engaged ART have the choice of either "signing
off" on the referred document if they believe that the docu-
ment does not need a detailed review from the perspective of
that area, or assigning the token holders (2 persons) that
will conduct the document review within a 2-week time frame.
Other members of the review teams are strongly encouraged to
provide feedback, but will not be held responsible for this by
the ADs.
Zinin [Page 5]
INTERNET DRAFT Early Document Review May 2005
4. Document reviewers bring their concerns to the attention of
the document authors and/or the WG via e-mail communication on
the WG mailing list and (if necessary) the mailing list of the
ART they are members of.
5. Based on the discussion with the authors/within the WG, the
reviewers provide their responsible AD with a recommendation
regarding the document. The recommendation includes the sum-
mary of the review, as well as the list of issues and their
resolution.
6. The authors/WG treat the feedback as part of the WG or IETF-
wide review process
<comment>
An important point here is that the discussion initiated by
this review process is integrated within the normal WG
process, rather than treated as a pronouncement from a higher
authority.
</comment>
7. If a document returns to an ART (e.g., the document is under
the IETF Last Call and was reviewed during the WG Last Call),
the same token holders will "own" the document whenever possi-
ble. The token holders check that the new revision of the doc-
ument reflects the previous discussion correctly. If no addi-
tional concerns arise, the recommendation to the ADs of the
ART remain the same.
8. The ADs have the power to bring up during the IESG review
process the reviewers' comments that were not addressed by the
authors/WG if they believe this is appropriate. The ADs also
have the power to override the comments from their correspond-
ing review teams.
<comment>
The above gives the ADs the ability to insist on fixing cer-
tain comments that they believe represent serious issues if
they were discarded while processing the cross-area review
feedback during the WG process as described above. This also
gives them the right to withdraw certain points from the con-
sideration or change them if they believe this is appropriate
for the progress of a document. This is consistent with the
concept that the ADs are responsible for the final document
approval and that the review teams provide their expert
Zinin [Page 6]
INTERNET DRAFT Early Document Review May 2005
recommendations based on the discussions within the WG.
</comment>
2.5 Role of Cross-Area Review within the Standards Process
The cross-function review process is integrated within the IETF Stan-
dards Process as described below:
1. WG process:
When initiated early within the life cycle of a document, the
feedback from the cross-area review process is considered part
of the WG discussion and consensus forming process.
2. WG Last Calls:
It is expected that the cross-area review will also be
requested during the WG LC period. Procedurally, this will
have the same value as during the WG process, but would ensure
that final cross-area checks are performed before the docu-
ments comes to the IESG.
3. AD-review process
It is expected that the ADs will use the review teams to dele-
gate the review function and thus off-load a considerable part
of this function when and as deemed appropriate. The ADs are
expected to coordinate with the ART members to ensure that
consistent review criteria is applied to documents so that
most issues that would otherwise be brought up during the AD-
review process are resolved earlier in the life cycle of the
document.
<comment>
Note that ADs are given a tool they can use to off-load docu-
ment review to the extent they believe is necessary, but they
are not required to do so. It is then left to the ADs to make
sure they are using this tool appropriately and sufficiently.
</comment>
4. IETF Last Call
As all ARTs are informed about IETF Last Calls, it is expected
that by the time the documents is on the IESG agenda, it will
Zinin [Page 7]
INTERNET DRAFT Early Document Review May 2005
have received adequate cross-functional review and all ADs
will have some recommendation on the document from their ARTs.
5. IESG review process
It is expected that individual ADs will organize the review
process in the review teams in such a way that a positive rec-
ommendation from the review team should be sufficient for the
ADs to feel comfortable that most of the possible technical
issues have been identified, followed up on, and resolved, and
that the ADs only need to look for very high-level, architec-
tural issues. This, in turn, should a) decrease the amount of
time the ADs need to spend on the document review and follow
up, and b) minimize the number of "late surprises" arising
during the IESG review process.
2.6 Initiation of Review Process
The cross-functional review process can be initiated either by an AD
or by a WG chair after consultation with the ADs.
<comment>
Consultation with the AD is a sanity check to make sure the set of
engage ARTs is chosen right
</comment>
The review process may be initiated at an early stage of a document
(e.g. when the WG is starting to consider an approach) to ensure
architectural validity and correctness of the general direction
The review process should be initiated as part of the WG LC to mini-
mize late surprises during the IESG review process
The review process must be initiated for all documents going through
the IETF Last Call.
The same process may be used by the ADs and the RFC-Editor to request
cross-functional review for individual submissions they are shepherd-
ing or checking for conflicts.
A simplified version of this process (no token holders, no issue
tracking within ARTs, etc.) can be used to inform ART members about
technical discussions and solicit their comments.
2.7 Documenting Results of Review
Zinin [Page 8]
INTERNET DRAFT Early Document Review May 2005
It is important that the feedback from ARTs and the results of the
discussion with the authors/WG are documented for later reference
during the IESG review process.
For WG documents this is ensures by the WG chairs who keep track of
the issues (as part of the improved WG process) and summarize them
when submitting the document to their ADs for IESG processing.
For individual submissions this function is either performed by the
review initiator (an AD) or delegated to a member of the review team
with a consequent report to the ADs
For ARTs in the other (non-hosting) area, token holders within the
review team assigned to the documents are responsible for tracking
the issues on their side and summarizing them in their recommendation
to the ADs
2.8 Trust, Responsibility, and Accountability
An important aspect of the proposal documented here is that the mod-
els of trust, responsibility, and accountability currently used and
practiced within the IETF are not changed.
More specifically, the ADs, selected by the NomCom process as indi-
viduals "trusted" by the community to perform the ultimate technical
review and document approval functions, are still held responsible
and accountable for these functions. In other words, the tool of del-
egating the document review function to the review teams does not
remove the responsibility of the ADs for the results of such review.
The ADs are expected to personally ensure that the individuals
selected by them as members of the review teams have appropriate
qualification (through required training if needed) to perform this
function. It is also the AD's responsibility to adjust the list of
members (hire more members or fire ill-performing ones) to maintain
adequacy of the review process and require level of off-loading.
2.10 Motivation and Credit
The following methods are proposed to make the role of an ART member
attractive and keep ART members motivated and acknowledged:
1. Formalization of the review team role.
Formal introduction of ARTs within the IETF standard process
should result in recognition of this role individual members
and their employers.
Zinin [Page 9]
INTERNET DRAFT Early Document Review May 2005
2. Open area meetings (plenary)
It is possible to give ART members public exposure by holding
ART plenary during the open area meetings
3. Acknowledging reviewers
Reviewers engaged in ARTs are acknowledged in the published
RFCs.
4. Dots on the badges
ART members are identified at the face-to-face meetings with
dots on their badges
3. Problems being addressed
The proposal described here addresses the following problems experi-
enced within the IESG and IETF in general:
1. Low document quality
It is expected that additional review of the documents should
substantially improve the quality of the IETF documents.
2. Individual AD load
It is expected that improved quality of the documents submit-
ted to the ADs for AD-review, should decrease the amount of
time spent on document review and follow-up on the issues.
Delegation of the document review function should also result
in considerable off-loading.
3. Overall IESG load
Is is expected that improved quality of the documents submit-
ted to the IESG should decrease the document review load on
the IESG. Reports from the ARTs on previously reviewed docu-
ments should also make it easier for individual IESG members
to assess the quality of incoming documents.
4. Late surprises
Earlier cross-functional review coordinated with later IESG
review should minimize the number of unexpected issues identi-
fied at the later stages of a document life cycle.
Zinin [Page 10]
INTERNET DRAFT Early Document Review May 2005
5. Lack of cross-area review
This proposal directly encourages cross-area review
6. Lack of cross-functional expertise
This proposal should encourage learning of technologies in
different areas of IETF and should help growing cross-area
expertise.
7. Growing future management
Involvement of more IETF participants in the Standards Process
should help increase the number of individuals capable of per-
forming the tasks of WG chairs and IESG members.
4. Security Considerations
This type of non-protocol document does not directly affect the secu-
rity of the Internet. However, the cross-functional review process
described here should improve the security aspects of specific
approaches being reviewed.
5. References
6. Acknowledgements
The author would like to thank Harald Alvestrand and Ted Hardie for
their comments on the document.
7. Author's address
Alex Zinin
Alcatel
E-mail: zinin@psg.com
Copyright (C) The Internet Society (2005). This document is subject
to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights."
This document and the information contained herein are provided on
an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRE-
SENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE
INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
Zinin [Page 11]
INTERNET DRAFT Early Document Review May 2005
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Zinin [Page 12]
