As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.
This version is dated 1 November 2019.
(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?
This is informational, and this is correctly noted on the title page.
(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:
This document specifies and extends the "Robots Exclusion
Protocol" <http://www.robotstxt.org/> method originally defined by
Martijn Koster in 1996 for service owners to control how content
served by their services may be accessed, if at all, by automatic
clients known as crawlers.
Working Group Summary:
This was not a working group document, though it was discussed by DISPATCH.
The specified mechanism is very widely implemented and deployed; is the corner of crawler behavior management.
Ted Hardie is the Document Shepherd? Murray Kucherawy is the Responsible Area Director?
(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.
This document was reviewed by the document shepherd and changes in the ABNF and certain descriptions requested; those have been made, and the shepherd now believes this is ready for publication.
(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?
(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.
No specialist reviews were requested. Note that the security considerations, specifically disclaims the use of robots.txt as a security mechanism. While this is not generally the approach taken for security considerations, it does highlight the principal concern: that the specification of crawler behavior might provide privacy properties or guarantees it does not.
(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.
I have no discomfort with this document.
(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?
Yes, they have.
(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.
(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?
This is not relevant, as this is not a WG document.
(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)
(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.
ID nits notes that some elements might benefit from <CODE BEGINS> and <CODE ENDS>. The shepherd did not advice this because the relevant parts of the document are ABNF.
(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.
These were not required.
(13) Have all references within this document been identified as either normative or informative?
(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?
(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.
(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.
(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 8126).
The document has no actions for IANA.
(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.
(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, YANG modules, etc.
The ABNF was checked using Bill Fenner's ABNF checker and passed.
(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools (https://trac.ietf.org/trac/ops/wiki/yang-review-tools) for syntax and formatting validation? If there are any resulting errors or warnings, what is the justification for not fixing them at this time? Does the YANG module comply with the Network Management Datastore Architecture (NMDA) as specified in RFC8342?
There is no YANG module in this document.