Robots Exclusion Protocol
draft-koster-rep-12

Approval announcement
Draft of message to be sent after approval:

Announcement

From: The IESG <iesg-secretary@ietf.org>
To: IETF-Announce <ietf-announce@ietf.org>
Cc: Ted Hardie <ted.ietf@gmail.com>, The IESG <iesg@ietf.org>, draft-koster-rep@ietf.org, rfc-editor@rfc-editor.org, superuser@gmail.com, ted.ietf@gmail.com
Subject: Protocol Action: 'Robots Exclusion Protocol' to Proposed Standard (draft-koster-rep-12.txt)

The IESG has approved the following document:
- 'Robots Exclusion Protocol'
  (draft-koster-rep-12.txt) as Proposed Standard

This document has been reviewed in the IETF but is not the product of an IETF
Working Group.

The IESG contact person is Murray Kucherawy.

A URL of this Internet Draft is:
https://datatracker.ietf.org/doc/draft-koster-rep/

Ballot Text

Technical Summary

   This document specifies and extends the "Robots Exclusion
   Protocol" <http://www.robotstxt.org/> method originally defined by
   Martijn Koster in 1996 for service owners to control how content
   served by their services may be accessed, if at all, by automatic
   clients known as crawlers.

Working Group Summary

   This is an AD-sponsored document.

   It went through two Last Calls.  The first had it at Informational with a
   "no derivative works" clause, and the community objected strongly to the latter
   and expressed a clear preference for the Standards Track, so the second Last
   Call was done after those changes were made.

   There is general support for publication.  Suggested changes were duly considered,
   so that feedback has been addressed.  There has been some expression of dismay that
   this didn't go via a working group.

Document Quality

   This document describes a protocol that has already been widely implemented.
   Some large web property operators have indicated their support for the
   protocol as documented.

Personnel

   Ted Hardie is the document shepherd.
   Murray Kucherawy is the responsible Area Director.

RFC Editor Note