Robots Exclusion Protocol
draft-koster-rep-12
Revision differences
Document history
Date | Rev. | By | Action |
---|---|---|---|
2024-01-26
|
12 | Gunter Van de Velde | Request closed, assignment withdrawn: Nagendra Nainar Last Call OPSDIR review |
2024-01-26
|
12 | Gunter Van de Velde | Closed request for Last Call review by OPSDIR with state 'Overtaken by Events': Cleaning up stale OPSDIR queue |
2022-09-02
|
12 | (System) | RFC Editor state changed to AUTH48-DONE from AUTH48 |
2022-08-26
|
12 | (System) | RFC Editor state changed to AUTH48 |
2022-08-16
|
12 | (System) | RFC Editor state changed to RFC-EDITOR from EDIT |
2022-07-14
|
12 | (System) | RFC Editor state changed to EDIT |
2022-07-14
|
12 | (System) | IESG state changed to RFC Ed Queue from Approved-announcement sent |
2022-07-14
|
12 | (System) | Announcement was received by RFC Editor |
2022-07-14
|
12 | Amy Vezza | Downref to RFC 1945 approved by Last Call for draft-koster-rep-12 |
2022-07-14
|
12 | (System) | IANA Action state changed to No IANA Actions from In Progress |
2022-07-14
|
12 | (System) | IANA Action state changed to In Progress |
2022-07-14
|
12 | Amy Vezza | IESG state changed to Approved-announcement sent from Approved-announcement to be sent |
2022-07-14
|
12 | Amy Vezza | IESG has approved the document |
2022-07-14
|
12 | Amy Vezza | Closed "Approve" ballot |
2022-07-14
|
12 | Amy Vezza | Ballot approval text was generated |
2022-07-14
|
12 | Murray Kucherawy | IESG state changed to Approved-announcement to be sent from Approved-announcement to be sent::AD Followup |
2022-07-06
|
12 | Gary Illyes | New version available: draft-koster-rep-12.txt |
2022-07-06
|
12 | Gary Illyes | New version accepted (logged-in submitter: Gary Illyes) |
2022-07-06
|
12 | Gary Illyes | Uploaded new revision |
2022-06-28
|
11 | Gary Illyes | New version available: draft-koster-rep-11.txt |
2022-06-28
|
11 | (System) | New version approved |
2022-06-28
|
11 | (System) | Request for posting confirmation emailed to previous authors: Gary Illyes , Henner Zeller , Lizzi Sassman , Martijn Koster |
2022-06-28
|
11 | Gary Illyes | Uploaded new revision |
2022-06-24
|
10 | Tero Kivinen | Request for Last Call review by SECDIR Completed: Has Issues. Reviewer: Tirumaleswar Reddy.K. Submission of review completed at an earlier date. |
2022-06-20
|
10 | Tero Kivinen | Request for Last Call review by SECDIR Completed: Has Issues. Reviewer: Tirumaleswar Reddy.K. |
2022-06-16
|
10 | (System) | Removed all action holders (IESG state changed) |
2022-06-16
|
10 | (System) | Sub state has been changed to AD Followup from Revised ID Needed |
2022-06-16
|
10 | Gary Illyes | New version available: draft-koster-rep-10.txt |
2022-06-16
|
10 | (System) | New version approved |
2022-06-16
|
10 | (System) | Request for posting confirmation emailed to previous authors: Gary Illyes , Henner Zeller , Lizzi Sassman , Martijn Koster |
2022-06-16
|
10 | Gary Illyes | Uploaded new revision |
2022-06-16
|
09 | (System) | Changed action holders to Martijn Koster, Gary Illyes, Henner Zeller, Lizzi Sassman (IESG state changed) |
2022-06-16
|
09 | Cindy Morgan | IESG state changed to Approved-announcement to be sent::Revised I-D Needed from IESG Evaluation |
2022-06-16
|
09 | Zaheduzzaman Sarker | [Ballot comment] Thanks for working on this specification. I have skimmed through the document due to shortage of my time and haven't notices any transport … [Ballot comment] Thanks for working on this specification. I have skimmed through the document due to shortage of my time and haven't notices any transport protocol related issues. |
2022-06-16
|
09 | Zaheduzzaman Sarker | [Ballot Position Update] New position, No Objection, has been recorded for Zaheduzzaman Sarker |
2022-06-16
|
09 | Andrew Alston | [Ballot comment] Thanks for the draft. In writing this comment I'm going to state up front that what's in here is normally pretty far from … [Ballot comment] Thanks for the draft. In writing this comment I'm going to state up front that what's in here is normally pretty far from my field of expertise - so please excuse any confusion or inaccuracies - happy to be corrected. In section 2.1: Crawlers MUST find the group that matches the product token exactly, and then obey the rules of the group. To me this seems to conflict with: The matching of the product token MUST be case-insensitive. I was also a little confused here - since I noticed in 2.2 it states: product-token = identifier / "*" This would seem to imply a wildcard - if I'm reading this correctly, which also seems to not correlate with an exact match. |
2022-06-16
|
09 | Andrew Alston | [Ballot Position Update] New position, No Objection, has been recorded for Andrew Alston |
2022-06-16
|
09 | Lars Eggert | [Ballot comment] # GEN AD review of draft-koster-rep-09 CC @larseggert ## Nits All comments below are about very minor potential issues that you may choose … [Ballot comment] # GEN AD review of draft-koster-rep-09 CC @larseggert ## Nits All comments below are about very minor potential issues that you may choose to address in some way - or ignore - as you see fit. Some were flagged by automated tools (via https://github.com/larseggert/ietf-reviewtool), so there will likely be some false positives. There is no need to let me know what you did with these suggestions. ### Outdated references Reference `[RFC2616]` to `RFC2616`, which was obsoleted by `RFC7235`, `RFC7234`, `RFC7233`, `RFC7232`, `RFC7231`, and `RFC7230` (this may be on purpose). ### Grammar/style #### Section 2.2.1, paragraph 5 ``` he URI. The matching SHOULD be case sensitive. The most specific match found ^^^^^^^^^^^^^^ ``` This word is normally spelled with a hyphen. #### Section 2.2.1, paragraph 7 ``` and disallow rule is equivalent, the allow SHOULD be used. If no match is f ^^^^^^^^^ ``` After "the", the verb "allow" doesn't fit. Is "allow" spelled correctly? If "allow" is the first word in a compound adjective, use a hyphen between the two words. Using the verb "allow" as a noun may be non-standard. ## Notes This review is in the ["IETF Comments" Markdown format][ICMF], You can use the [`ietf-comments` tool][ICT] to automatically convert this review into individual GitHub issues. Review generated by the [`ietf-reviewtool`][IRT]. [ICMF]: https://github.com/mnot/ietf-comments/blob/main/format.md [ICT]: https://github.com/mnot/ietf-comments [IRT]: https://github.com/larseggert/ietf-reviewtool |
2022-06-16
|
09 | Lars Eggert | [Ballot Position Update] New position, No Objection, has been recorded for Lars Eggert |
2022-06-16
|
09 | Murray Kucherawy | [Ballot comment] IESG Note: The shepherd writeup claims Informational status, but consensus during the first Last Call was to move it to Proposed Standard, hence … [Ballot comment] IESG Note: The shepherd writeup claims Informational status, but consensus during the first Last Call was to move it to Proposed Standard, hence the discrepancy. |
2022-06-16
|
09 | Murray Kucherawy | Ballot comment text updated for Murray Kucherawy |
2022-06-16
|
09 | Francesca Palombini | [Ballot comment] Thank you for the work on this document. Many thanks to Todd Herr for his ART ART review: https://mailarchive.ietf.org/arch/msg/art/oaAMYwH7lWH4Xty01nSh1qSu7XU/. I wonder if … [Ballot comment] Thank you for the work on this document. Many thanks to Todd Herr for his ART ART review: https://mailarchive.ietf.org/arch/msg/art/oaAMYwH7lWH4Xty01nSh1qSu7XU/. I wonder if the reference to RFC 2616 should not be replaced with RFC 9111, or is there a reason to keep the old obsoleted reference? Francesca |
2022-06-16
|
09 | Francesca Palombini | Ballot comment text updated for Francesca Palombini |
2022-06-16
|
09 | Francesca Palombini | [Ballot comment] Thank you for the work on this document. Many thanks to Todd Herr for his ART ART review: https://mailarchive.ietf.org/arch/msg/art/oaAMYwH7lWH4Xty01nSh1qSu7XU/. Francesca |
2022-06-16
|
09 | Francesca Palombini | [Ballot Position Update] New position, No Objection, has been recorded for Francesca Palombini |
2022-06-15
|
09 | Paul Wouters | [Ballot comment] Thanks for doing this work, it is good to have it formally specified in an RFC. My only comment: This document … [Ballot comment] Thanks for doing this work, it is good to have it formally specified in an RFC. My only comment: This document specifies and extends the "Robots Exclusion Protocol" It would be nice if it said what the extending part is, as those parts might be less supported in the wild as these are presumably new and not widely supported yet. |
2022-06-15
|
09 | Paul Wouters | [Ballot Position Update] New position, Yes, has been recorded for Paul Wouters |
2022-06-15
|
09 | Éric Vyncke | [Ballot comment] # Éric Vyncke, INT AD, review of # Éric Vyncke, INT AD, review of draft-koster-rep-09 CC @evyncke Thank you for the work put … [Ballot comment] # Éric Vyncke, INT AD, review of # Éric Vyncke, INT AD, review of draft-koster-rep-09 CC @evyncke Thank you for the work put into this document. As a webmaster for some minor websites, this robots.txt is simple but efficient. Please find below some non-blocking COMMENT points (but replies would be appreciated even if only for my own education). Special thanks to Ted Hardie for the shepherd's detailed write-up even if I had liked to have the justification of the intended status. I hope that this helps to improve the document, Regards, -éric ## COMMENTS ### Section 1.1 As always, I personally find weird when an informational document uses BCP14 normative language; but the IESG has agreed on this use of BCP14. So, let it be ;-) ### Section 2.3 Suggest to remove the "http" example. ### Section 2.3.1.2 In `The server may respond to a robots.txt fetch request` should this be a `MAY`(to contradict my previous comment on the use of BCP14) ### Section 2.5 I must admit that this is the first time that I read "kibibyte"... Please use normal units or add a reference, e.g., https://simple.wikipedia.org/wiki/Kibibyte (BTW, thank you, I have learned a new word!) The sentence is rather hard to parse, could it be replaced by "Crawlers MUST be able to parse at least 500 KiB)" ? ## Notes This review is in the ["IETF Comments" Markdown format][ICMF], You can use the [`ietf-comments` tool][ICT] to automatically convert this review into individual GitHub issues. [ICMF]: https://github.com/mnot/ietf-comments/blob/main/format.md [ICT]: https://github.com/mnot/ietf-comments |
2022-06-15
|
09 | Éric Vyncke | [Ballot Position Update] New position, No Objection, has been recorded for Éric Vyncke |
2022-06-15
|
09 | Alvaro Retana | [Ballot Position Update] New position, No Objection, has been recorded for Alvaro Retana |
2022-06-14
|
09 | Roman Danyliw | [Ballot comment] Thank you for this work which formally standardizes long standing Internet practice. ** I recommend an editorial pass to consistently use client, crawlers … [Ballot comment] Thank you for this work which formally standardizes long standing Internet practice. ** I recommend an editorial pass to consistently use client, crawlers or robots. ** Is robots.txt practically used beyond the HTTPS, HTTP and FTP schemes? If not, it might be worth writing an applicability statement around these limited protocols, suggesting but not committing to broader use, and then be much more specific in Section 2.3 about error handling. ** Section 1. This document specifies the rules originally defined by the "Robots Exclusion Protocol" [ROBOTSTXT] that crawlers are expected to obey when accessing URIs. “… expected to obey when accessing URIs” seems too strong. Maybe “… are requested to honor.” There is no enforcement mechanism or even requirement that a crawler even check for robots.txt. ** Section 2.2.2. Crawlers SHOULD allow the following special characters. -- The subject of the normative language seems reversed. Isn’t this really (roughly) “the robots.txt file supports the following additional grammar”? It is in the ABNF. -- In a sense, the crawler is free to ignore anything the robots.txt says. Why is providing flexibility with a SHOULD any different than the rest of the content? ** Section 2.2.4. I’m not familiar with the operational practices that add additional “protocols” into robots.txt. How would a ‘sitemap’ file be added to robots.txt? Sitemaps is an XML blob. It would make for a non-conforming robot.txt and an invalid XML. ** Section 2.3.1.3 and 2.3.1.4. These sections map neatly to HTTP error codes. This protocol is scoped as being generic. What is the expected behavior if a protocol doesn’t distinguish between unavailable and unreachable? ** Section 2.3.1.3 and 2.3.1.4. Could equivalent error semantics be described for FTP. ** Section 2.3.1.4. For other undefined status codes, the crawler MUST assume the robots.txt is unreachable. Are those other HTTP code, or is this sentence a reference to other protocols? ** Section 2.3.1.4 If the robots.txt is undefined for a reasonably long period of time (for example, 30 days), clients MAY assume the robots.txt is Unavailable What does this persistent “unavailability” suggest about the crawler’s behavior. Since this is a “MAY”, does it continue to assume “complete disallow” or follow Section 2.3.1.3 behavior of “complete allow”? ** Section 2.3.1.5. (a) Crawlers SHOULD try to parse each line of the robots.txt file. (b) Crawlers MUST use the parseable rules. I appreciate that there is context from the Section header, but as written these appear conditional. Since (a) is a SHOULD, then (b) won’t always be true since I can choose not to parse certain lines. Is there a reason why (a) can’t be a MUST? ** Section 3. Recommend acknowledging the obvious that this protocol is for crawlers willing to following directives in the file. Also, paths, not URIs, are in the file. OLD The Robots Exclusion Protocol is not a substitute for more valid content security measures. Listing URIs in the robots.txt file exposes the URI publicly and thus makes the URIs discoverable. NEW The Robots Exclusion Protocol provides no access control and is advisory in nature to conforming crawlers. Non-compliant crawlers can avoid accessing the robots.txt file or ignore directives in it. Listing paths in the robots.txt file exposes them publicly and thus makes the discoverable. |
2022-06-14
|
09 | Roman Danyliw | [Ballot Position Update] New position, No Objection, has been recorded for Roman Danyliw |
2022-06-14
|
09 | Erik Kline | [Ballot Position Update] New position, No Objection, has been recorded for Erik Kline |
2022-06-09
|
09 | Robert Wilton | [Ballot comment] Hi, Thanks for this document, it looks like a useful update and it was a pretty easy read. I just have a few … [Ballot comment] Hi, Thanks for this document, it looks like a useful update and it was a pretty easy read. I just have a few comments/clarifications for the authors to consider that may improve this document: 1. Crawlers MUST find the group that matches the product token exactly, and then obey the rules of the group. The matching of the product token MUST be case-insensitive For me, matching exactly and also being case-insensitive seem to slightly contradict each other. Would the text be more clear if these were combined into a single sentence? 2. The most specific match is the match that has the most octets. Should the document specify that the match is from the root of the path? Otherwise it implies that any substring match of the path may be acceptable. 3. For the examples: Would it be helpful to include a line that has no space between the allow: and the match (e.g., allow:/foobar)? 4. Disallow: /*.gif$ Would this also be allowed without the initial '/' character? Is this worth clarifying/specifying? Thanks, Rob |
2022-06-09
|
09 | Robert Wilton | [Ballot Position Update] New position, No Objection, has been recorded for Robert Wilton |
2022-06-06
|
09 | Amanda Baber | IANA Review state changed to IANA OK - No Actions Needed from Version Changed - Review Needed |
2022-06-06
|
09 | Cindy Morgan | Placed on agenda for telechat - 2022-06-16 |
2022-06-05
|
09 | Murray Kucherawy | Ballot has been issued |
2022-06-05
|
09 | Murray Kucherawy | [Ballot Position Update] New position, Yes, has been recorded for Murray Kucherawy |
2022-06-05
|
09 | Murray Kucherawy | Created "Approve" ballot |
2022-06-05
|
09 | (System) | Changed action holders to Murray Kucherawy (IESG state changed) |
2022-06-05
|
09 | Murray Kucherawy | IESG state changed to IESG Evaluation from Waiting for AD Go-Ahead::AD Followup |
2022-06-05
|
09 | Murray Kucherawy | Ballot writeup was changed |
2022-06-03
|
09 | (System) | Removed all action holders (IESG state changed) |
2022-06-03
|
09 | (System) | Sub state has been changed to AD Followup from Revised ID Needed |
2022-06-03
|
09 | (System) | IANA Review state changed to Version Changed - Review Needed from IANA OK - No Actions Needed |
2022-06-03
|
09 | Gary Illyes | New version available: draft-koster-rep-09.txt |
2022-06-03
|
09 | (System) | New version approved |
2022-06-03
|
09 | (System) | Request for posting confirmation emailed to previous authors: Gary Illyes , Henner Zeller , Lizzi Harvey , Martijn Koster |
2022-06-03
|
09 | Gary Illyes | Uploaded new revision |
2022-06-02
|
08 | Murray Kucherawy | IESG state changed to Waiting for AD Go-Ahead::Revised I-D Needed from Waiting for AD Go-Ahead::AD Followup |
2022-06-02
|
08 | Murray Kucherawy | Changed action holders to Martijn Koster, Gary Illyes, Henner Zeller, Lizzi Harvey |
2022-06-02
|
08 | Murray Kucherawy | Awaiting response to INTDIR review. |
2022-06-02
|
08 | Murray Kucherawy | IESG state changed to Waiting for AD Go-Ahead::AD Followup from Waiting for Writeup |
2022-06-02
|
08 | (System) | IESG state changed to Waiting for Writeup from In Last Call |
2022-05-29
|
08 | Ralf Weber | Request for Last Call review by INTDIR Completed: Ready with Issues. Reviewer: Ralf Weber. Sent review to list. |
2022-05-17
|
08 | Michelle Thangtamsatid | (BEGIN IANA COMMENTS) IESG/Authors/WG Chairs: The IANA Functions Operator has reviewed draft-koster-rep-08, which is currently in Last Call, and has the following comments: We … (BEGIN IANA COMMENTS) IESG/Authors/WG Chairs: The IANA Functions Operator has reviewed draft-koster-rep-08, which is currently in Last Call, and has the following comments: We understand that this document doesn't require any registry actions. While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object. If this assessment is not accurate, please respond as soon as possible. For definitions of IANA review states, please see: https://datatracker.ietf.org/help/state/draft/iana-review Thank you, Michelle Thangtamsatid IANA Services Specialist (END IANA COMMENTS) |
2022-05-17
|
08 | (System) | IANA Review state changed to IANA OK - No Actions Needed from Version Changed - Review Needed |
2022-05-11
|
08 | Carlos Jesús Bernardos | Request for Last Call review by INTDIR is assigned to Ralf Weber |
2022-05-11
|
08 | Carlos Jesús Bernardos | Request for Last Call review by INTDIR is assigned to Ralf Weber |
2022-05-11
|
08 | Éric Vyncke | Requested Last Call review by INTDIR |
2022-05-06
|
08 | Todd Herr | Request for Last Call review by ARTART Completed: Ready with Nits. Reviewer: Todd Herr. Sent review to list. |
2022-05-05
|
08 | Barry Leiba | Request for Last Call review by ARTART is assigned to Todd Herr |
2022-05-05
|
08 | Barry Leiba | Request for Last Call review by ARTART is assigned to Todd Herr |
2022-05-05
|
08 | Cindy Morgan | The following Last Call announcement was sent out (ends 2022-06-02): From: The IESG To: IETF-Announce CC: Ted Hardie , draft-koster-rep@ietf.org, superuser@gmail.com, ted.ietf@gmail.com Reply-To: … The following Last Call announcement was sent out (ends 2022-06-02): From: The IESG To: IETF-Announce CC: Ted Hardie , draft-koster-rep@ietf.org, superuser@gmail.com, ted.ietf@gmail.com Reply-To: last-call@ietf.org Sender: Subject: Last Call: (Robots Exclusion Protocol) to Proposed Standard The IESG has received a request from an individual submitter to consider the following document: - 'Robots Exclusion Protocol' as Proposed Standard The IESG plans to make a decision in the next few weeks, and solicits final comments on this action. Please send substantive comments to the last-call@ietf.org mailing lists by 2022-06-02. Exceptionally, comments may be sent to iesg@ietf.org instead. In either case, please retain the beginning of the Subject line to allow automated sorting. Abstract This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1996 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. The file can be obtained via https://datatracker.ietf.org/doc/draft-koster-rep/ No IPR declarations have been submitted directly on this I-D. The document contains these normative downward references. See RFC 3967 for additional information: rfc1945: Hypertext Transfer Protocol -- HTTP/1.0 (Informational - Internet Engineering Task Force (IETF)) |
2022-05-05
|
08 | Cindy Morgan | IESG state changed to In Last Call from Last Call Requested |
2022-05-05
|
08 | Murray Kucherawy | Last call was requested |
2022-05-05
|
08 | Murray Kucherawy | IESG state changed to Last Call Requested from Waiting for AD Go-Ahead::AD Followup |
2022-05-05
|
08 | Murray Kucherawy | Last call announcement was generated |
2022-05-05
|
08 | Murray Kucherawy | Intended Status changed to Proposed Standard from Informational |
2022-05-05
|
08 | Murray Kucherawy | Last call announcement was generated |
2022-05-05
|
08 | Gary Illyes | New version available: draft-koster-rep-08.txt |
2022-05-05
|
08 | (System) | New version approved |
2022-05-05
|
08 | (System) | Request for posting confirmation emailed to previous authors: Gary Illyes , Henner Zeller , Lizzi Harvey , Martijn Koster |
2022-05-05
|
08 | Gary Illyes | Uploaded new revision |
2022-05-05
|
07 | (System) | Changed action holders to Murray Kucherawy (IESG state changed) |
2022-05-05
|
07 | (System) | Sub state has been changed to AD Followup from Revised ID Needed |
2022-05-05
|
07 | (System) | IANA Review state changed to Version Changed - Review Needed from IANA OK - No Actions Needed |
2022-05-05
|
07 | Gary Illyes | New version available: draft-koster-rep-07.txt |
2022-05-05
|
07 | (System) | New version approved |
2022-05-05
|
07 | (System) | Request for posting confirmation emailed to previous authors: Gary Illyes , Henner Zeller , Lizzi Harvey , Martijn Koster |
2022-05-05
|
07 | Gary Illyes | Uploaded new revision |
2022-04-21
|
06 | Jean Mahoney | Closed request for Last Call review by GENART with state 'Team Will not Review Version': AD will issue LC on a new version |
2022-04-21
|
06 | Jean Mahoney | Assignment of request for Last Call review by GENART to Robert Sparks was withdrawn |
2022-04-08
|
06 | Jean Mahoney | Request for Last Call review by GENART is assigned to Robert Sparks |
2022-04-08
|
06 | Jean Mahoney | Request for Last Call review by GENART is assigned to Robert Sparks |
2022-04-08
|
06 | Jean Mahoney | Assignment of request for Last Call review by GENART to Francis Dupont was withdrawn |
2022-04-07
|
06 | (System) | Changed action holders to Martijn Koster, Murray Kucherawy, Gary Illyes, Henner Zeller, Lizzi Harvey (IESG state changed) |
2022-04-07
|
06 | Murray Kucherawy | IESG state changed to Waiting for AD Go-Ahead::Revised I-D Needed from Waiting for Writeup |
2022-04-07
|
06 | (System) | IESG state changed to Waiting for Writeup from In Last Call |
2022-03-15
|
06 | (System) | IANA Review state changed to IANA OK - No Actions Needed from IANA - Review Needed |
2022-03-15
|
06 | (System) | (Via drafts-lastcall-comment@iana.org): IESG/Authors/WG Chairs: The IANA Functions Operator has reviewed draft-koster-rep-06, which is currently in Last Call, and has the following comments: We … (Via drafts-lastcall-comment@iana.org): IESG/Authors/WG Chairs: The IANA Functions Operator has reviewed draft-koster-rep-06, which is currently in Last Call, and has the following comments: We understand that this document doesn't require any registry actions. While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object. If this assessment is not accurate, please respond as soon as possible. Thank you, Michelle Thangtamsatid IANA Services Specialist |
2022-03-09
|
06 | Gunter Van de Velde | Request for Last Call review by OPSDIR is assigned to Nagendra Nainar |
2022-03-09
|
06 | Gunter Van de Velde | Request for Last Call review by OPSDIR is assigned to Nagendra Nainar |
2022-03-04
|
06 | Jean Mahoney | Request for Last Call review by GENART is assigned to Francis Dupont |
2022-03-04
|
06 | Jean Mahoney | Request for Last Call review by GENART is assigned to Francis Dupont |
2022-03-03
|
06 | Todd Herr | Request for Last Call review by ARTART Completed: Ready with Issues. Reviewer: Todd Herr. Sent review to list. |
2022-03-03
|
06 | Barry Leiba | Request for Last Call review by ARTART is assigned to Todd Herr |
2022-03-03
|
06 | Barry Leiba | Request for Last Call review by ARTART is assigned to Todd Herr |
2022-03-03
|
06 | Tero Kivinen | Request for Last Call review by SECDIR is assigned to Tirumaleswar Reddy.K |
2022-03-03
|
06 | Tero Kivinen | Request for Last Call review by SECDIR is assigned to Tirumaleswar Reddy.K |
2022-03-01
|
06 | Murray Kucherawy | Changed consensus to Yes from Unknown |
2022-02-28
|
06 | Cindy Morgan | IANA Review state changed to IANA - Review Needed |
2022-02-28
|
06 | Cindy Morgan | The following Last Call announcement was sent out (ends 2022-04-07): From: The IESG To: IETF-Announce CC: Ted Hardie , draft-koster-rep@ietf.org, superuser@gmail.com, ted.ietf@gmail.com Reply-To: … The following Last Call announcement was sent out (ends 2022-04-07): From: The IESG To: IETF-Announce CC: Ted Hardie , draft-koster-rep@ietf.org, superuser@gmail.com, ted.ietf@gmail.com Reply-To: last-call@ietf.org Sender: Subject: Last Call: (Robots Exclusion Protocol) to Informational RFC The IESG has received a request from an individual submitter to consider the following document: - 'Robots Exclusion Protocol' as Informational RFC The IESG plans to make a decision in the next few weeks, and solicits final comments on this action. Please send substantive comments to the last-call@ietf.org mailing lists by 2022-04-07. Exceptionally, comments may be sent to iesg@ietf.org instead. In either case, please retain the beginning of the Subject line to allow automated sorting. Abstract This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1996 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. The file can be obtained via https://datatracker.ietf.org/doc/draft-koster-rep/ No IPR declarations have been submitted directly on this I-D. |
2022-02-28
|
06 | Cindy Morgan | IESG state changed to In Last Call from Last Call Requested |
2022-02-28
|
06 | Cindy Morgan | Last call announcement was changed |
2022-02-28
|
06 | Murray Kucherawy | Last call was requested |
2022-02-28
|
06 | Murray Kucherawy | Ballot approval text was generated |
2022-02-28
|
06 | Murray Kucherawy | Ballot writeup was generated |
2022-02-28
|
06 | (System) | Changed action holders to Murray Kucherawy (IESG state changed) |
2022-02-28
|
06 | Murray Kucherawy | IESG state changed to Last Call Requested from AD Evaluation::AD Followup |
2022-02-28
|
06 | Murray Kucherawy | Last call announcement was generated |
2021-11-07
|
06 | (System) | Removed all action holders (IESG state changed) |
2021-11-07
|
06 | (System) | Sub state has been changed to AD Followup from Revised ID Needed |
2021-11-07
|
06 | Gary Illyes | New version available: draft-koster-rep-06.txt |
2021-11-07
|
06 | (System) | New version approved |
2021-11-07
|
06 | (System) | Request for posting confirmation emailed to previous authors: Gary Illyes , Henner Zeller , Lizzi Harvey , Martijn Koster |
2021-11-07
|
06 | Gary Illyes | Uploaded new revision |
2021-06-09
|
05 | (System) | Changed action holders to Martijn Koster, Gary Illyes, Henner Zeller, Lizzi Harvey (IESG state changed) |
2021-06-09
|
05 | Murray Kucherawy | IESG state changed to AD Evaluation::Revised I-D Needed from AD Evaluation::AD Followup |
2021-06-05
|
05 | (System) | Sub state has been changed to AD Followup from Revised ID Needed |
2021-06-05
|
05 | Gary Illyes | New version available: draft-koster-rep-05.txt |
2021-06-05
|
05 | (System) | New version approved |
2021-06-05
|
05 | (System) | Request for posting confirmation emailed to previous authors: Gary Illyes , Henner Zeller , Lizzi Harvey , Martijn Koster |
2021-06-05
|
05 | Gary Illyes | Uploaded new revision |
2020-12-13
|
04 | Murray Kucherawy | IESG state changed to AD Evaluation::Revised I-D Needed from AD Evaluation |
2020-12-10
|
04 | Murray Kucherawy | IESG state changed to AD Evaluation from Publication Requested |
2020-12-10
|
04 | Murray Kucherawy | As required by RFC 4858, this is the current template for the Document Shepherd Write-Up. Changes are expected over time. This version is dated … As required by RFC 4858, this is the current template for the Document Shepherd Write-Up. Changes are expected over time. This version is dated 1 November 2019. (1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header? This is informational, and this is correctly noted on the title page. (2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections: Technical Summary: This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1996 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Working Group Summary: This was not a working group document, though it was discussed by DISPATCH. Document Quality: The specified mechanism is very widely implemented and deployed; is the corner of crawler behavior management. Personnel: Ted Hardie is the Document Shepherd? Murray Kucherawy is the Responsible Area Director? (3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG. This document was reviewed by the document shepherd and changes in the ABNF and certain descriptions requested; those have been made, and the shepherd now believes this is ready for publication. (4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed? No. (5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place. No specialist reviews were requested. Note that the security considerations, specifically disclaims the use of robots.txt as a security mechanism. While this is not generally the approach taken for security considerations, it does highlight the principal concern: that the specification of crawler behavior might provide privacy properties or guarantees it does not. (6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here. I have no discomfort with this document. (7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why? Yes, they have. (8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures. No. (9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it? This is not relevant, as this is not a WG document. (10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.) No. (11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough. ID nits notes that some elements might benefit from and |
2020-12-10
|
04 | Murray Kucherawy | IETF WG state changed to Submitted to IESG for Publication |
2020-12-10
|
04 | Murray Kucherawy | IESG state changed to Publication Requested from AD is watching |
2020-12-08
|
04 | Ted Hardie | As required by RFC 4858, this is the current template for the Document Shepherd Write-Up. Changes are expected over time. This version is dated … As required by RFC 4858, this is the current template for the Document Shepherd Write-Up. Changes are expected over time. This version is dated 1 November 2019. (1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header? This is informational, and this is correctly noted on the title page. (2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections: Technical Summary: This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1996 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Working Group Summary: This was not a working group document, though it was discussed by DISPATCH. Document Quality: The specified mechanism is very widely implemented and deployed; is the corner of crawler behavior management. Personnel: Ted Hardie is the Document Shepherd? Murray Kucherawy is the Responsible Area Director? (3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG. This document was reviewed by the document shepherd and changes in the ABNF and certain descriptions requested; those have been made, and the shepherd now believes this is ready for publication. (4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed? No. (5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place. No specialist reviews were requested. Note that the security considerations, specifically disclaims the use of robots.txt as a security mechanism. While this is not generally the approach taken for security considerations, it does highlight the principal concern: that the specification of crawler behavior might provide privacy properties or guarantees it does not. (6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here. I have no discomfort with this document. (7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why? Yes, they have. (8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures. No. (9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it? This is not relevant, as this is not a WG document. (10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.) No. (11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough. ID nits notes that some elements might benefit from and |
2020-12-08
|
04 | Gary Illyes | New version available: draft-koster-rep-04.txt |
2020-12-08
|
04 | (System) | New version approved |
2020-12-08
|
04 | (System) | Request for posting confirmation emailed to previous authors: Lizzi Harvey , Gary Illyes , Henner Zeller , Martijn Koster |
2020-12-08
|
04 | Gary Illyes | Uploaded new revision |
2020-12-08
|
03 | Gary Illyes | New version available: draft-koster-rep-03.txt |
2020-12-08
|
03 | (System) | New version approved |
2020-12-08
|
03 | (System) | Request for posting confirmation emailed to previous authors: Henner Zeller , Martijn Koster , Lizzi Harvey , Gary Illyes |
2020-12-08
|
03 | Gary Illyes | Uploaded new revision |
2020-06-09
|
02 | Gary Illyes | New version available: draft-koster-rep-02.txt |
2020-06-09
|
02 | (System) | New version approved |
2020-06-09
|
02 | (System) | Request for posting confirmation emailed to previous authors: Lizzi Harvey , Henner Zeller , Martijn Koster , Gary Illyes |
2020-06-09
|
02 | Gary Illyes | Uploaded new revision |
2020-05-04
|
01 | Murray Kucherawy | Notification list changed to Ted Hardie <ted.ietf@gmail.com> |
2020-05-04
|
01 | Murray Kucherawy | Document shepherd changed to Ted Hardie |
2020-04-29
|
01 | Murray Kucherawy | Assigned to Applications and Real-Time Area |
2020-04-29
|
01 | Murray Kucherawy | Responsible AD changed to Murray Kucherawy |
2020-04-29
|
01 | Murray Kucherawy | IESG process started in state AD is watching |
2020-04-29
|
01 | (System) | Earlier history may be found in the Comment Log for /doc/draft-rep-wg-topic/ |
2020-04-29
|
01 | Murray Kucherawy | Intended Status changed to Informational from None |
2020-04-29
|
01 | Murray Kucherawy | Stream changed to IETF from None |
2020-01-07
|
01 | Gary Illyes | New version available: draft-koster-rep-01.txt |
2020-01-07
|
01 | (System) | New version approved |
2020-01-07
|
01 | (System) | Request for posting confirmation emailed to previous authors: Gary Illyes , Lizzi Harvey , Martijn Koster , Henner Zeller |
2020-01-07
|
01 | Gary Illyes | Uploaded new revision |
2019-07-07
|
00 | (System) | This document now replaces draft-rep-wg-topic instead of None |
2019-07-07
|
00 | Gary Illyes | New version available: draft-koster-rep-00.txt |
2019-07-07
|
00 | (System) | New version approved |
2019-07-07
|
00 | Gary Illyes | Request for posting confirmation emailed to submitter and authors: Gary Illyes , Lizzi Harvey , Martijn Koster , Henner Zeller |
2019-07-07
|
00 | Gary Illyes | Uploaded new revision |