"Son of 1036": News Article Format and Transmission
draft-spencer-usefor-son-of-1036-01

".bitnet" pseudo-domains is
   permissible, for registered hosts in them, but discouraged) and its
   relayer name, specify the date when the reply was generated and the
   message ID of the whogets message being replied to, give the path
   list (from the Path header) of the whogets message (which MAY, if
   absolutely necessary, be truncated to a convenient length, but MUST
   contain at least the leading three relayer names), and indicate the
   version of relayer software responding.  Note that these lines are
   part of the BODY even though their format resembles that of headers.
   Despite the apparently-fixed order specified by the syntax above,
   they can appear in any order, but there must be exactly one of each.

   After those preliminaries, and an empty line to unambiguously define
   their end, the remaining lines are the relayer names (which MAY be
   accompanied by the corresponding domain names, if known) of systems
   which the responding system passes the target newsgroup to.  Only the

Henry Spencer             Expires January 2010                 [Page 62]
Internet-Draft                Son of 1036                  22 July 2009

   names of news relayers are to be included.

      NOTE:  It is desirable for a reply to identify its source by
      both domain name and relayer name because news propagation is
      governed by the latter but location in a broader context is
      best determined by the former.  The date and whogets message ID
      should, in principle, be present in the MAIL headers, but are
      included in the body for robustness in the presence of
      uncooperative mail systems.  The reason for the path list is
      discussed below.  Adding version information eliminates the
      need for a separate message to gather it.

      NOTE:  The limitation of pass lines to contain only names of
      news relayers is meant to exclude names used within a single
      host (as identifiers for mail gateways, portions of
      ihave/sendme implementations, etc.), which do not actually
      refer to other hosts.

   A relayer which is unaware of the existence of the target newsgroup
   MUST NOT reply to a whogets message at all, although this MUST NOT
   influence decisions on whether to pass the article on to other
   relayers.

      NOTE:  While this may result in discontinuous maps in cases
      where some hosts have not honored requests for creation of a
      newsgroup, it will also prevent a flood of useless responses in
      the event that a whogets message intended to map a small region
      "leaks" out to a larger one.  The possibility of discontinuous
      recognition of a newsgroup does make it important that the
      whogets message itself continue to propagate (if other criteria
      permit).  This is also the reason for the inclusion of the
      whogets message's path list, or at least the leading portion of
      it, in the reply:  to permit reconstruction of at least small
      gaps in maps.

   Different networks set different rules for the legitimacy of these
   messages, given that they may reveal details of organization-internal
   topology that are sometimes considered proprietary.

      NOTE:  On Usenet, in particular, willingness to respond to
      these messages is held to be a condition of network membership:
      the topology of Usenet is public information.  Organizations
      wishing to belong to such networks while keeping their internal
      topology confidential might wish to organize their internal
      news software so that all articles reaching outsiders appear to
      be from a single "gatekeeper" system, with the details of
      internal topology hidden behind that system.

      UNRESOLVED ISSUE:  It might be useful to have a way to set some
      sort of hop limit for these.

Henry Spencer             Expires January 2010                 [Page 63]
Internet-Draft                Son of 1036                  22 July 2009

7.6.  checkgroups

   The checkgroups control message contains a supposedly authoritative
   list of the valid newsgroups within some subset of the newsgroup name
   space:

      checkgroups-arguments  =
      checkgroups-body       = [ invalidation ] valid-groups
                             / invalidation
      invalidation           = "!" plain-component
                               *( "," plain-component ) eol
      valid-groups           = 1*( description-line eol )

   There are no arguments.  The body lines (except possibly for an
   initial invalidation) each contain a description line for a
   newsgroup, as defined under the newgroup message (section 7.3).

      NOTE:  Some other, ill-defined, forms of the checkgroups body
      were formerly used.  See appendix A.

   The checkgroups message applies to all hierarchies containing any of
   the newsgroups listed in the body.  The checkgroups message asserts
   that the newsgroups it lists are the only newsgroups in those
   hierarchies.  If there is an invalidation, it asserts that the
   hierarchies it names no longer contain any newsgroups.

   Processing a checkgroups message MAY cause a local list of newsgroup
   descriptions to be updated.  It SHOULD also cause the local lists of
   newsgroups (and their moderation statuses) in the mentioned
   hierarchies to be checked against the message.  The results of the
   check MAY be used for automatic corrective action, or MAY be reported
   to the news administrator in some way.

      NOTE:  Automatically updating descriptions of existing
      newsgroups is relatively safe.  In the case of newsgroup
      additions or deletions, simply notifying the administrator is
      generally the wisest action, unless perhaps the message can be
      determined to have originated within a cooperating subnet whose
      members are considered trustworthy.

      NOTE:  There is a problem with the checkgroups concept:  not
      all newsgroups in a hierarchy necessarily propagate to the same
      set of machines.  (Notably, there is a set of newsgroups known
      as the "inet" newsgroups, which have relatively limited
      distribution but coexist in several hierarchies with more
      widely-distributed newsgroups.)  The advice of checkgroups
      should always be taken with a grain of salt, and should never
      be followed blindly.

8.  Transmission Formats

   While this Draft does not specify transmission methods except to
   place a few constraints on them, there are some data formats used
   only for transmission that are unique to news.

Henry Spencer             Expires January 2010                 [Page 64]
Internet-Draft                Son of 1036                  22 July 2009

8.1.  Batches

   For efficient bulk transmission and processing of news articles, it
   is often desirable to transmit a number of them as a single block of
   data, a "batch".  The format of a batch is:

      batch         = 1*( batch-header article )
      batch-header  = "#! rnews " article-size eol
      article-size  = 1*digit

   A batch is a sequence of articles, each prefixed by a header line
   that includes its size.  The article size is a decimal count of the
   octets in the article, counting each EOL as one octet regardless of
   how it is actually represented.

      NOTE:  A relayer might wish to accept either a single article
      or a batch as input.  Since "#" cannot appear in a header name,
      examination of the first octet of the input will reveal its
      nature.

      NOTE:  In the header line, there is exactly one blank before
      "rnews", there is exactly one blank after "rnews", and the EOL
      immediately follows the article size.  Beware that some
      software inserts non-standard trash after the size.

      NOTE:  Despite the similarity of this format to the
      executable-script format used by some operating systems, it is
      EXTREMELY unwise to just feed incoming batches to a command
      interpreter in the anticipation that it will run a command
      named "rnews" to process the batch.  Unless arrangements are
      made to very tightly restrict the range of commands that can be
      executed by this means, the security implications are
      disastrous.

8.2.  Encoded Batches

   When transmitting news, especially over communications links that are
   slow or are billed by the bit, it is often desirable to batch news
   and apply data compression to the batches.  Transmission links
   sending compressed batches SHOULD use out-of-band means of
   communication to specify the compression algorithm being used.  If
   there is no way to send out-of-band information along with a batch,
   the following encapsulation for a compressed batch MAY be used:

      ec-batch             = "#! " compression-keyword eol
                             compressed-batch
      compression-keyword  = "cunbatch"

   A line containing a keyword indicating the type of compression is
   followed by the compressed batch.  The only truly widespread
   compression keyword at present is "cunbatch", indicating compression
   using the widely-distributed "compress" program.  Other compression
   keywords MAY be used by mutual agreement between the hosts involved.

Henry Spencer             Expires January 2010                 [Page 65]
Internet-Draft                Son of 1036                  22 July 2009

      NOTE:  An encapsulated compressed batch is NOT, in general, a
      text file, despite having an initial text line.  This
      combination of text and non-text data is often awkward to
      handle; for example, standard decompression programs cannot be
      used without first stripping off the initial line, and that in
      turn is painful to do because many text-handling tools that are
      superficially suited to the job do not cope well with non-text
      data.  Hence the recommendation that out-of-band communication
      be used instead when possible.

      NOTE:  For UUCP transmission, where a batch is typically
      transmitted by invoking the remote command "rnews" with the
      batch as its input stream, a plausible out-of-band method for
      indicating a compression type would be to give a compression
      keyword in an option to "rnews", perhaps in the form:

         rnews -d decompressor

      where "decompressor" is the name of a decompression program
      (e.g. "uncompress" for a batch compressed with "compress" or
      "gunzip" for a batch compressed with "gzip").  How this
      decompression program is located and invoked by the receiving
      relayer is implementation-specific.

      NOTE:  See the notes in section 8.1 on the inadvisability of
      feeding batches directly to command interpreters.

      NOTE:  There is exactly one blank between "#!" and the
      compression keyword, and the EOL immediately follows the
      keyword.

8.3.  News Within Mail

   It is often desirable to transmit news as mail, either for the
   convenience of a human recipient or because that is the only type of
   transmission available on a restrictive communication path.

   Given the similarity between the news format and the MAIL format, it
   is superficially attractive to just send the news article as a mail
   message.  This is typically a mistake:  mail-handling software often
   feels free to manipulate various headers in undesirable ways (in some
   cases, such as Sender, such manipulation is actually mandatory), and
   mail transmission problems etc. MUST be reported to the
   administrators responsible for the mail transmission rather than to
   the article's author.  In general, news sent as mail should be
   encapsulated to separate the mail headers and the news headers.

   When the intended recipient is a human, any convenient form of
   encapsulation may be used.  Recommended practice is to use MIME
   encapsulation with a content type of "message/news", given that news
   articles have additional semantics beyond what "message/rfc822"
   implies.

      NOTE:  "message/news" was registered as a standard subtype by

Henry Spencer             Expires January 2010                 [Page 66]
Internet-Draft                Son of 1036                  22 July 2009

      IANA 22 June 1993.

   When mail is being used as a transmission path between two relayers,
   however, a standard method is desirable.  Currently the standard
   method is to send the mail to an address whose local part is "rnews",
   with whatever mail headers are necessary for successful transmission.
   The news article (including its headers) is sent as the body of the
   mail message, with an "N" prepended to each line.

      NOTE:  The "N" reduces the probability of an innocent line in a
      news article being taken as a magic command to mail software,
      and makes it easy for receiving software to strip off any lines
      added by mail software (e.g. the trailing empty line added by
      some UUCP mail software).

   This method has its weaknesses.  In particular, it assumes that the
   mail transmission channel can transmit nearly-arbitrary body text
   undamaged.  When mail is being used as a transmission path of last
   resort, however, the mail system often has inconvenient preconceived
   notions about the format of message bodies.  Various ad-hoc encoding
   schemes have been used to avoid such problems.  The recommended
   method is to send a news article or batch as the body of a MIME mail
   message, using content type "application/news-transmission" and
   MIME's "base64" encoding (which is specifically designed to survive
   all known major mail systems).

      NOTE:  In the process, MIME conventions could be used to
      fragment and reassemble an article which is too large to be
      sent as a single mail message over a transmission path that
      restricts message length.  In addition, the "conversions"
      parameter to the content type could be used to indicate what
      (if any) compression method has been used.  And the Content-MD5
      header [RFC1544] can be used as a "checksum" to provide high
      confidence of detecting accidental damage to the contents.

      UNRESOLVED ISSUE:  The "conversions" parameter no longer
      exists.  What should be done about this, if anything?

      NOTE:  It might look tempting to use a content type such as
      "message/X-netnews", but MIME bans non-trivial encodings of the
      entire body of messages with content type "message".  The
      intent is to avoid obscuring nested structure underneath
      encodings.  For inter-relayer news transmission, there is no
      nested structure of interest, and it is important that the
      entire article (including its headers, not just its body) be
      protected against the vagaries of intervening mail software.
      This situation appears to fit the MIME description of
      circumstances in which "application" is the proper content
      type.

      NOTE:  "application/news-transmission", with a "conversions"
      parameter, was registered as a standard subtype by IANA 22 June
      1993.

Henry Spencer             Expires January 2010                 [Page 67]
Internet-Draft                Son of 1036                  22 July 2009

      UNRESOLVED ISSUE:  The "conversions" parameter no longer exists
      in MIME.  What should we do about this?

8.4.  Partial Batches

      UNRESOLVED ISSUE:  The existing batch conventions assemble
      (potentially) many articles into one batch.  Handling very
      large articles would be substantially less troublesome if there
      was also a fragmentation convention for splitting a large
      article into several batches.  Is this worth defining at this
      time?

9.  Propagation and Processing

   Most aspects of news propagation and processing are implementation-
   specific.  The basic propagation algorithms, and certain details of
   how they are implemented, nevertheless need to be standard.

   There are two important principles that news implementors (and
   administrators) need to keep in mind.  The first is the well-known
   Internet Robustness Principle:

      Be liberal in what you accept, and conservative in what you send.

   However, in the case of news there is an even more important
   principle, derived from a much older code of practice, the
   Hippocratic Oath (we will thus call this the Hippocratic Principle):

      First, do no harm.

   It is VITAL to realize that decisions which might be merely
   suboptimal in a smaller context can become devastating mistakes when
   amplified by the actions of thousands of hosts within a few hours.

9.1.  Relayer General Issues

   Relayers MUST NOT alter the content of articles unnecessarily.
   Well-intentioned attempts to "improve" headers, in particular,
   typically do more harm than good.  It is necessary for a relayer to
   prepend its own name to the Path content (see section 5.6) and
   permissible for it to rewrite or delete the Xref header (see section
   6.12).  Relayers MAY delete the thoroughly-obsolete headers described
   in appendix A.3, although this behavior no longer seems useful enough
   to encourage.  Other alterations SHOULD be avoided at all costs, as
   per the Hippocratic Principle.

      NOTE:  As discussed in section 2.3, tidying up the headers of a
      user-prepared article is the job of the posting agent, not the
      relayer.  The relayer's purpose is to move already-compliant
      articles around efficiently without damaging them.  Note that
      in existing implementations, specific programs may contain both
      posting-agent functions and relayer functions.  The distinction
      is that posting-agent functions are invoked only on articles
      posted by local posters, never on articles received from other

Henry Spencer             Expires January 2010                 [Page 68]
Internet-Draft                Son of 1036                  22 July 2009

      relayers.

      NOTE:  A particular corollary of this rule is that relayers
      should not add headers unless truly necessary.  In particular,
      this is not SMTP; do not add Received headers.

   Relayers MUST NOT pass non-conforming articles on to other relayers,
   except perhaps in a cooperating subnet that has agreed to permit
   certain kinds of non-conforming behavior.  This is a direct
   consequence of the Internet Robustness Principle.

   The two preceding paragraphs may appear to be in conflict.  What is
   to be done when a non-conforming article is received?  The Robustness
   Principle argues that it should be accepted but must not be passed on
   to other relayers while still non-conforming, and the Hippocratic
   Principle strongly discourages attempts at repair.  The conclusion
   that this appears to lead to is correct:  a non-conforming article
   MAY be accepted for local filing and processing, or it MAY be
   discarded entirely, but it MUST NOT be passed on to other relayers.

   A relayer MUST NOT respond to the arrival of an article by sending
   mail to any destination, other than a local administrator, except by
   explicit prearrangement with the recipient.  Neither posting an
   article (other than certain types of control message, see section
   7.5) nor being the moderator of a moderated newsgroup constitutes
   such prearrangement.  UNDER NO CIRCUMSTANCES WHATSOEVER may a relayer
   attempt to send mail to either an article's originator or a
   moderator.

      NOTE:  Reporting apparent errors in message composition is the
      job of a posting agent, not a relayer.  The same is true of
      mailing moderated-newsgroup postings to moderators.  In
      networks of thousands of cooperating relayers, it is simply
      unacceptable for there to be any circumstance whatsoever that
      causes any significant fraction of them to simultaneously send
      mail to the same destination.  (Some control messages are
      exceptions, although perhaps ill-advised ones.)  What might, in
      a smaller network, be a useful notification or forwarding
      becomes a deluge of near-identical messages that can bring mail
      software to its knees and severely inconvenience recipients.
      Moderators, in particular, historically have suffered
      grievously from this.

   Notification of problems in incoming articles MAY go to local
   administrators, or at most (by prearrangement!)  to the
   administrators of the neighboring relayer(s) that passed on the
   problematic articles.

      NOTE:  It would be desirable to notify the author that his
      posting is not propagating as he expects.  However, there is no
      known method for doing this that will scale up gracefully.  (In
      particular, "notify only if within N relayers of the
      originator" falls down in the presence of commercial news
      services like UUNET:  there may be hundreds or thousands of

Henry Spencer             Expires January 2010                 [Page 69]
Internet-Draft                Son of 1036                  22 July 2009

      relayers within a couple of hops of the originator.)  The best
      that can be done right now is to notify neighbors, in hopes
      that the word will eventually propagate up the line, or
      organize regional monitoring at major hubs.

   If it is necessary to alter an article, e.g. translate it to another
   character set or alter its EOL representation, strenuous efforts
   should be made to ensure that such transformations are reversible,
   and that relayers or other software that might wish to reverse them
   know exactly how to do so.

      NOTE:  For example, a cooperating subnet that exchanges
      articles using a non-ASCII character set like EBCDIC should
      define a standard, reversible ASCII-EBCDIC mapping and take
      pains to see that it is used at all points where the subnet
      meets the outside.  If the only reason for using EBCDIC is that
      the readers typically employ EBCDIC devices, it would be more
      robust to employ ASCII as the interchange format and do the
      transformation in the reading and posting agents.

9.2.  Article Acceptance And Propagation

   When a relayer first receives an article, it must decide whether to
   accept it.  (This applies regardless of whether the article arrived
   by itself or as part of a batch, and in principle regardless of
   whether it originated as a local posting or as traffic from another
   relayer.)  In a cooperating subnet with well-controlled propagation
   paths, some of the tests specified here MAY be delegated to
   centrally-located relayers; that is, relayers that can receive news
   ONLY via one of the central relayers might simplify acceptance
   testing based on the assumption that incoming traffic has already
   passed the full set of tests at a central relayer.

   The wording that follows is based on a model in which articles arrive
   on a relayer's host before acceptance tests are done.  However,
   depending on the degree of integration of the transport mechanisms
   and the relayer, some or all of these tests MAY be done before the
   article is actually transmitted, so that articles which definitely
   will not be accepted need not be transmitted at all.

   The wording that follows also specifies a particular order for the
   acceptance tests.  While this order is the obvious one, the tests MAY
   be done in any order.

   First, the relayer MUST verify that the article is a legal news
   article, with all mandatory headers present with legal contents.

      NOTE:  This check in principle is done by the first relayer to
      see an article, so an article received from another relayer
      should always be legal, but there is enough old software still
      operational that this cannot be taken for granted; see the
      discussion of the Internet Robustness Principle in section 9.1.

Henry Spencer             Expires January 2010                 [Page 70]
Internet-Draft                Son of 1036                  22 July 2009

   Second, the relayer MUST determine whether it has already seen this
   article (identified by its message ID).  This is normally done by
   retaining a history of all article message IDs seen in the last N
   days, where the value of N is decided by the relayer's administrator
   but SHOULD be at least 7.  Since N cannot practically be infinite,
   articles whose Date content indicates that they are older than N days
   are declared "stale" and are deemed to have been seen already.

      NOTE:  This check is important because news propagation
      topology is typically redundant, often highly so, and it is not
      at all uncommon for a relayer to receive the same article from
      several neighbors.  The history of already-seen message IDs can
      get quite large, hence the desire to limit its length... but it
      is important that it be long enough that slowly-propagating
      articles are not classed as stale.  News propagation within the
      Internet is normally very rapid, but when UUCP links are
      involved, end-to-end delays of several days are not rare, so a
      week is not a particularly generous minimum.

      NOTE:  Despite generally more rapid propagation in recent
      times, it is still not unheard-of for some propagation paths to
      be very slow.  This can introduce the possibility of old
      articles arriving again after they are gone from the history.
      Hence the "stale" rule.

   Third, the relayer MUST determine whether any of the article's
   newsgroups are "subscribed to" by the host, i.e. fit a description of
   what hierarchies or newsgroups the site wants to receive.

      NOTE:  This check is significant because information on what
      newsgroups a relayer wishes to receive is often stored at its
      neighbors, who may not have up-to-date information or may
      simplify the rules for implementation reasons.  As a hedge
      against the possibility of missed or delayed newgroup control
      messages, relayers may wish to observe a notion of a newsgroup
      subscription that is independent of the list of newsgroups
      actually known to the relayer.  This would permit reception and
      relaying of articles in newsgroups that the relayer is not
      (yet) aware of, subject to more general criteria indicating
      that they are likely to be of interest.

   Once an article has been accepted, it may be passed on to other
   relayers.  The fundamental news propagation rule is a flooding
   algorithm:  on receiving and accepting an article, send it to all
   neighboring relayers not already in its path list that are sent its
   newsgroup(s) and distribution(s).

      NOTE:  The path list's role in loop prevention may appear
      relatively unimportant, given that looping articles would
      typically be rejected as duplicates anyway.  However, the path
      list's role in preventing superfluous transmissions is not
      trivial.  In particular, the path list is the only thing that
      prevents relayer X, on receiving an article from relayer Y,
      from sending it back to Y again.  (Indeed, the usual symptom of

Henry Spencer             Expires January 2010                 [Page 71]
Internet-Draft                Son of 1036                  22 July 2009

      confusion about relayer names is that incoming news loops back
      in this manner.)  The looping articles would be rejected as
      duplicates, but doubling the communications load on every news
      transmission path is not to be taken lightly!

   In general, relayers SHOULD NOT make propagation decisions by
   "anticipation":  relayer X, noting that the article's path list
   already contains relayer Y, decides not to send it to relayer Z
   because X anticipates that Z will get the article by a better path.
   If that is generally true, then why is there a news feed from X to Z
   at all?  In fact, the "better path" may be running slowly or may be
   down.  News propagation is very robust precisely because some
   redundant transmission is done "just in case".  If it is imperative
   to limit unnecessary traffic on a path, use of NNTP [RFC 977] or
   ihave/sendme (see section 7.2) to pass articles only when necessary
   is better than arbitrary decisions not to pass articles at all.

   Anticipation is occasionally justified in special cases.  Such cases
   should involve both (1) a cooperating subnet whose propagation paths
   are well-understood and well-monitored, with failures and slowdowns
   noticed and dealt with promptly, and (2) a persistent pattern of
   heavy unnecessary traffic on a path that is either slow or costly.
   In addition, there should be some reason why neither NNTP nor
   ihave/sendme is suitable as a solution to the problem.

9.3.  Administrator Contact

   It is desirable to have a standardized contact address for a
   relayer's administrators, in the spirit of the "postmaster" address
   for mail administrators.  Mail addressed to "newsmaster" on a
   relayer's host MUST go to the administrator(s) of that relayer.  Mail
   addressed to "usenet" on the relayer's host SHOULD be handled
   likewise.  Mail addressed to either address on other hosts using the
   same news database SHOULD be handled likewise.

      NOTE:  These addresses are case-sensitive, although it would be
      desirable for sequences equivalent to them using case-
      insensitive comparison to be handled likewise.  While
      "newsmaster" seems the preferred network-independent address,
      by analogy to "postmaster", there is an existing practice of
      using "usenet" for this purpose, and so "usenet" should be
      supported if at all possible (especially on hosts belonging to
      Usenet!).  The address `news" is also sometimes used for
      purposes like this, but less consistently.

10.  Gatewaying

   Gatewaying of traffic between news networks using this Draft and
   those using other exchange mechanisms can be useful, but must be done
   cautiously.  Gateway administrators are taking on significant
   responsibilities, and must recognize that the consequences of error
   can be quite serious.

Henry Spencer             Expires January 2010                 [Page 72]
Internet-Draft                Son of 1036                  22 July 2009

10.1.  General Gatewaying Issues

   This section will primarily address the problems of gatewaying
   traffic INTO news networks.  Little can be said about the other
   direction without some specific knowledge of the network(s) involved.
   However, the two issues are not entirely independent:  if a non-news
   network is gatewayed into a news network at more than one point,
   traffic injected into the non-news network by one gateway may appear
   at another as a candidate for injection back into the news network.

   This raises a more general principle, the single most important issue
   for gatewaying:

      Above all, prevent loops.

   The normal loop prevention of news transmission is vitally dependent
   on the Message-ID header.  Any gateway which finds it necessary to
   remove this header, alter it, or supersede it (by moving it into the
   body), MUST take equally effective precautions against looping.

      NOTE:  There are few things more effective at turning news
      readers into a lynch mob than a malfunctioning gateway, or pair
      of gateways, that takes in news articles, mangles them just
      enough to prevent news relayers from recognizing them as
      duplicates, and regurgitates them back into the news stream.
      This happens rather too often.

   Gateway implementors should realize that gateways have all the
   responsibilities of relayers, plus the added complications introduced
   by transformations between different information formats.  Much of
   section 9's discussion of relayer issues is relevant to gateways as
   well.  In particular, gateways SHOULD keep a history of recently-seen
   articles, as described in section 9.2, and not assume that articles
   will never reappear.  This is particularly important for networks
   that have their own concept analogous to message IDs:  a gateway
   should keep a history of traffic seen from BOTH directions.

   If at all possible, articles entering the non-news network SHOULD be
   marked in some way so that they will NOT be re-gatewayed back into
   news.  Multiple gateways obviously must agree on the marking method
   used; if it is done by having them know each others' names, name
   changes MUST be coordinated with great care.  If marking cannot be
   done, all transformations MUST be reversible so that a re-gatewayed
   article is identical to the original (except perhaps for a longer
   Path header).

   Gateways MUST NOT pass control messages (articles containing Control,
   Also-Control, or Supersedes headers) without removing the headers
   that make them control messages, unless there are compelling reasons
   to believe that they are relevant to both sides and that conventions
   are compatible.  If it is truly desirable to pass them unaltered,
   suitable precautions MUST be taken to ensure that there is NO
   POSSIBILITY of a looping control message.

Henry Spencer             Expires January 2010                 [Page 73]
Internet-Draft                Son of 1036                  22 July 2009

      NOTE:  The damage done by looping articles is multiplied a
      thousandfold if one of the affected articles is something like
      a sendsys message (see section 7.3) that requests multiple
      automatic replies.  Most gateways simply should not pass
      control messages at all.  If some unusual reason dictates doing
      so, gateway implementors and administrators are urged to
      consider bulletproof rate-limiting measures for the more
      destructive ones like sendsys, e.g. passing only one per hour
      no matter how many are offered.

   Gateways, like relayers, SHOULD make determined efforts to avoid
   mangling articles unnecessarily.  In the case of gateways, some
   transformations may be inevitable, but keeping them to a minimum and
   ensuring that they are reversible is still highly desirable.

   Gateways MUST avoid destroying information.  In particular, the
   restrictions of section 4.2.2 are best taken with a grain of salt in
   the context of gateways.  Information that does not translate
   directly into news headers SHOULD be retained, perhaps in "X-"
   headers, both because it may be of interest to sophisticated readers
   and because it may be crucial to tracing propagation problems.

   Gateway implementors should take particular note of the discussion of
   mailed replies, or more precisely the ban on same, in section 9.1.
   Gateway problems MUST be reported to the local administration, not to
   the innocent originator of traffic.  "Gateway problems" here includes
   all forms of propagation anomaly on the non-news side of the gateway,
   e.g. unreachable addresses on a mailing list.  Note that this
   requires consideration of possible misbehavior of "downstream" hosts,
   not just the gateway host.

10.2.  Header Synthesis

   News articles prepared by gateways MUST be legal news articles.  In
   particular, they MUST include all of the mandatory headers (see
   section 5) and MUST fully conform to the restrictions on said
   headers.  This often requires that a gateway function not only as a
   relayer, but also partly as a posting agent, aiding in the synthesis
   of a conforming article from non-conforming input.

      NOTE:  The full-conformance requirement needs particularly
      careful attention when gatewaying mailing lists to news,
      because a number of constructs that are legal in MAIL headers
      are NOT permissible in news headers.  (Note also that not all
      mail traffic fully conforms to even the MAIL specification.)
      The rest of this section will be phrased in terms of mail-to-
      news gatewaying, but most of it is more generally applicable.

   The mandatory headers generally present few problems.

   If no date information is available, the gateway should supply a Date
   header with the gateway's current date.  If only partial information
   is available (e.g. date but not time), this should be fleshed out to
   a full Date header by adding default values, not by mixing in parts

Henry Spencer             Expires January 2010                 [Page 74]
Internet-Draft                Son of 1036                  22 July 2009

   of the gateway's current date.  (Defaults should be chosen so that
   fleshed-out dates will not be in the future!)  It may be necessary to
   map timezone information to the restricted forms permitted in the
   news Date header.  See section 5.1.

      NOTE:  The prohibition of mixing dates is on the theory that it
      is better to admit ignorance than to lie.

   If the author's address as supplied in the original message is not
   suitable for inclusion in a From header, the gateway MUST transform
   it so it is, e.g. by use of the "% hack" and the domain address of
   the gateway.  The desire to preserve information is NOT an excuse for
   violating the rules.  If the transformation is drastic enough that
   there is reason to suspect loss of information, it may be desirable
   to include the original form in an X- header, but the From header's
   contents MUST be as specified in section 5.2.

   If the message contains a Message-ID header, the contents should be
   dealt with as discussed in section 10.3.  If there is no message ID
   present, it will be necessary to synthesize one, following the news
   rules (see section 5.3).

   Every effort should be made to produce a meaningful Subject header;
   see section 5.4.  Many news readers select articles to read based on
   Subject headers, and inserting a placeholder like "<no subject
   available>" is considered highly objectionable.  Even synthesizing a
   Subject header by picking out the first half-dozen nouns and
   adjectives in the article body is better than using a placeholder,
   since it offers SOME indication of what the article might contain.

   The contents of the Newsgroups header (section 5.5) are usually
   predetermined by gateway configuration, but a gateway to a network
   that has its own concept of newsgroups or discussions might have to
   make transformations.  Such transformations should be reversible;
   otherwise confusion is likely on both sides.

   It will rarely be possible for gateways to provide a Path header that
   is both an accurate history of the relayers the article has passed
   through AS NEWS and a usable reply address.  The history function
   MUST be given priority; see the discussion in section 5.6.  It will
   usually be necessary for a gateway to supply an empty path list,
   abandoning the reply function.

   It is desirable for gatewayed articles to convey as much useful
   information as possible, e.g. by use of optional news headers (see
   section 6) when the relevant information is available.  Synthesis of
   optional headers can generally follow similar rules.

   Software synthesizing References headers should note the discussion
   in section 6.5 concerning the incompatibility between MAIL and news.
   Also of interest is the possibility of incorporating information from
   In-Reply-To headers and from attribution lines in the body; an
   incomplete or somewhat conjectural References header is much better
   than none at all, and reading agents already have to cope with

Henry Spencer             Expires January 2010                 [Page 75]
Internet-Draft                Son of 1036                  22 July 2009

   incomplete or slightly erroneous References lists.

10.3.  Message ID Mapping

   This section, like the previous one, is phrased in terms of mail
   being gatewayed into news, but most of the discussion should be more
   generally applicable.

   A particularly sticky problem of gatewaying mail into news is
   supplying legal news message IDs.  Note, in particular, that not all
   MAIL message IDs are legal in news; the news syntax (specified in
   section 5.3, with related material in 5.2) is more restrictive.
   Generating a fully-conforming news article from a mail message may
   require transforming the message ID somewhat.

   Generation and transformation of message IDs assumes particular
   importance if a given mailing list (or whatever) is being handled by
   more than one gateway.  It is highly desirable that the same article
   contents not appear twice in the same newsgroup, which requires that
   they receive the same message ID from all gateways.  Gateways SHOULD
   use the following algorithm (possibly modified by the later
   discussion of gatewaying into more than one newsgroup) unless local
   considerations dictate another:

     1. Separate message ID from surroundings, if necessary.  A
        plausible method for this is to start at the first "<", end at
        the next ">", and reject the message if no ">" is found or a
        second "<" is seen before the ">".  Also reject the message if
        the message ID contains no "@" or more than one "@", or if it
        contains no ".".  Also reject the message if the message ID
        contains non-ASCII characters, ASCII control characters, or
        white space.

           NOTE:  Any legitimate domain will include at least one
           ".". [RFC 822] section 6.2.2 forbids white space in this
           context when passing mail on to non-MAIL software.

     2. Delete the leading "<" and trailing ">".  Separate message ID
        into local part and domain at the "@".

     3. In both components, transliterate leading dots (".", ASCII 46),
        trailing dots, and dots after the first in sequences of two or
        more consecutive dots, into underscores (ASCII 95).

     4. In both components, transliterate disallowed characters other
        than dots (see the definition of <unquoted-char> in section 5.2)
        to underscores (ASCII 95).

     5. Form the message ID as

           "<" local-part "@" domain ">"

      NOTE:  This algorithm is approximately that of Rich Salz's

Henry Spencer             Expires January 2010                 [Page 76]
Internet-Draft                Son of 1036                  22 July 2009

      successful gatewaying package.

   Despite the desire to keep message IDs consistent across multiple
   gateways, there is also a more subtle issue that can require a
   different approach.  If the same articles are being gatewayed into
   more than one newsgroup, and it is not possible to arrange that all
   gateways gateway them to the same cross-posted set of newsgroups,
   then the message IDs in the different newsgroups MUST be DIFFERENT.

      NOTE:  Otherwise, arrival of an article in one newsgroup will
      prevent it from appearing in another, and which newsgroup a
      particular article appears in will be an accident of which
      direction it arrives from first.  It is very difficult to
      maintain a coherent discussion when each participant sees a
      randomly-selected 50% of the traffic.  The fundamental problem
      here is that the basic assumption behind message IDs is being
      violated:  the gateways are assigning the same message ID to
      articles that differ in an important respect (Newsgroups
      header).

   In such cases, it is suggested that the newsgroup name, or an
   agreed-on abbreviation thereof, be prepended to the local part of the
   message ID (with a separating ".") by the gateway.  This will ensure
   that multiple gateways generate the same message ID, while also
   ensuring that different newsgroups can be read independently.

      NOTE:  It is preferable to have the gateway(s) cross-post the
      article, avoiding the issue altogether, but this may not be
      feasible, especially if one newsgroup is widespread and the
      other is purely local.

10.4.  Mail to and from News

   Gatewaying mail to news, and vice-versa, is the most obvious form of
   news gatewaying.  It is common to set up gateways between news and
   mail rather too casually.

   It is hard to go very wrong in gatewaying news into a mailing list,
   except for the non-trivial matter of making sure that error reports
   go to the local administration rather than to the authors of news
   articles.  (This requires attention to the "envelope address" as well
   as to the message headers.)  Doing the reverse connection correctly
   is much harder than it looks.

      NOTE:  In particular, just feeding the mail message to
      "inews -h" or the equivalent is NOT, repeat NOT, adequate to
      gateway mail to news.  Significant gatewaying software is
      necessary to do it right.  Not all headers of mail messages
      conform to even the MAIL specifications, never mind the
      stricter rules for news.

   It is useful to distinguish between two different forms of mail-to-
   news gatewaying:  gatewaying a mailing list into a newsgroup, and
   operating a "post-by-mail" service in which individual articles can

Henry Spencer             Expires January 2010                 [Page 77]
Internet-Draft                Son of 1036                  22 July 2009

   be posted to a newsgroup by mailing them to a specific address.  In
   the first case, the message is already being "broadcast", and the
   situation can be viewed as gatewaying one form of news into another.
   The second case is closer to that of a moderator posting submissions
   to a moderated newsgroup.

   In either case, the discussions in the preceding two sections are
   relevant, as is the Hippocratic Principle of section 9.  However,
   some additional considerations are specific to mail-to-news
   gatewaying.

   As mentioned in section 6, point-to-point headers like To and Cc
   SHOULD NOT appear as such in news, although it is suggested that they
   be transformed to "X-" headers, e.g. X-To and X-Cc, to preserve their
   information content for possible use by readers or troubleshooters.
   The Received header is entirely specific to MAIL and SHOULD be
   deleted completely during gatewaying, except perhaps for the Received
   header supplied by the gateway host itself.

   The Sender header is a tricky case, one where mailing-list and post-
   by-mail practice should differ.  For gatewaying mailing lists, the
   mailing-list host should be considered a relayer, and the From and
   Sender headers supplied in its transmissions left strictly untouched.
   For post-by-mail, as for a moderator posting a mailed submission, the
   Sender header should reflect the poster rather than the author.  If a
   post-by-mail gateway receives a message with its own Sender header,
   it might wish to preserve the content in an X-Sender header.

   It will generally be necessary to transform between mail's In-Reply-
   To/References convention and news's References/See-Also convention,
   to preserve correct semantics of cross references.  This also
   requires attention when going the other way, from news to mail.  See
   the discussion of the difference in section 6.5.

10.5.  Gateway Administration

   Any news system will benefit from an attentive administrator,
   preferably assisted by automated monitoring for anomalies.  This is
   particularly true of gateways.  Gateway software SHOULD be
   instrumented so that unusual occurrences, such as sudden massive
   surges in traffic, are reported promptly.  It is desirable, in fact,
   to go further:  gateway software SHOULD endeavour to limit damage in
   the event that the administrator does not respond promptly.

      NOTE:  For example, software might limit the gatewaying rate by
      queueing incoming traffic and emptying the queue at a finite
      maximum rate (well below the maximum that the host is capable
      of!) which is set by the administrator and is not raised
      automatically.

   Traffic gatewayed into a news network SHOULD include a suitable
   header, perhaps X-Gateway-Administrator, giving an electronic address
   that can be used to report problems.  This SHOULD be an address that
   goes direct to a human, not to a "routine administrative issues"

Henry Spencer             Expires January 2010                 [Page 78]
Internet-Draft                Son of 1036                  22 July 2009

   mailbox that is examined only occasionally, since the point is to be
   able to reach the administrator quickly in an emergency.  Gateway
   administrators SHOULD arrange substitutes to cover gateway operation
   (with suitable redirection of mail) when they are on vacation etc.

11.  Security And Related Issues

   Although the interchange format itself raises no significant security
   issues, the wider context does.

11.1.  Leakage

   The most obvious form of security problem with news is "leakage" of
   articles which are intended to have only restricted circulation.  The
   flooding algorithm is EXTREMELY good at finding any path by which
   articles can leave a subnet with supposedly-restrictive boundaries.
   Substantial administrative effort is required to ensure that local
   newsgroups remain local, unless connections to the outside world are
   tightly restricted.

   A related problem is that the sendme control message can be used to
   ask for any article by its message ID.  The usefulness of this has
   declined as message-ID generation algorithms have become less
   predictable, but it remains a potential problem for "secure"
   newsgroups.  Hosts with such newsgroups may wish to disable the
   sendme control message entirely.

   The sendsys, version, and whogets control messages also allow
   "outsiders" to request information from "inside", which may reveal
   details of internal topology (etc.)  that are considered
   confidential.  (Note that at least limited openness about such
   matters may be a condition of membership in such networks, e.g.
   Usenet.)

   Organizations wishing to control these forms of leakage are strongly
   advised to designate a small number of "official gateway" hosts to
   handle all news exchange with the outside world, so that a bounded
   amount of administrative effort is needed to control propagation and
   eliminate problems.  Attempts to keep news out entirely, by refusing
   to support an official gateway, typically result in large numbers of
   unofficial partial gateways appearing over time.  Such a
   configuration is much more difficult to troubleshoot.

   A somewhat-related problem is the possibility of proprietary material
   being disclosed unintentionally by a poster who does not realize how
   far his words will propagate, either from sheer misunderstanding or
   because of errors made (by human or software) in followup
   preparation.  There is little that can be done about this except
   education.

Henry Spencer             Expires January 2010                 [Page 79]
Internet-Draft                Son of 1036                  22 July 2009

11.2.  Attacks

   Although the limitations of the medium restrict what can be done to
   attack a host via news, some possibilities exist, most of them
   problems news shares with mail.

   If reading agents are careless about transmitting non-printable
   characters to output devices, malicious posters may post articles
   containing control sequences ("letterbombs") meant to have various
   destructive effects on output devices.  Possible effects depend on
   the device, but they can include hardware damage (e.g. by repeated
   writing of values into configuration memories that can tolerate only
   a limited number of write cycles) and security violation (e.g. by
   reprogramming function keys potentially used by privileged readers).

   A more sophisticated variation on the letterbomb is inclusion of
   "Trojan horses" in programs.  Obviously, readers must be cautious
   about using software found in news, but more subtly, reading agents
   must also exercise care.  MIME messages can include material that is
   executable in some sense, such as PostScript documents (which are
   programs!), and letterbombs may be introduced into such material.

   Given the presence of finite resources and other software
   limitations, some degree of system disruption can be achieved by
   posting otherwise-innocent material in great volume, either in single
   huge articles (see section 4.6) or in a stream of modest-sized
   articles.  (Some would say that the steady growth of Usenet volume
   constitutes a subtle and unintentional attack of the latter type;
   certainly it can have disruptive effects if administrators are
   inattentive.)  Systems need some ability to cope with surges, because
   single huge articles occur occasionally as the result of software
   error, innocent misunderstanding, or deliberate malice, and downtime
   at upstream hosts can cause droughts, followed by floods, of
   legitimate articles.  (There is also a certain amount of normal
   variation; for example, Usenet traffic is noticeably lighter on
   weekends and during Christmas holidays, and rises noticeably at the
   start of the school term of North American universities.)  However, a
   site that normally receives little traffic may be quite vulnerable to
   "swamping" attack if its software is insufficiently careful.

   In general, careless implementation may open doors that are not
   intrinsic to news.  In particular, implementation of control messages
   (see sections 6.6 and 7) and unbatchers (see section 8.1 and 8.2) via
   a command interpreter requires substantial precautions to ensure that
   only the intended capabilities are available.  Care must also be
   taken that article-supplied text is not fed to programs that have
   escapes to command interpreters.

   Finally, there is considerable potential for malice in the sendsys,
   version, and whogets control messages.  They are not harmful to the
   hosts receiving them as news, but they can be used to enlist those
   hosts (by the thousands) as unwitting allies in a mail-swamping
   attack on a victim who may not even receive news.  The precautions
   discussed in section 7.5 can reduce the potential for such attacks

Henry Spencer             Expires January 2010                 [Page 80]
Internet-Draft                Son of 1036                  22 July 2009

   considerably, but the hazard cannot be eliminated as long as these
   control messages exist.

11.3.  Anarchy

   The highly distributed nature of news propagation, and the lack of
   adequate authentication protocols (especially for use over the less-
   interactive transport mechanisms such as UUCP), make article forgery
   relatively straightforward.  It may be possible to at least track a
   forgery to its source, once it is recognized as such, but clever
   forgers can make even that relatively difficult.  The assumption that
   forgeries will be recognized as such is also not to be taken for
   granted; readers are notoriously prone to blindly assuming
   authenticity.  If a forged article's initial path list includes the
   relayer name of the supposed poster's host, the article will never be
   sent to that host, and the alleged author may learn about the forgery
   secondhand or not at all.

   A particularly noxious form of forgery is the forged "cancel" control
   message.  Notably, it is relatively straightforward to write software
   that will automatically send out a (forged) cancel message for any
   article meeting some criterion, e.g. written by a specific author.
   The authentication problems discussed in section 7.1 make it
   difficult to solve this without crippling cancel's important
   functionality.

   A related problem is the possibility of disagreements over newsgroup
   creation, on networks where such things are not decided by central
   authorities.  There have been cases of "rmgroup wars", where one
   poster persistently sends out newgroup messages to create a newsgroup
   and another, equally persistently, sends out rmgroup messages asking
   that it be removed.  This is not particularly damaging, if relayers
   are configured to be cautious, but can cause serious confusion among
   innocent third parties who just want to know whether they can use the
   newsgroup for communication or not.

11.4.  Liability

   News shares the legal uncertainty surrounding other forms of
   electronic communication:  what rules apply to this new medium of
   information exchange?  News is a particularly problematic case
   because it is a broadcast medium rather than a point-to-point one
   like mail, and analogies to older forms of communication are
   particularly weak.

   Are news-carrying hosts common carriers, like the phone companies,
   providing communications paths without having either authority over
   or responsibility for content?  Or are they publishers, responsible
   for the content regardless of whether they are aware of it or not?
   Or something in between?  Such questions are particularly significant
   when the content is technically criminal, e.g. some types of
   sexually-oriented material in some jurisdictions, in which case
   ignorance of its presence may not be an adequate defence.

Henry Spencer             Expires January 2010                 [Page 81]
Internet-Draft                Son of 1036                  22 July 2009

   Even in milder situations such as libel or copyright violation, the
   responsibilities of the poster, his host, and other hosts carrying
   the traffic are unclear.  Note, in particular, the problems arising
   when the article is a forgery, or when the alleged author claims it
   is a forgery but cannot prove this.

12.  References

   [ISO/IEC 9899]
              "Information technology - Programming Language C", ISO/IEC
              9899:1990 {more recently 9899:1999}, 1990.

   [Metamail] N. Borenstein,
              <http://ftp.funet.fi/pub/unix/mail/metamail/ANNOUNCE>,
              February 1994.

   [RFC 821]  Jonathan B. Postel, "Simple Mail Transfer Protocol", RFC
              821, August 1982.

   [RFC 822]  D. Crocker, "Standard for the Format of ARPA Internet Text
              Messages.", STD 11, RFC 822, August 1982.

   [RFC 850]  Mark R. Horton, "Standard for interchange of Usenet
              messages", RFC 850, June 1983.

   [RFC 977]  Brian Kantor and Phil Lapsley, "Network news transfer
              protocol - a proposed standard for the stream-based
              transmission of news", RFC 977, February 1986.

   [RFC1036]  M. Horton and R. Adams, "Standard for Interchange of
              USENET Messages", RFC 1036, December 1987.

   [RFC1123]  R. Braden, "Requirements for Internet Hosts -- Application
              and Support", RFC 1123, October 1989.

   [RFC1341]  N. Borenstein and N. Freed, "MIME (Multipurpose Internet
              Mail Extensions): Mechanisms for Specifying and Describing
              the Format of Internet Message Bodies", RFC 1341, June
              1992.

   [RFC1342]  K Moore, "Representation of Non-Ascii Text in Internet
              Message Headers", RFC 1342, June 1992.

   [RFC1345]  K. Simonsen, "Character Mnemonics & Character Sets", RFC
              1345, June 1992.

   [RFC1413]  M. St. Johns, "Identification Protocol", RFC 1413,
              February 1993.

   [RFC1456]  Vietnamese Standardization Working Group, "Conventions for
              Encoding the Vietnamese Language", RFC 1456, May 1993.

Henry Spencer             Expires January 2010                 [Page 82]
Internet-Draft                Son of 1036                  22 July 2009

   [RFC1544]  M. Rose, "The Content-MD5 Header Field", RFC 1544,
              November 1993.

   [RFC1896]  P. Resnick and A. Walker, "The text/enriched MIME
              Content-type", RFC 1896, February 1996.

   [RFC2045]  N. Freed and N. Borenstein, "Multipurpose Internet Mail
              Extensions (MIME) Part One: Format of Internet Message
              Bodies", RFC 2045, November 1996.

   [RFC2046]  N. Freed and N. Borenstein, "Multipurpose Internet Mail
              Extensions (MIME) Part Two: Media Types", RFC 2046,
              November 1996.

   [RFC2047]  K. Moore, "MIME (Multipurpose Internet Mail Extensions)
              Part Three: Message Header Extensions for Non-ASCII Text",
              RFC 2047, November 1996.

   [RFC2049]  N. Freed and N. Borenstein, "Multipurpose Internet Mail
              Extensions (MIME) Part Five: Conformance Criteria and
              Examples", RFC 2049, November 1996.

   [RFC2822]  P. Resnick, "Internet Message Format", RFC 2822, April
              2001.

   [RFC3977]  C. Feather, "Network News Transport Protocol (NNTP)", RFC
              3977.

   [RFC5322]  P. Resnick, "Internet Message Format", RFC 5322, October
              2008.

   [RFC5536]  K. Murchison, C. H. Lindsey, and D. Kohn, "News Article
              Format", RFC 5536, May 2009.

   [RFC5537]  R. Allbery and C. H. Lindsey, "News Article Architecture
              and Protocols", RFC 5537, May 2009.

   [Sanderson]
              David Sanderson, Smileys, O'Reilly & Associates Ltd.,
              1993.

   [UUCP]     Tim O'Reilly and Grace Todino, Managing UUCP and Usenet,
              O'Reilly & Associates Ltd., January 1992.

   [X3.4]     "American National Standard for Information Systems -
              Coded Character Sets - 7-Bit American National Standard
              Code for Information Interchange (7-Bit ASCII)", ANSI
              X3.4, 1986.

Henry Spencer             Expires January 2010                 [Page 83]
Internet-Draft                Son of 1036                  22 July 2009

A. Archaeological Notes

A.1. A-News Article Format

   The obsolete "A News" article format consisted of exactly five lines
   of header information, followed by the body.  For example:

      Aeagle.642
      news.misc
      cbosgd!mhuxj!mhuxt!eagle!jerry
      Fri Nov 19 16:14:55 1982
      Usenet Etiquette - Please Read
      body
      body
      body

   The first line consisted of an "A" followed by an article ID
   (analogous to a message ID and used for similar purposes).  The
   second line was the list of newsgroups.  The third line was the path.
   The fourth was the date, in the format above (all fields fixed
   width), resembling an Internet date but not quite the same.  The
   fifth was the subject.

   This format is documented for archaeological purposes only.  Do not
   generate articles in this format.

A.2. Early B-News Article Format

   The obsolete pseudo-Internet article format, used briefly during the
   transition between the A News format and the modern format, followed
   the general outline of a MAIL message but with some non-standard
   headers.  For example:

      From: cbosgd!mhuxj!mhuxt!eagle!jerry (Jerry Schwarz)
      Newsgroups: news.misc
      Title: Usenet Etiquette -- Please Read
      Article-I.D.: eagle.642
      Posted: Fri Nov 19 16:14:55 1982
      Received: Fri Nov 19 16:59:30 1982
      Expires: Mon Jan 1 00:00:00 1990

      body
      body
      body

   The From header contained the information now found in the Path
   header, plus possibly the full name now typically found in the From
   header.  The Title header contained what is now the Subject content.
   The Posted header contained what is now the Date content.  The
   Article-I.D. header contained an article ID, analogous to a message
   ID and used for similar purposes.  The Newsgroups and Expires headers
   were approximately as now.  The Received header contained the date

Henry Spencer             Expires January 2010                 [Page 84]
Internet-Draft                Son of 1036                  22 July 2009

   when the latest relayer to process the article first saw it.  All
   dates were in the above format, with all fields fixed width,
   resembling an Internet date but not quite the same.

   This format is documented for archaeological purposes only.  Do not
   generate articles in this format.

A.3. Obsolete Headers

   Early versions of news software following the modern format sometimes
   generated headers like the following:

      Relay-Version: version B 2.10 2/13/83; site cbosgd.UUCP
      Posting-Version: version B 2.10 2/13/83; site eagle.UUCP
      Date-Received: Friday, 19-Nov-82 16:59:30 EST

   Relay-Version contained version information about the relayer that
   last processed the article.  Posting-Version contained version
   information about the posting agent that posted the article.  Date-
   Received contained the date when the last relayer to process the
   article first saw it (in a slightly nonstandard format).

   These headers are documented for archaeological purposes only.  Do
   not generate articles using them.

A.4. Obsolete Control Messages

   There once was a senduuname control message, resembling sendsys but
   requesting transmission of the list of hosts that the receiving host
   had UUCP connections to.  This rapidly ceased to be of much use, and
   many organizations consider information about their internal
   connectivity to be confidential.

   Historically, a checkgroups body consisting of one or two lines, the
   first of the form "-n newsgroup", caused checkgroups to apply to only
   that single newsgroup.  This form is documented for archaeological
   purposes only; do not use it.

   Historically, an article posted to a newsgroup whose name had exactly
   three components of which the third was "ctl" signified that article
   was to be taken as a control message.  The Subject header specified
   the actions, in the same way the Control header does now.  This form
   is documented for archaeological purposes only; do not use it; do not
   implement it.

B. A Quick Tour Of MIME

   (The editor wishes to thank Luc Rooijakkers; most of this appendix is
   a lightly-edited version of a summary he kindly supplied.)

Henry Spencer             Expires January 2010                 [Page 85]
Internet-Draft                Son of 1036                  22 July 2009

   MIME (Multipurpose Internet Mail Extensions) is an upward-compatible
   set of extensions to [RFC 822], currently documented
   in [RFC2045], [RFC2046] and [RFC2047].  This appendix summarizes
   these documents.  See the MIME RFCs for more information; they are
   very readable.

      UNRESOLVED ISSUE:  These RFC numbers (here and elsewhere in
      this Draft) need updating when the new MIME RFCs come out {now
      resolved!}.

   MIME defines the following new headers:

      MIME-Version
      Content-Type
      Content-Transfer-Encoding
      Content-ID
      Content-Description

   The MIME-Version header is mandatory for all messages conforming to
   the MIME specification and carries the version number of the MIME
   specification.  Example:

      MIME-Version: 1.0

   The Content-Type header indicates the content type of the message.
   Content types are split into a top-level type and a subtype,
   separated by a slash.  Auxiliary information can also be supplied,
   using an attribute-value notation.  Example:

      Content-Type: text/plain; charset=us-ascii

   (In the absence of a Content-Type header this is in fact the default
   content type.)

   Important type/subtype combinations are

   text/plain             Plain text, possibly in a non-ASCII character
                          set.

   text/enriched          A very simple wordprocessor-like language
                          supporting character attributes (e.g.,
                          underlining), justification control, and
                          multiple character sets.  (This proposal has
                          gone through several iterations and has
                          recently split off from the main MIME RFCs
                          into a separate document [RFC1896].)

   message/rfc822         A mail message conforming to a slightly-
                          relaxed version of [RFC 822].

Henry Spencer             Expires January 2010                 [Page 86]
Internet-Draft                Son of 1036                  22 July 2009

   message/partial        Part of a message (supporting the transparent
                          splitting and joining of messages when they
                          are too large to be handled by some transport
                          agent).

   message/external-body  A message whose body is external.  Possible
                          access methods include via mail, FTP, local
                          file, etc.

   multipart/mixed        A message whose body consists of multiple
                          parts, possibly of different types, intended
                          to be viewed in serial order.  Each part looks
                          like an [RFC 822] message, consisting of
                          headers and a body.  Most of the [RFC 822]
                          headers have no defined semantics for body
                          parts.

   multipart/parallel     Likewise, except that the parts are intended
                          to be viewed in parallel (on user agents that
                          support it).

   multipart/alternative  Likewise, except that the parts are intended
                          to be semantically equivalent such that the
                          part that best matches the capabilities of the
                          environment should be displayed.  For example,
                          a message may include plain-text, enriched-
                          text, and postscript versions of some
                          document.

   multipart/digest       A variant of multipart/mixed especially
                          intended for message digests (the default type
                          of the parts is message/rfc822 instead of
                          text/plain, saving on the number of headers
                          for the parts).

   application/postscript A PostScript document.  (PostScript is a
                          trademark of Adobe.)

   Other top-level types exist for still images, audio, and video
   samples.

   Some of the above types require the ability to transport binary data.
   Since the existing message systems usually do not support this, MIME
   provides a Content-Transfer-Encoding header to indicate the kind of
   encoding used.  The possible encodings are:

   7bit              No encoding; the data consists of short (less than
                     1000 characters) lines of 7-bit ASCII data,
                     delimited by EOL sequences.  This is the default
                     encoding.

   8bit              Like 7bit, except that bytes with the high-order
                     bit set may be present.  Many transmission paths
                     are incapable of carrying messages which use this

Henry Spencer             Expires January 2010                 [Page 87]
Internet-Draft                Son of 1036                  22 July 2009

                     encoding.

   binary            No encoding; any sequence of bytes may be present.
                     Many transmission paths are incapable of carrying
                     messages which use this encoding.

   base64            The data is encoded by representing every group of
                     3 bytes as 4 characters from the alphabet "A-Za-
                     z0-9+/", which was chosen for its high robustness
                     through mail gateways (the alphabet used by
                     uuencode does not survive ASCII-EBCDIC-ASCII
                     translations).  In the final group of 4 characters,
                     "=" is used for those characters not representing
                     data bytes.  Line length is limited and EOLs in the
                     encoded form are ignored.

   quoted-printable  Any byte can be represented by a three character
                     "=XX" sequence where the X's are upper case
                     hexadecimal digits.  Bytes representing printable
                     7-bit US-ASCII characters except "=" may be
                     represented literally.  Tabs and blanks may be
                     represented literally if not at the end of a line.
                     Line length is limited, and an EOL preceded by "="
                     was inserted for this purpose and is not present in
                     the original.

   The base64 and quoted-printable encodings are applied to data in
   Internet canonical form, which means that any EOL encoded as anything
   but EOL must be an Internet canonical EOL:  CR followed by LF.

   The Content-Description header allows further description of a body
   part, analogous to the use of Subject for messages.

   Finally, the Content-ID header can be used to assign an
   identification to body parts, analogous to the assignment of
   identifications to messages by Message-ID.

   Note that most of these headers are structured header fields, as
   defined in [RFC 822].  Consequently, comments are allowed in their
   values.  The following is a legal MIME header:

      Content-Type: (a comment) text (yeah)   /
              plain    (and now some params:) ; charset= (guess what)
         iso-8859-1 (we don't have iso-10646 yet, pity)

      NOTE:  Although the MIME specification was developed for mail,
      there is nothing precluding its use for news as well.  While it
      might simplify implementation to restrict the MIME headers
      somewhat, in the same way that other news headers (e.g. From)
      are restricted subsets of the [RFC 822] originals, this would
      add yet another divergence between two formats that ought to be
      as compatible as possible.  In the case of the MIME headers,
      there is no body of existing code posing compatibility

Henry Spencer             Expires January 2010                 [Page 88]
Internet-Draft                Son of 1036                  22 July 2009

      concerns.  A full-featured MIME reading agent needs a full [RFC
      822] parser anyway, to properly handle body parts of types like
      message/rfc822, so there is little gain from restricting MIME
      headers.  Adopting the MIME specification unchanged seems best.
      However, article-level MIME headers must still comply with the
      overall news header syntax given in section 4, so that news
      software which is NOT interested in MIME need not contain a
      full [RFC 822] parser.

   The second part of MIME,  [RFC2047] (Message Header Extensions for
   Non-ASCII Text), addresses the problem of non-ASCII characters in
   headers.  An example of a header using the [RFC2047] mechanism is

      From: =?ISO-8859-1?Q?Andr=E9_?= Pirard <PIRARD@vm1.ulg.ac.be>

   Such encodings are allowed in selected headers, subject to the
   restrictions listed in [RFC2047].

   The MIME effort has also produced an RFC defining a Content-MD5
   header [RFC1544] containing an MD5-based "checksum" of the contents
   of an article or body part, giving high confidence of detecting
   accidental modifications to the contents.

   The "metamail" software package [Metamail] helps provide MIME support
   with minimal changes to mailers, and may also be relevant to news
   reading agents.

   The PEM (Privacy Enhanced Mail) effort is pursuing analogous
   facilities to offer stronger guarantees against malicious
   modifications, unauthorized eavesdropping, and forgery.  This work
   too may be applicable to news, once it is reconciled with MIME (by
   efforts now underway).

C. Summary of Changes Since RFC 1036

   This Draft is much longer than [RFC1036], so there is obviously much
   change in content.  Much of this is just increased precision and
   rigor.  Noteworthy changes and additions include:

     + section 4.3's restrictions on article bodies

     + all references to MIME facilities

     + size limits on articles

     + precise specification of Date-content syntax

     + message IDs must never be re-used, ever

     + "!" is the only Path delimiter

Henry Spencer             Expires January 2010                 [Page 89]
Internet-Draft                Son of 1036                  22 July 2009

     + multiple moderators in the Approved header

     + rules on References trimming, and the _-_ mechanism

     + generalization of the Xref rules

     + multiple message IDs in Cancel and Supersedes

     + Also-Control

     + See-Also

     + Article-Names

     + Article-Updates

     + more precise rules for cancellation

     + cancellation authorization based on From, not Sender

     + "unmoderated" and descriptors in newgroup messages

     + restrictive rules on handling of sendsys and version messages

     + the whogets control message

     + precise specification of checkgroups messages

     + compression type preferably specified out-of-band

     + rules for encapsulating news in MIME mail

     + tighter specification of relayer functioning (section 9.1)

     + the "newsmaster" contact address

     + rules for gatewaying (section 10)

     + discussion of security issues (section 11)

D. Summary of Completely New Features

   Most of this Draft merely documents existing practice, preferred
   versions thereof, or straightforward generalizations of it, but there
   are a few outright inventions.  These are:

     + the _-_ mechanism for References trimming

     + Also-Control

     + See-Also

Henry Spencer             Expires January 2010                 [Page 90]
Internet-Draft                Son of 1036                  22 July 2009

     + Article-Names

     + Article-Updates

     + the whogets control message

E. Summary of Differences From RFC 822+1123

   The following are noteworthy differences between this Draft's
   articles and MAIL messages:

     + generally less-permissive header syntax

     + notably, limited From syntax

     + MAIL header comments allowed in only a few contexts

     + slightly more restricted message-ID syntax

     + several more mandatory headers

     + duplicate headers forbidden

     + References/See-Also versus In-Reply-To/References (section 6.5)

     + case sensitivity in some contexts

     + point-to-point headers, e.g. To and Cc, forbidden (section 6)

     + several new headers

Author's Address

      Henry Spencer
      henry@zoo.utoronto.ca

      SP Systems
      Box 280 Stn. A
      Toronto, Ont. M5W1B2  Canada

Henry Spencer             Expires January 2010                 [Page 91]