Internet Architecture Board P. Faltstrom
Internet-Draft IAB
Expires: May 27, 2004 November 27, 2003
Synchronization of Stringprep with Unicode Normalization rules
draft-faltstrom-unicode-synchronisation-00.txt
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at http://
www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on May 27, 2004.
Copyright Notice
Copyright (C) The Internet Society (2003). All Rights Reserved.
Abstract
This memo provides information about potential problems for
applications that use the Unicode Character set in IETF standards. It
especially examines differences between normalization rules in
different versions of the Unicode character set.
1. The problem
The Unicode Standard Annex #15 (Unicode Normalization Forms) [3]
specify how the normalization rules are to be applied to strings. In
Annex 12 (Corrigenda) differences between normalization rules between
versions of Unicode are discussed.
The IETF uses these Normalization rules in various standards,
Faltstrom Expires May 27, 2004 [Page 1]
Internet-Draft Normalization Rules November 2003
especially the ones creating profiles of stringprep [RFC3454] [2].
The Unicode Consortium has well-defined policies in place to govern
changes that affect backwards compatibility. Once a character is
encoded, its canonical combining class and decomposition mapping will
not be changed in a way that will destabilize normalization.
What this means is: If a string contains only characters from a given
version of the Unicode Standard (e.g., Unicode 3.1.1), and it is put
into a normalized form in accordance with that version of Unicode,
then it will be in normalized form according to any past or future
versions of Unicode.
This guarantee has been in place for Unicode 3.1 and after. It has
been necessary to correct the decompositions of a small number of
characters since Unicode 3.1, as listed in the Normalization
Corrections data file, but such corrections are in accordance with
the above principles: all text normalized on old systems will test as
normalized in future systems. All text normalized in future systems
will test as normalized on past systems. What may change, for those
few characters, is that unnormalized text may normalize differently
on past and future systems.
2. Scenario
Assume a client receives a non-normalized string, and then applies
normalization according to normalization rules in a particular
version of Unicode. If the client passes the normalized string to a
server that also has normalized a non-normalized copy of the string,
but has used a different version of the Unicode normalization rules,
the two strings might not match.
Example: In version 3.1 of Unicode, codepoint U+2F874 is normalized
to U+5F33. In version 3.2 U+2F874 is normalized to U+5F53. Say we
have on the Internet nodes A and B. Assume that A is using version
3.1 of Unicode, and B is using version 3.2. U+2F874 is passed to both
A and B. After normalization they will store the strings U+5F33 and
U+5F53 respectively. The end result is that even if the same
codepoint, U+2F874, is passed to both nodes, they will after
normalization have different strings (U+5F33 and U+5F53). If A sends
a message with normalized version of U+2F874 (U+5F33) to B as a
search string, there will be no match at B because B has normalized
the data (U+2F874) to U+5F53.
For the problem to exist, the string (only consisting of the
codepoint U+2F874 in the example above) needs only include at least
one of the codepoints in the correction list (see appendix A). As of
version 4.0.0 of Unicode, the list of corrections (since Unicode 3.1)
Faltstrom Expires May 27, 2004 [Page 2]
Internet-Draft Normalization Rules November 2003
consists of exactly 5 codepoints. Over time, as additional errors in
the normalization rules are found, this list will grow. The list is
controlled by the Unicode Consortium; IETF has little or no specific
input into it.
3. Recommendation
Applications that implement stringprep or one of its profiles must be
aware of the existence of the corrections table [4]. Version 4.0.0 of
this correction list can be found in Appendix A. If a string that is
to be used for matching includes any of these codepoints, unexpected
results (non-matching when matching should occur) may occur. Because
of this, it is recommended that in sensitive applications /
deployments, special care should be taken.
Examples of problems include (but are not limited to) problems in
protocols which use stringprep and pass a normalized version of
strings received from a human. Such protocols include the DNS [5]
(dispute resolution at the time of domain name registration) and
protocols using domain names (HTTP [6], SMTP [7] etc), LDAP [8]
(characters in the domain name labels as well as searches on
attribute values), Kerberos [9]Kerberos, SASL [10] (authentication
mechanism), iSCSI [11] (names of volumes).
As codepoints can be added to the list at any time, addition of
codepoints can affect already normalized strings. Say a registry
accepts registrations of domain names. If a domain name U+2F868 is to
be registered, according to nameprep profile in Unicode 3.2 the
string U+2136A is to be registered. If later the registry switches to
use version 4.0 of Unicode, the question is whether the registered
string U+2136A is to stay, or whether it should be changed to U+36FC.
It might even be the case that U+36FC is already registered, and by a
different domain name holder. The change in normalization rules in
this case create a potential dispute resolution.
3.1 Message to the Unicode Consortium
The IETF strongly encourages the Unicode Consortium to keep the size
and rate of change of the correction list to an absolute minimum, as
it will be impossible for implementations (applications) to know what
version of the normalization tables which are in use. This is
because, in practice, the tables in many cases will be part of the
operating system. The end user will expect the same normalization
rules to be used in all applications in her environment.
3.2 Alternatives for the IETF
When the Stringprep [2] specification is updated in the IETF, there
Faltstrom Expires May 27, 2004 [Page 3]
Internet-Draft Normalization Rules November 2003
will be three possible paths forward and a choice must be made:
1. Stay with use of Unicode 3.2
2. Change to a later version of Unicode than 3.2, but without the
changes listed in the correction list at that time
3. Change to a later version of Unicode than 3.2, and accept
incompatible changes in the normalization tables
4. Security Considerations
This memo discusses the impact that corrections to the Unicode
normalization rules will have on protocols in the IETF that uses
those rules. Inconsistencies among versions of the rules will create
non-backward compatibility problems. Even if protocols and
implementations are created correctly, this will lead to strings that
should match in a search or other operation being reported as not
matching.
These false negatives for strings that include the codepoints in the
Unicode Correction Table might lead, for example, to the following
problems:
o Domain names lookups that should succeed fail instead
o Collisions between registered domain names occur (i.e., two
different names appear to match, even when there were no
collisions at registration time
o Searches in LDAP databases fail
o Searching for iSCSI devices fail
o Authentication to Kerberos realms (logging in to systems using
Kerberos) fail
Normative References
[1] The Unicode Consortium, "The Unicode Standard", ISBN
0-321-18578-1 The Unicode Standard 4.0, April 2003.
[2] Hoffman, P. and M. Blanchet, "Preparation of Internationalized
Strings ("stringprep")", RFC 3454, December 2002.
[3] Davis, M. and M. Durst, "Unicode Normalization Forms", Unicode
Technical Report 15, April 2003.
[4] The Unicode Consortium, "Normalization Corrections", http://
www.unicode.org/Public/UNIDATA/NormalizationCorrections.txt
Version 4.0.0, April 2003.
Informative References
[5] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep Profile
Faltstrom Expires May 27, 2004 [Page 4]
Internet-Draft Normalization Rules November 2003
for Internationalized Domain Names (IDN)", RFC 3491, March
2003.
[6] Fielding, R., Gettys, J., Mogul, J., Nielsen, H., Masinter, L.,
Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol --
HTTP/1.1", RFC 2616, June 1999.
[7] Klensin, J., "Simple Mail Transfer Protocol", RFC 2821, April
2001.
[8] Zeilenga, K., "LDAP: Internationalized String Preparation",
draft-zeilenga-ldapbis-strprep-00.txt (work in progress), May
2003.
[9] Altman, J., "Preparation of Internationalized Strings Profile
for Kerberos UTF-8 Strings",
draft-ietf-krb-wg-utf8-profile-01.txt (work in progress),
February 2003.
[10] Zeilenga, K., "SASLprep: Stringprep profile for user names and
passwords", draft-ietf-sasl-saslprep-03.txt (work in progress),
June 2003.
[11] Bakke, M., "String Profile for iSCSI Names",
draft-ietf-ips-iscsi-string-prep-04.txt (work in progress),
March 2003.
Author's Address
Patrik Faltstroms
Internet Architecture Board
EMail: paf@cisco.com
Appendix A. Appendix A
# NormalizationCorrections-4.0.0.txt
#
# This file is a normative contributory data file in the
# Unicode Character Database.
#
# The normalization stabilization policy of the Unicode
# Consortium ordinarily precludes any change to the decomposition
# for any character, once established in a relevant version
# of the UnicodeData.txt data file. However, under certain
# exceptional (and rare) conditions, an error in a decomposition
Faltstrom Expires May 27, 2004 [Page 5]
Internet-Draft Normalization Rules November 2003
# mapping may be discovered that is truly just an unintended
# typo in the data, and not a matter of dubious interpretation.
#
# Whenever such an error may be found, and if it meets the
# requirements for possible exceptions to normalization
# stability, the correction is entered in this data file,
# so that any implementation depending on absolute stability
# of normalization, *including* any errors in the data, can
# safely reconstruct the exact state of the data tables at
# any given version of Unicode.
#
# Currently this list has exactly six entries in it, one for the
# typo found and corrected in Corrigendum #3, and five for
# the typos and misidentifications found and corrected in
# Corrigendum #4. All efforts
# will be made to keep the entries limited to just those fixes.
#
# Interpretation of the fields:
# Field 1: Unicode code point
# Field 2: Original (erroneous) decomposition
# Field 3: Corrected decomposition
# Field 4: Version of Unicode for which the correction was
# entered into UnicodeData.txt, in n.n.n format.
# Comment: Indicates the Unicode Corrigendum which documents
# the correction
#
#
F951;96FB;964B;3.2.0 # Corrigendum 3
2F868;2136A;36FC;4.0.0 # Corrigendum 4
2F874;5F33;5F53;4.0.0 # Corrigendum 4
2F91F;43AB;243AB;4.0.0 # Corrigendum 4
2F95F;7AAE;7AEE;4.0.0 # Corrigendum 4
2F9BF;4D57;45D7;4.0.0 # Corrigendum 4
Faltstrom Expires May 27, 2004 [Page 6]
Internet-Draft Normalization Rules November 2003
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
intellectual property or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; neither does it represent that it
has made any effort to identify any such rights. Information on the
IETF's procedures with respect to rights in standards-track and
standards-related documentation can be found in BCP-11. Copies of
claims of rights made available for publication and any assurances of
licenses to be made available, or the result of an attempt made to
obtain a general license or permission for the use of such
proprietary rights by implementors or users of this specification can
be obtained from the IETF Secretariat.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights which may cover technology that may be required to practice
this standard. Please address the information to the IETF Executive
Director.
Full Copyright Statement
Copyright (C) The Internet Society (2003). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assignees.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
Faltstrom Expires May 27, 2004 [Page 7]
Internet-Draft Normalization Rules November 2003
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
Faltstrom Expires May 27, 2004 [Page 8]
</x-flowed>