i;unicode-casemap - Simple Unicode Collation Algorithm
draft-crispin-collation-unicasemap-07

[Ballot comment]
Based on Gen-ART Review from Christian Vogt.

  Section 1 ends with an applicability statement for the algorithm.
  The section says that, while the algorithm is well-suited for
  technical languages, it does not work correctly in certain cases
  when applied to natural language.  My suggestion is to move the
  applicability statement to a more prominent place, perhaps into a
  new section preceding current section 1.

  3rd paragraph of section 1:  s/using using/using/

[Ballot comment]
In response to Sam's discuss, the normalization form described is
Normalization form KD excluding the "Canonical Ordering Behavior" of
non-spacing marks.

Unicode TR 15 (http://www.unicode.org/reports/tr15/) defines form KD
as "compatibility decomposition" and the version of the Unicode
specification I have handy (v 2.0) defines compatibility decomposition
as recursively applying both compatibility and canonical mappings and
then re-ordering non-spacing marks (I assume this hasn't changed).  As
the UniData file contains both compatibility and canonical decompositions
in the "decomposition property" (the Unicode Character Database overview
document calls this the Decomposition_Mapping field) the core of this
algorithm is the same as NFKD.

If this variant of NFKD was a form visible on the wire, I would
consider that problematic, but as it is a form used internal to the
algorithm only, I do not consider it problematic as long as it is
intentional.  The one place where this difference from normalization
form KD might have surprising results is if one of the input strings
contains a non-canonical decomposition of a letter with multiple diacritical marks.  In that case the character would not match under
the equality function.  As the intention of this collation is a 'cheap'
collation rather than a linguistically correct collation (just as i;ascii-casemap is not a proper English collation), I consider this simplification justifiable.  The "canonical ordering behavior" section
of the Unicode specification is non-trivial and provides little
incremental benefit (indeed little visible change) to this collation
for that additional complexity.

The specification could be made more helpful to implementors with a
Unicode library by giving advice about whether or not it is acceptable
to substitute NFKD for the partial-NFKD described here.

On the issue of tracking the Unicode standard, I'll mention that many
runtime environments have built-in Unicode support now and it can be
difficult to determine which version of Unicode is active and most such
environments provide no way simple way to bind to a previous version of
UniData.txt.  So I consider tracking the current version of Unicode to
be pragmatic and necessary to make implementations feasible.  I would
look to the "running code" principle of the IETF to support this
position.

[Ballot discuss]
I don't expect to hold this discuss significantly past the telechat
and would not be surprised if no change is required.

I want to ask how much review has been done for two issues:

1) How well does the decomposition normalization in this spec align
with NFKD.  The text says it is effectively the same, but where does
it produce different results?  Do we care?

2) Are we comfortable with the unstable reference to unicode data?  Do we at least need to discuss the security considerations of these changes?

APPs-REVIEW

There are a couple of clarifications I would suggest:

Section 1, Para 1:

"All input is valid." - it is not clear that this refers to the validity test in 4790 as opposed to input to the tests described in the previous sentence. Suggested alternative: "The validity test operation always returns a valid result."

Even with that change, later the spec states that "strings in other character sets and/or encodings can not be used with this collation" so wouldn't those return an invalid response if used in the validity test? Same for invalid UTF-8 sequences?

Section 1, Para 5:
I would like to see a informative reference pointing to the current UnicodeData.txt file rather than just having the generic [UNICODE] reference.

Section 5, Para 6:
"The resulting two titlecased canonicalized UTF-8 strings are then treated as in i;octet for equality and ordering."

shouldn't that also mention substring? Suggested alternative:

"The resulting two titlecased canonicalized UTF-8 strings are then treated as in i;octet for equality, substring and ordering operations."

Other than that looks good. This should proceed to publication asap as it is needed by a lot of apps.

--Cyrus Daboo

IANA Last Call Comments:

Upon approval of this document, the IANA will make the
following assignments in the "Collation Registry" registry
located at

http://www.iana.org/assignments/collation/collation-index.html

Coallition: See Section 2 of [RFC-crispin-collation-unicasemap-02]

Description:
The i;unicode-casemap collation is well suited to to
use with many Internet protocols and computer languages.
Use with natural language is often inappropriate; even
though the collation apparently supports languages such
as Swahili and English, in real-world use it tends to
mis-sort a number of types of string

Reference: [RFC-crispin-collation-unicasemap-02]


We understand the above to be the only IANA Action for
this document.

Issues raised to deal with in next version or RFC Ed notes:
- conversion to titlecased canonicalized UTF8 must be applied recursively
- deal with apparent conflict between paragraphs on using other Unicode mappings

PROTO Writeup

  (1.a)  Alexey Melnikov  is the document
shepherd for this document.  The document is ready for publication.

  (1.b)  This document was reviewed by several active and experienced
IMAPEXT and Sieve WG members.  There are no concerns about the depth of
the reviews.

Also note that this document is a dependency for the IMAP I18N document
and an indirect dependency for Lemonade Profile Bis document
(draft-ietf-lemonade-profile-bis-XX.txt).

  (1.c)  No concerns requiring additional review.  In particular,
this document was reviewed by Arnt Gulbrandsen, who is one of the editors
of the RFC 4790.

  (1.d) No specific concerns. No IPR disclosure was filed for this
document.

  (1.e)  This document is an individual submission.

  (1.f)  No appeals threatened.

  (1.g)  IDnits 2.04.07 was used to verify the document. Excessively
long lines were found, but this is purely editorial.  It also reports
2 Missing Reference, but they are defined. I think this is a bug in
IDnits.  Also there are some reports on possible DOWNREFs. 2 of them
are informative, the other 2 point to Unicode documents.

  (1.h)  References are properly split. There are no downward normative
references.

  (1.i)  An IANA considerations section exists and is clearly defined. It
contains a registration of a new collation algorithm.

  (1.j)  The document doesn't have any ABNF, MIB, etc.
The XML registration template opens fine with Mozilla.

  (1.k)  Document Announcement Write-Up

        Technical Summary

This document describes "i;unicode-casemap", a simple
case-insensitive collation for Unicode strings. It provides
equality, substring and ordering operations.

        Working Group Summary

This document is an individual submission. It was informally last
called in the IMAPEXT WG and consensus was reached that this collation
would be easier to implement than i;basic, thus it has a better chance
of being deployed.

        Document Quality

There is at least one server implementation of this document. At least
2 other server vendors are interested in implementing it.

        Personnel

Alexey Melnikov  is the document shepherd
for this document.
Lisa Dusseault  is the responsible AD.