Internet Draft M. Duerst
<draft-ietf-acap-langtag-00.txt> University of Zurich
Expires in six months June 1997
Two Alternative Proposals for Language Taging in ACAP
Status of this Memo
This document is an Internet-Draft. Internet-Drafts are working doc-
uments of the Internet Engineering Task Force (IETF), its areas, and
its working groups. Note that other groups may also distribute work-
ing documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months. Internet-Drafts may be updated, replaced, or obsoleted by
other documents at any time. It is not appropriate to use Internet-
Drafts as reference material or to cite them other than as a "working
draft" or "work in progress".
To learn the current status of any Internet-Draft, please check the
1id-abstracts.txt listing contained in the Internet-Drafts Shadow
Directories on ds.internic.net (US East Coast), nic.nordu.net
(Europe), ftp.isi.edu (US West Coast), or munnari.oz.au (Pacific
Rim).
Abstract
For various computing applications, it is helpful to know the lan-
guage of the text being processed. This can be the case even if oth-
erwise only pure character sequences (so-called plain text) are han-
dled. From several sides, the need for such a scheme for ACAP has
been claimed. One specific scheme, called MLSF, has also been pro-
posed, see draft-ietf-acap-mlsf-01.txt for details. This document
proposes two alternatives to MLSF. One alternative is using
text/enriched-like markup. The second alternative is using a special
tag-introduction character. Advantages and disadvantages of the var-
ious proposals are discussed. Some general comments about the topic
of language tagging are given in the introduction.
1. Introduction
This introduction contains some considerations about language infor-
mation that should help to better understand why and where language
Expires in six months [Page 1]
Internet Draft Alternative Language Tagings for ACAP May 1997
information can be beneficial. They are intended for general informa-
tion, and are not directly related to the specifics of the proposals
made in this document.
1.1 Multilingual Text
It is sometimes claimed that text, in order to be multilingual, has
to contain language information of some kind or another. This is
definitely not the case. Multilingual text has existed for centuries
on paper and other writing materials, and for decades in computers,
without language information. A huan reader with the necessary lan-
guage background is always able to understand a multilingual text
without explicitly being told which language each word or character
belongs to. In some cases, there may be ambiguities, but this is
either intended, such as in a pun or joke, it is because the reader
is not fully familliar with the involved languages, or it is because
the writer was no precise enough.
The overwhelming majority of characters always has been used in vari-
ous languages, and a character per se therefore cannot be associated
to a single language. This likewise applies to words (and sometimes
even phrases) out of context.
1.2 Language Taging
While the human reader does not need special language information,
such information can be useful for the purpose of automatic process-
ing of various kinds. These in particular include indexing and
searching, text-to-speech conversion, other conversion operations
such as case conversion and transliteration, spelling and grammar
checks, and high-quality typography.
Two other operations are frequently mentionned as benefiting from
language information: Sorting and machine translation. However, in
the case of sorting, this has to occur according to the expectations
of the viewer, frequently encapsulated as a so-called locale. The
language of each of the items being sorted is not relevant. In the
case of machine translation, the knowledge and effort to translate a
language is by magnitudes higher than the knowledge needed to decide
whether a certain word or sentence belong to a given language.
Explicit language information is therefore of marginal importance.
This also applies to other operations, such as text-to-speech conver-
sion, in particular for high quality and for languages with a compli-
cated relationship between spelling and pronounciation (such as
English).
Expires in six months [Page 2]
Internet Draft Alternative Language Tagings for ACAP May 1997
1.2 CJK(V) Glyph Disambiguation
The reason one hears most for the necessity of language information
is the need to disambiguate CJK(V) ideographic glyphs, i.e. to select
specific typographic variants of certain ideographic characters,
variants which can differ somewhat between Chinese (simplified or
traditional), Japanese, Korean, or classical Vietnamese. Some such
distinctions can indeed be made by using language information to
indicate typographic tradition. However, the usefullness of this
approach is limited by a series of facts that are not all very widely
known:
- Even if a glyph is by mistake taken from another typographic tra-
dition, readability, in particular in context, is never affected.
- In running text, the differences resulting from the use of differ-
ent fonts (e.g. Song-style font vs. Mincho-style) as well as from
different weights of the same font are much more visible than the
differences resulting from glyph variant details.
- In many fonts, glyphs vary, for sund aestetic and historic rea-
sons, to a similar or higher degree than that exhibited in the
"reference" glyphs for each typographic tradition as given in [ISO
10646]. This applies to print, but even more to handwriting.
- National standards explicitly or implicitly allow for a certain
variance of glyph shapes. In particular, the newest edition of the
basic Japanese character standard, JIS X 0208-1997 [JIS1997],
explicitly mentionnes a large number of permitted variants (pp.
12-22) for Mincho fonts only. It also explicitly allows a list of
29 much wider-reaching variants as a consequence of some unfortu-
nate changes to the standard in 1983 (p. 22).
- Long-standing typographic practice does not use special glyph
variants for representing short inclusions of foreign origin (such
as names of persons, places, or institutions) in native text.
- Some glyph variants are seen by some persons as explicit proper-
ties of their names. Identifying a name by a particular language
and assuming that this implies a particular typographic tradition
can in some cases lead to the desired result. However, the results
cannot be guaranteed due to design differences between different
fonts used in the same typographic tradition, and due to the fact
that even national standards glyph standards considerably unify
glyph variants.
All the above facts clearly limit the usefullness of language tags
for CJK(V) glyph variant selection. Language taging should therefore
Expires in six months [Page 3]
Internet Draft Alternative Language Tagings for ACAP May 1997
not be advertized as a comprehensive solution to the various problems
of CJK(V) glyph variant selection.
2. A Text/Enriched-like Notation for Language Tags (TELT)
This section specifies a text/enriched-like notation for language
tags, leading to a format simmilar to text/enriched. It can be used
with any character encoding that contains the necessary subset of the
US-ASCII character repertoire.
Language tags are of the form "<LANG=xxxxx>" where xxxxx is a lan-
guage tag as defined in [RFC1766], with all letters written in upper
case. No whitespace of any kind is allowed between "<" and ">".
Language alternatives are started by "<ALTLANG>". Again, no whites-
pace is allowed between "<" and ">".
The use of the character sequences "<LANG=" and "<ALTLANG>" is not
allowed in the text itself. Code to convert from this notation to
MLSF and back and to test for false positives in plain text search is
given in an appendix.
3. Language Tags using a Start Tag Character (STLT)
This method of language taging is only useable with character encod-
ings that can represent the BMP of the Universal Character Set
[ISO10646]. For the purpose of illustration, the character PILCROW
SIGN (paragraph sign, U+00B6) is used as the tag start character. It
would be preferable to officially define a currently unused code
point exclusively for this purpose, but such a definition is outside
of the scope of this document and outside of the scope of IETF work.
If this solution is seriously considered for adoption by the IETF for
use in some of it's protocols, a request for such a codepoint should
be made through the appropriate channels.
For possible future expansions, tag syntax after the start tag char-
acter is kept very simple and general. Tags are defined to start with
a tag start character, contain only characters from the US-ASCII
repertoire (U+0021 through U+007E, inclusive), excluding the tag end
character, and end with a tag end character. The character "#" is
choosen as a tag end character.
Language tags proper are formed by a start tag character, a language
tag according to [RFC1766], with all letters in upper case, and a "#"
as an end tag character, without any intervening white space.
Expires in six months [Page 4]
Internet Draft Alternative Language Tagings for ACAP May 1997
Language alternatives are marked by a sequence of a start tag charac-
ter, a "%", and a "#" as an end tag character, again without any
intevening white space. Code to convert from this notation to MLSF
and back and to test for false positives in plain text search is
given in an appendix.
4. Conformance
Conforming protocols using either of the solutions proposed above
MUST clearly define in which places they do so, and in which places
they don't. If there are other mechanisms in the procotol that can be
used for language taging, these mechanisms should be considered and
used. In particular, storing language information separate from the
actual text is beneficial in many cases because it allows the proto-
col to treat language information and language alternatives in a way
appropriate to the protocol, i.e. only selecting and transmitting
language alternatives desired by the client, and so on.
Conforming protocols and their implementations MUST at all costs
avoid that language tags leak into parts of the protocol where they
are not allowed or into other channels where they are not allowed. In
the absence of specific information to the countrary, a protocol or
implementation MUST assume that another protocol or implementation
does not allow language tags.
In interfaces to protocols and formats that use other ways of lan-
guage taging (for an example HTML, see [RFC2070]), conforming proto-
cols SHOULD convert language tags appropriately or MAY eliminate
them.
If text including language tags as defined in this document leaks
outside the protocol positions where it is explicitly allowed, it
should be treated in the same way other text is treated, with no spe-
cial processing.
5. Discussion
Two alternative forms for language taging have been proposed in this
document. Because they are very simillar, only one of them should
finally be choosen. Compared to [MLSF], their main advantages are
that they can be used with character encodings other than UTF-8, that
they are easily distinguished from UTF-8 by implementors and users,
and that they are advantageous in case of debuging and initial string
composition.
The MLSF proposal has a number of interesting properties that makes
Expires in six months [Page 5]
Internet Draft Alternative Language Tagings for ACAP May 1997
it very suitable for efficient internal processing in certain scenar-
ios. We therefore in particular give conversion functions between
MLSF and our proposals in the appendices.
MLSF continues a long tradition of utilizing unused bit combinations
for internal processing speedups. Exposing such methods to the out-
side of an implementation, however, can lead to serious restrictions
and undesired biases towards certain implementations.
The main difference between the two proposals given here is that TELT
has to exclude certain character sequences from the untagged text,
whereas STLT has a potential to use a special, newly defined, code-
point, that is guaranteed not to appear in text per se.
Acknowledgements
The motivation to write this document came from Harald Alvestran.
Further acknowledgements go to Lisa Moore, Mark Davis, Ken Whistler,
Glenn Adams, and others from the UTC (Unicode Technical Committee),
to Rob Pike, and to Chris Newman, Ned Freed, and Mark Crispin from
the IETF ACAP working group.
Bibliography
[Unicode2] The Unicode Standard, Version 2, Addison-Wesley, Read-
ing, MA, 1996.
[ISO-10646] ISO/IEC 10646-1:1993. International Standard -- Infor-
mation technology -- Universal Multiple-Octet Coded
Character Set (UCS) -- Part 1: Architecture and Basic
Multilingual Plane.
[JIS1997] Japanese Industrial Standard, "7-bit and 8-bit Double
Byte Coded Kanji Sets for Information Interchange",
JIS X 0208:1997.
[MLSF] C. Newman, "Multi-Lingual String Format (MLSF)",
draft-ietf-acap-mlsf-01.txt, work in progress, June
1997.
[RFC1766] Alvestran, H., "Tags for the Identification of Lan-
guages", RFC 1766.
Expires in six months [Page 6]
Internet Draft Alternative Language Tagings for ACAP May 1997
[RFC2070] F. Yergeau, G. Nicol, G. Adams, and M. Duerst, "Inter-
nationalization of the Hypertext Markup Language", RFC
2070, January 1997.
Author's Address
Martin J. Duerst
Multimedia-Laboratory
Department of Computer Science
University of Zurich
Winterthurerstrasse 190
CH-8057 Zurich
Switzerland
Tel: +41 1 257 43 16
Fax: +41 1 363 00 35
E-mail: mduerst@ifi.unizh.ch
NOTE -- Please write the author's name with u-Umlaut wherever
possible, e.g. in HTML as Dürst.
Appendix A. Conversion from TELT to MLSF
This is sample code to convert from text/enriched-style language tag-
ing to MLSF. It is assumed that the source is in UTF-8, and that the
output buffer (outp) is long enough to hold the result. The code uses
the functions defined in [MLSF] for convenience.
#include <string.h>
void TELTtoMLSF (unsigned char *outp, unsigned char *inp)
{
unsigned char tagbuff[256];
unsigned char *temp;
while (*inp) {
Expires in six months [Page 7]
Internet Draft Alternative Language Tagings for ACAP May 1997
if (!strncmp(inp, "<ALTLANG>", 9)) {
inp += 9;
*outp++ = 0xFE;
}
else if (!strncmp(inp, "<LANG=", 6)) {
inp += 6;
temp= tagbuff;
while (*inp != '>')
*temp++ = *imp++;
*temp= 0;
outp += MLSFlangencode(outp, tagbuff);
}
else
*inp++= *outp++;
}
*outp= 0;
}
Appendix B. Conversion from STLT to MLSF
This is sample code to convert from Start Tag Character style lan-
guage taging to MLSF. It is assumed that the source is in UTF-8, and
that the output buffer (outp) is long enough to hold the result. The
code uses the functions defined in [MLSF] for convenience.
void STLTtoMLSF (unsigned char *outp, unsigned char *inp)
{
unsigned char tagbuff[256];
unsigned char *temp;
while (*inp) {
if (!strncmp(inp, "\xC2\xA7%#", 4)) {
inp += 4;
*outp++ = 0xFE;
}
else if (!strncmp(inp, "\xC2\xA7", 2)) {
inp += 2;
temp= tagbuff;
while (*inp != '&')
*temp++ = *imp++;
*temp= 0;
outp += MLSFlangencode(outp, tagbuff);
}
Expires in six months [Page 8]
Internet Draft Alternative Language Tagings for ACAP May 1997
else
*inp++= *outp++;
}
*outp= 0;
}
Appendix C. Conversion from MLSF to TELT
This is sample code to convert from MLSF to text/enriched-style lan-
guage taging. It is assumed that the output buffer (outp) is long
enough to hold the result. The code uses the functions defined in
[MLSF] for convenience.
void MLSFtoTELT (unsigned char *outp, unsigned char *inp)
{
unsigned char tagbuff[256];
unsigned char *temp;
int len;
while (*inp) {
/* for speed, first insert a test (*inp != "<") */
if (*inp == 0xFE) {
inp++;
strcpy (outp, "<ALTLANG>");
outp+= 9;
}
else if (*inp >= 0xC0 && inp[1] > 0xC0) {
inp+= MLSFlangdecode(tagbuff, inp);
strcpy (outp, "<LANG=");
outp+= 6;
temp= tagbuff;
while (*temp)
*outp++ = *temp++;
*outp++ = ">";
}
else { /* maybe just *outp++ = *inp++ is enough here? */
len = utlen[*inp];
if (len > 6) break;
while (len-- && *src)
*outp++ = *inp++;
}
}
*outp= 0;
}
Expires in six months [Page 9]
Internet Draft Alternative Language Tagings for ACAP May 1997
Appendix D. Conversion from MLSF to Start Tag Character
This is sample code to convert from MLSF to Start Tag Character style
language taging. It is assumed that the output buffer (outp) is long
enough to hold the result. The code uses the functions defined in
[MLSF] for convenience.
void MLSFtoSTLT (unsigned char *outp, unsigned char *inp)
{
unsigned char tagbuff[256];
unsigned char *temp;
int len;
while (*inp) {
if (*inp == 0xFE) {
inp++;
strcpy (outp, "\xC2\xA7%#");
outp+= 9;
}
else if (*inp >= 0xC0 && inp[1] > 0xC0) {
inp+= MLSFlangdecode(tagbuff, inp);
strcpy (outp, "\xC2\xA7");
outp+= 6;
temp= tagbuff;
while (*temp)
*outp++ = *temp++;
*outp++ = "&";
}
else { /* maybe just *outp++ = *inp++ is enough here? */
len = utlen[*inp];
if (len > 6) break;
while (len-- && *src)
*outp++ = *inp++;
}
}
*outp= 0;
}
Appendix E. Elimination of False Positives in TELT
This is sample code to eliminate false positives in TELT (i.e. check-
ing whether a match found by a search routine starts inside a tag).
The elimination of false positives in STLT is structurally equivalent
Expires in six months [Page 10]
Internet Draft Alternative Language Tagings for ACAP May 1997
and therefore not given explicitly here.
int TELTfalse (unsigned char *inp, unsigned char *pos)
{
while (inp <= pos) {
if (!strncmp(inp, "<ALTLANG>", 9)) {
inp += 9;
if (inp > pos)
return 1;
}
else (!strncmp(inp, "<LANG=", 6)) {
inp += 6;
while (*inp++ != '>') ;
if (inp > pos)
return 1;
}
}
return 0;
}
Appendix F. Elimination of Tags from TELT
This is sample code to eliminate tags from TELT (in UTF-8), thereby
leaving only plain text. The elimination of tags from STLT is struc-
turally equivalent and therefore not given explicitly here.
void TELTclean (unsigned char *inp, unsigned char *outp)
{
while (*inp) {
if (!strncmp(inp, "<ALTLANG>", 9))
inp += 9;
else if (!strncmp(inp, "<LANG=", 6)) {
while (*inp++ != '>') ;
}
else
*inp++= *outp++;
}
*outp= 0;
}
Expires in six months [Page 11]
Internet Draft Alternative Language Tagings for ACAP May 1997
Appendix G. Selection of the "best" alternative from TELT
This is sample code selects the "best" language match from TELT.
Assume input language tag has been converted to upper case. Assume
language tags won't exceed 256 characters. Returns a pointer to the
start of the "best" match. Code for STLT is structurally equivalent
and therefore not given explicitly here.
unsigned char TELTselect (unsigned char *inp, unsigned char *tag)
{
unsigned char tagbuff[256];
unsigned char *match1, match2;
unsigned char best= str;
int bestlen= 0;
int start= 1;
int mlen;
if (tag == NULL || !*tag)
return;
while (*inp) {
if (!strncmp(inp, "<ALTLANG>", 9)) {
inp += 9;
*outp++ = 0xFE;
start= 1;
}
if (start) {
mlen= 0;
/* get tag into tagbuff */
if (!strncmp(inp, "<LANG=", 6)) {
inp += 6;
match1= tagbuff;
while (*inp != '>')
*match1++ = *imp++;
*match1= 0;
inp++;
}
else *tagbuff= 0;
/* check match */
match1= tagbuff;
match2= tag;
while (*match1 && *match1++ == *match2++) {
if (*match2=="-" && (*match2=="-" || !*match2))
mlen = match1 - tagbuff;
}
Expires in six months [Page 12]
Internet Draft Alternative Language Tagings for ACAP May 1997
if (!*match2 && (*match1=='-' || !*match1)) {
best = str;
break;
}
if (mlen > bestlen) {
best = str;
bestlen = mlen;
}
/* search next alternative */
start = 0;
}
else
inp++;
}
return best;
}
Expires in six months [Page 13]
