Q12. What is JPerl?

A12. JPerl (Japanized Perl) may also be used for Japanese localization.

JPerl for MS-Windows JPerl patch for Unix

Lunde's CJKV also has references to JPerl on pp. 412 and 444-446.

If I remember correctly, kipp would create a patch that would Japanize Perl after each Perl release. Finally, JH rolled most of his changes into core Perl.

There is also a Macintosh Version of JPerl

/>/usr/local/bin/perl -v

This is perl, version 5.004_04 built for i386-freebsd

Copyright 1987-1997, Larry Wall

Japanization patch 4 by Yasushi Saito, 1996

Modified by Hirofumi Watanabe, 1996-1998
jperl5.004_04-980303
LATIN version

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5.0 source kit.

Q13. Can I just do nothing and let my program be agnostic?

A13. Some legacy applications work without knowledge of which or how many native encodings are being used as long as the round-trip from client to server and back does not mangle the data. Generally this only works when no date or time display or casing or sorting are required and the client encoding can be assumed.

This would work better for European character sets which are usually single-byte in a single encoding than Japanese for example. Japanese has a multitude of multi-byte encodings, the most common being ISO-2022-JP, S-JIS, EUC-JP and many more.

Web applications adopt the common practice of storing data in Unicode and sending data to the browser in native encodings. When multiple character encodings are required on a single web page, (like Japanese combined with non-English text), UTF-8 is used.

Q14. Why and where should I use Unicode instead of native encodings?

A14. The simple answer is that Unicode should be used internally for processing and databases now, and in the future sent to the client program.

The advantage of internal Unicode is the ability to simultaneously store multiple languages in a common format without creating separate databases for Korean, Japanese, Chinese, and English, for example.

The current advantage of sending and receiving native character sets to the user is legacy support for browsers or other older clients and operating systems.

Although Netscape and IE 4.0+ have basic Unicode support, use after testing. (The MyNetscape feature of netscape.com uses UTF-8 when displaying languages that would normally require multiple character sets on the same web page, like Japanese and non-English languages.)

Browsers generally seem to have the most difficulty displaying UTF-8 in forms, alt text and JavaScript strings (older versions of Netscape browsers would not parse non-ASCII text correctly just before a double-quote).

Andreas Koenig's Perl Conference 3.0 HTML slides are in UTF-8

Q15. What is Unicode normalization and why is it important?

A15. Normalization is a transformation applied to a Unicode character sequence. The transformation is performed according to the Unicode Standard and involves table lookups to do substitution and reordering of characters as defined in the Unicode Character Database. Normalization is necessary because Unicode allows multiple representations of the same string with combining characters and precomposed characters. Normalized character sequences allow accurate binary comparison of strings. This is especially important for purposes such as digital signatures, variable identifiers, mailbox folder names and directory searches.

Early normalization is performing normalization as close as possible to the text source, before transmitting the text to another application. This must be done for XML, HTML, URIs, and other W3C standard systems.

Late normalization is performing normalization after receiving text from another process or network transfer.

Canonical equivalence of 2 character sequences is determined by applying operations defined in the Unicode standard, without changing the visual appearance of resulting text.

Compatibility equivalence of 2 character sequences is determined by applying operations defined in the Unicode standard, and may change the visual appearance of text (for example, input text with both half and full-width katakana would result in full-width only output).

There are four normal forms (D, C, KD and KC), documented in Unicode Standard Annex #15.

Briefly:

• Form D - Canonical Decomposition
• Form C - Canonical Decomposition followed by Canonical Composition (required in W3C, c14n)
• Form KD - Compatibility Decomposition
• Form KC - Compatibility Decompostion followed by Canonical Composition

Martin Dürst of the World Wide Web Consortium (W3) has written reference code in Perl for normalization, called Charlint - A Character Normalization Tool

Q16. How do I do auto-detection of Unicode streams?

A16. There is no mandatory signature, making this hard to do reliably. The optional Byte Order Mark (BOM) U+FFFE may help if present. UTF-7 and UTF-8 strings have properties that would make them identifiable most of the time unless there is substantial leading US-ASCII.

Without a signature you would need a moderate amount of text to do a reliable detection. An example of an input source that is probably not long enough would be a search widget on a web page. Statistical analysis of octets is another unreliable method.

Frank Tang's site has some UTF-8 auto detection scripts

See Ken Lunde's CJKV book, Appendix W: Perl Code Examples, for numerous character encoding, detection, and regular expressions in perl.

Q17. Is Unicode big endian or little endian?

A17. Unicode characters are not considered to be endian when stored in memory.

However, UTF-16 and UTF-32 must be serialized when saved in a file, emitted as a stream, or transferred over the network. Network byte ordering (most significant byte first or big-endian) is preferred, but you would not likely see it on Intel hardware. Mozilla 5 supports both UTF-16-LE and UTF-16-BE, etc!

Q18. Is there an EBCDIC-safe transformation of Unicode?

A18. Yes. UTF-EBCDIC stands for EBCDIC-friendly Unicode (or UCS) Transformation Format. Unicode Technical Report #16

PP says:

As of Perl 5.6.1 (and also in bleedperl developers versions 5.7.1++) the use utf8;
pragma on an EBCDIC machine will have the same effect with respect to
UTF-EBCDIC that use utf8; has with respect to UTF-8 on an ASCII based
machine.  Hence there is about as much support for UTF-EBCDIC in current
Perls as there is for UTF-8 on ASCII-based machines.

Where to Use UTF-EBCDIC?

UTF-EBCDIC is intended to be used inside EBCDIC systems or in closed networks where there is a dependency on EBCDIC hard-coding assumptions. It is not meant to be used for open interchange among heterogeneous platforms using different data encodings. Due to specific requirements for ASCII encoding for line endings in some Internet protocols, UTF-EBCDIC is unsuitable for use over the Internet using such protocols. UTF-8 or UTF-16 forms should be used in open interchange.

Peter Prymmer's Perl EBCDIC document

Q19. Are there security implications in i18N?

A19. Yes. Since locale affects Perl's understanding of strings, operations like character comparison may produce unexpected results when the locale is manipulated.

The UTF-8 Corrigendum better defines UTF-8 by forbidding generation and interpretation of non-shortest forms, which will help disambiguate characters.

Perl taints strings affected by locale.

An example is using a regex with \w to check user input. \w may accept character codes for punctuation characters if the locale "says" they are alpha codes.

From the Sun man attributes page:

For applications to get full support of internationalization services, dynamic binding has to be applied. Statically bound programs will only get support for C and POSIX locales.

CERT* Advisory CA-97.10 - Topic: Vulnerability in Natural Language Service

Some comments from Bruce Schneier

Q20. Are there performance issues in i18N?

A20. Yes. Locale-dependent collation is generally slower than binary comparisons.

Some of the reasons are:

• While ASCII is single-byte, UTF-8 and UTF-16 are multi-byte encodings.
• Often client data is accepted in native encodings and stored in Unicode, requiring conversion.
• Unicode transformations require more storage space than native encodings generally, due to the tables required.

A contributor reports that when using regular expressions like /^([\w]*)([\W]*)/ in perl 5.6, performance has been measured to be similar to that in pre-5.6 on straight ASCII.

Q21. How do I localize strings in my program?

A21. The best way is to rip them out and save them in an external string resource file. The program API should load strings by string_id (preferably a descriptive text literal rather than a cryptic numeric code like Windows resources) and lang_locale (lang_locale should be either default locale or explicit locale). An additional complication is that text may have parameters that are substituted in locale- dependent positioning (Perl does not have the printf reorder edit descriptors $1, etc. yet). Some programmers provide the default locale's string as an argument to the catalog API call. This may be ok where the program is small and the localizer is also a developer with access to the source code, but I do not recommend this for large projects or where strings are often updated. Leaving interface text in the source code is bad.

Another advantage of using string resource files is that translators can work on the files without touching your source code. This reduces cost, avoids inadvertant source mgmt. errors and limits write access to the repository.

There's 4 strategies for resource file formats:

1. ICU resource bundles
   ICU locales are literals of the form lang_region_variant that inherit from a top-level locale known as root. Resource elements similarly inherit from root.
2. stored in Perl hash or array
3. in gettext-compatible external file
   Phillip Vandry has written Locale::gettext (a perl wrapper around C routines), and James has written a CPAN module called Gettext [sic] that does a backquote call to gettext to do this.
4. in non-gettext-compatible external file or database. dbm and dbFile seem to be popular.
   Mike Shoyher has written Locale::PGetText (which is pure perl and does not rely on the system locale, but does not follow the gettext API standard or file format and uses dbm). I have seen a program use dbFile.
   Even more interesting would be using database tables for the string catalog.

Programming for Internationalization FAQ talks about gettext, by Michael K. Gschwind.
GNU gettext manual and GNU gettext download
SunDocs has information on gettext here.

Q22. I do database programming with Perl. Can I use Unicode?

A22. This is well-supported with Oracle 8 and Sybase. Usually your database must be installed with the correct character set. WS from GlobalSight recommends setting your data locale to whichever your office most commonly uses for reporting.

Oracle 8

Oracle has broad i18N support. It is important to read the correct documentation for your version of Oracle.

Likely for legacy reasons, Oracle servers must be configured with 2 character sets:

1. the default database character set must be single-byte or variable-length multi-byte and should be a superset of the national character set (explained below). This character set affects almost everything - table names, most field names, PL/SQL symbols, etc. char(20) and varchar(20) always mean 20 bytes.
2. the national character set can be fixed-width or variable-with multi-byte. The only fields affected are NCHAR, NVARCHAR2, and NCLOB. nchar(20) and nvarchar2(20) mean 20 *characters* if fixed-width, 20 *bytes* if variable-width.

Oracle National Language Support (NLS)
Oracle 8 National Language Support: An Oracle Technical White Paper

Oracle iAS (Internet Application Server) is a very interesting product. It is bundled with Apache, and integrates Apache language content negotiation with Java local support for impressive locale mgmt. There is a demo called World-O-Books.

Oracle will likely add a UTF-16 encoding in 2001.

It has been reported that DBI does not work with UTF-8 stored in Oracle nvarchar2 fields.

Sybase

Configure with UTF-8. Sybase has an alpha version of UTF-16 with beta release in November, 2000 and full release in 2001. New column types unichar and unitext have been added, which may be optionally configured for surrogate use or not. They added UTF-16 to meet user requests for a wide Unicode datapath for Java and NT applications.

Microsoft SQL Server

Michka Kaplan has this to say:
SQL Server supports the datatypes NTEXT, NCHAR, and NVARCHAR, all of which are of type UCS-2. When such a column indexed, then the index is Unicode. SQL Server 7.0 only supports one language collation at the server level.... this choice affects the actual ordering of all such indexes.
SQL Server 2000 supports a COLLATE keyword that allows you to specify a collation at the database or field level and thus choose a different language for such columns/indexes if you like (I discuss practical details and implications of this feature in an upcoming article in the Visual Basic Programmer's Journal, tentatively scheduled for November). See also his book, Internationalization with Visual Basic.
In any case, you can certainly query and such field in either SQL 7.0 or in SQL 2000.

Informix

*** TBA ***

DB2

*** TBA ***

MySQL

MySQL supports many encodings, but not Unicode. And these many encodings can only be used 1 per database. (So you could create a Korean-only or Japanese-only database.)

The MySQL manual lists Unicode as one of the "things that must done in the real near future."

By default, accented characters are collated to the same value. This can be overridden by/compensated by:

• defining the column as binary
• defining the column as non-unique
• creating a new character table

A work-around would be to use the default ISO-8859-1 and have all your database routines convert to UTF-8 and translate to native encodings. You would lose the ability to sort automatically and would need to do your own conversion to native character encodings for presentation to the client.

Progress Software is considering combining ICU and MySQL. Progress has experience adding Unicode to their Progress database product.

Basis Technology has added Unicode support to MySQL using Rosette's Unicode library and may be involved with adding ICU support also.

PostgreSQL

There is an install option --enable-multibyte-=UNICODE for unicode support. The option --enable-unicode-conversion enables Unicode to legacy conversion tables. No further information so far on how well it works.
PostgreSQL Localization Documentation

Q23. I do database programming with Perl. What are the i18N issues?

A23. The usual issues must be dealy with, like collating (sorting) and conversion to native encodings for application input and output.

A new issue is ensuring your database columns are wide enough. UTF-8 for example can triple the size of a Japanese column.

Also, if your fields were designed for English text, they will have to be up to 100% wider for other languages, like French.

So to be safe, I would quadruple originally English ASCII text columns for use with UTF-8 databases that will store CJKV characters.

While many people use HTML entity codes ("entification") to represent accented characters in static HTML, this aggravates collation so I recommend using actual character values in databases and content management systems instead of entity values. A post-processing filter can always be used if there is a need for entity codes. One such requirement would be for foreign language support in the older Mac browser versions. Also beware of user page creation tools that gratuitously emit smart quotes HTML output.

Q24. How do other programming languages implement Unicode and i18N?

A24. Programming Languages

Ken Lunde's CJKV, p. 410 also describes some features of C/C++, Java, Python and TCL.
Tony Graham's A Primer for Unicode also has a language survey chapter.

Unicode: APrimer has a good comparison of programming languages

Java language

Java has what programmers generally acclaim as the best Unicode support. Since Java 1.1, i18N support is more or less based on ICU.

• characters and strings - always Unicode UTF-16. Java 1.0 supports Unicode 1.1, and Java 1.1 supports Unicode 2.0.
• character set conversion - Java 1.1 has methods to import/export native encodings or UTF-8
• localized number formatting - yes
• localized date and time - yes
• localized calendar functions - yes
• collation - yes
• localized string resources - yes, resource bundles are used to load string resources by locale and can even handle dynamic substitution of parameters.

Oddly, there is an interface module to Java called Java.pm, a Perl module for extracting information from Java class files. No functions to parse Java resource bundles though.

See Also

O'Reilly's Java In a Nutshell, Chapter 11
Sun's Java i18N Tutorial
IBM's Unicode site

C language

C provides the wchar_t and wchar_t * type for handling wide characters and strings. Since wchar_t varies in width by compiler, for portable programming you should define a macro of the appropriate width rather than use wchar_t directly. ISO C Amendment 1 and ISO C 99 both added a rich set of new library functions for handling wchar_t strings to the language. C can also handle Unicode by using UTF-8 as the multi-byte encoding in the char * type, such that upgrades from ASCII to UTF-8 can be implemented with relatively minor changes in most existing software.

Regular C string functions like strcmp may not work with embedded null values.

Since most operating systems are substantially written in C, the C locale libraries are available to applications as well.

ICU (International Classes for Unicode) may be used to internationalize C/C++ programs.
Basis Technology's Rosette Library C/C++ is commonly used for adding Unicode support to custom programs. For a demo, try Lycos.
If you are porting old C/C++ source to Unicode on Unix or Windows, you may want to try OneRealm's Eliminator program for converting and auditing source code.

C++ language

See C language above.

ECMAscript (aka JavaScript, LiveScript)

• strings - always UTF-16. There is no character type. No normalization is done internally, so it expects to receive Unicode Normalization Form C, which the W3C recommends.
• localized number formatting - yes
• localized date and time - yes
• localized calendar calculations - primitive

Ada95 language

Ada95 was designed for Unicode support and the Ada95 standard library features special ISO 10646-1 data types Wide_Character and Wide_String, as well as numerous associated procedures and functions. The GNU Ada95 compiler (gnat-3.11 or newer) supports UTF-8 as the external encoding of wide characters. This allows you to use UTF-8 in both source code and application I/O. To activate it in the application, use "WCEM=8" in the FORM string when opening a file, and use compiler option "-gnatW8" if the source code is in UTF-8. See the GNAT and Ada95 reference manuals for details.

XML

XML requires parsers to support both UTF-8 and UTF-16. Optionally, XML parsers may support other character sets, such as ASCII.

A contributor writes:
"It's like so: if you're using UTF-16, you *must* have the BOM. If you're using UTF-16-LE or BE, then even though these are really UTF-16, you can avoid using the BOM. Some people over at the IETF are trying to *forbid* the BOM with the -BE and -LE versions, which essentially means that they shouldn't be used for XML, since XML is all about interoperability, and byte order just can't relied on outsde the firewall."

Extensible Markup Language (XML) 1.0 - Character Encoding in Entities
Extensible Markup Language (XML) 1.0 - Autodetection of Character Encodings (Non-Normative)

The CPAN module XML::Parser has its own character transcoding routines (multi-byte no less), which are documented in the source in Expat/expat/xmlparse/xmlparse.h, Expat/encoding.h and Expat/Expat.xs. There is an interesting description of the lack of standards in the S-JIS and EUC-JP character set encodings in Japanese_Encodings.msg.

XML::Parser uses wchar_t, which the Unicode Standard recommends against as wchar_t varies in width depending on the compiler. XML::Parser also engages in old-fashioned C character processing loops, which are inadvisable in i18N programming.

Python

Python i18N SIG

TCL

How to write a transformation (channel)

Q25. What support for Unicode do web browsers have?

A25. This varies depending on browser version.

Netscape 4.x and IE 4.x and above support UTF-8 and possibly other transformation formats reasonably well. Statistics I have seen show that 99% of page views are with 4.x and above. UTF-8 is fine for content pages.

Some problems that have been observed are:

• in Windows Communicator 4.7, View Source on a UTF-8 encoded page while the ISO-8859-1 character set is selected will show hollow rectangles between characters.
• in pre-4.06 Netscape Communicator, JavaScript strings can break if they end in a binary value that could be confused with a quote character
• forms maybe tricky with UTF-8
• the default Japanese fonts used with Navigator for Linux have been described as "unreadable". Try TwinBridge for different fonts.
• It has been reported that you lose the ability to get script-less auto-download of language support for ARA/HEB/THA/VIE/CHT/CHS/KOR/JPN for people using IE 5.0/5.01/5.5/etc.

Q26. How can I i18N my web pages and CGI programs?

A26. In a nutshell, by generating the correct headers and HTML meta tags.

HTML4 uses UCS as its document character set, but the document may be transferred with a different character encoding. That is why numeric entity codes index UCS code values rather than S-JIS or ISO-8859-1 or whatever character encoding was used to transfer the document.

Please seee Notes on Internationalization for details. To summarize, it recommends configuring your web server to send the correct Content-type character encoding response and not using META CHARSET tags, especially with ISO-8859-1 text. Some older browsers are known to ignore charsets that are not ISO-8859-1, so this may not be a perfect solution.

When generating static pages, I would emit the correct META tag to help browsers of foreign readers default to the correct character encoding. (Do a view source on this page to see one.)

The correct meta tag is helpful even when the server could send the correct header, because some browser versions do not understand non ISO-8859-1 charset set header names, like ISO-8859-5 for example. Otherwise older browsers may see the file as an unrecognizable encoding and treat it as a binary that must be saved to disk. Although the language meta tag should appear as the first tag in the head block, it may not make that much difference where it is placed as some believe the entire head tag is processed as a unit.

To make your HTML render correctly, avoid font face tags in CJKV pages that users may not have. Otherwise mojibake or hollow squares will likely result (observed with Netscape browsers on both Windows and and Mac platforms.) Using a very general font tag like <font face="sans-serif,Arial,Helvetica,serif"> may work by providing enough defaults so some font will match. Also, the non-breaking space tag may cause visual problems when displaying Japanese text on the Mac in older Netscape browsers.

A work-around for displaying multiple native encodings on the scree simultaneously is to use multiple frames, one for each charset.

It has been reported that as of now (Sept. 12), UTF-8 is not usable as a general charset for public Internet use when Asian languages are displayed on Netscape 4.x because of issues of an available Unicode font and low likelihood that your audience can find and install a Unicode font.

HTML::Entities::{de,en}code_entities can be used to translate HTML entity codes from de to en for example. W3C SGML Entities

When generating dynamic pages, emit the correct server response header:

    use CGI.pm qw /:standard/;
    print header("text/html; charset=iso-8859-1"),
       start_html();

print start_html( -head => q{

});

Forms may be constructed with a hidden field with charset input and locale (language and country) fields.

IE5 and later will fill a field called _charset_.

<input type="hidden" name="_charset_">

Forms may also be constructed have a hidden field with known sample text to be tested for mangling. This would be especially effective with Japanese, since multiple character sets are in common use, including ISO-2202-JP, Shift-JIS, and JIS.

RFC2070 recommends using the ACCEPT-CHARSET hint, like ACCEPT-CHARSET="utf-8" on both the FORM element and the FORM controls. There is substantial ambiguity in how the user, browser client, and server would interact with ACCEPT-CHARSET, so this method would need some experimentation.

With

FORM METHOD=GET ...  and
FORM METHOD=POST ENCTYPE="application/x-www-form-urlencoded" ...

the %HH codes represent octets but there is not standard for specifying the character encoding. Only ENCTYPE=multipart/formdata (sp?) provides the opportunity to send the encoding information. So far only lynx uses it.

Altavista's international country selector page illustrates using .GIF images to render multiple character sets on the same non-Unicode encoded page.

The recommended character set encoding for Japanese web pages is Shift-JIS.

Fascinating email on browser charset issues

Q27. How should I structure my web server directories for international content?

A27. If you have more than a few pages for each locale, the usual practice is to have separate directories for each locale to make content management easier. This is especially effective for truly localized content like local news, or when some locales get more frequent updates than others.

Here's an example:

http://home.netscape.com/fr/index.html, or perhaps even better depending on your requirements:
http://home.netscape.com/fr_FR/index.html

Q28. Can web servers automatically detect the language of the browser and display the correct localized page?

A28. Yes. HTTP/1.1 defines the details of how content negotiation works, including language content.

WWW browsers send an Accept-Language request header specifying which languages are preferred for responses. This technique works fairly well, although some versions of Netscape Navigator send an improperly formatted request parameter. Also, switching language preferences in either Navigator or IE 4 doesn't always "take" without first deleting a language hint.

Few sites do content-negotiation on language, and interestingly enough I do not know of any major portals doing this. One site that does is Sun's documentation library at SunDocs. Debian.org does a very nice job of using Apache content negotiation wih languages and also has some nice help info too on Setting the Default Language.

Apache's Content Negotiation features will select the right page to return, whether HTML or image file. Annoyingly, the match logic is very literal, so a browser request for en-us will not match a server entry of en except as a last resort. Any other exact match will win over en, even if en-us was first preference.

There are 2 ways of doing content negotiation with Apache: type or variant maps and multiviews.

Variant Maps

In httpd.conf, disable Options Multiviews if configured and add

AddType type-var var
DirectoryIndex index.var

Then create an index.var file like this:

URI: start; vary="type,language"

URI: index.html.en
Content-type: text/html
Content-language: en-GB

URI: index.html.en
Content-type: text/html
Content-language: en-US

URI: index.html.en
Content-type: text/html
Content-language: en

URI: index.html.fr
Content-type: text/html
Content-language: fr-CA

URI: index.html.fr
Content-type: text/html
Content-language: fr

URI: index.html.es
Content-type: text/html
Content-language: es

Multiviews

The multiviews technique works like this. This does add extra server load, as each content directory must be scanned for the variant document names.

index.html is localized into variant documents such as:

• index.html.en (or index.en.html)
• index.html.fr
• index.html.es
• index.html (or index.html.html as a reader has recommended) symlinked to one of the above as the last resort.

Here's an example of http.conf directives for this:

     # in httpd.conf
     AddLanguage en .en
     AddLanguage fr .fr
     AddLanguage es .es

     #
     # LanguagePriority allows you to give precedence to some languages
     # in case of a tie during content negotiation.
     # Just list the languages in decreasing order of preference.
     #
     LanguagePriority en es fr de pt
     
     Options Multiviews
     
     # end of httpd.conf fragment

Starting in Apache 1.2, you may also create documents with multiple language extensions.

O'Reilly's Apache - The Definitive Guide, Chapter 6
ApacheWeek article on Language Negotiation
Another ApacheWeek article on Language Negotiation

Netscape Enterprise Web Server

Yes, the Enterprise International Server edition supports multiple languages with content directories. Also LDAP may be populated with UTF-8 data, and server side javascript and the search feature can have a default language specified.

When a request for this page is made by a browser with an accept language request header for: http://www.someplace.com/somepage.html the server translates that to the following URL: http://www.someplace.com/xx/somepage.html, where xx is a language code.

If not found, the default page is served.

The magnus.conf directives that control this are:

ClientLanguage (en|fr|de|etc.) for client error messages like "Not Found" or "Access Denied"
DefaultLanguage(en|fr|de|etc.) for other error messages
AcceptLanguage (on|off)

Microsoft IIS

*** TBA ***

Q29. What format do I send strings to the translator?

A29. Ask the translator for the recommended string file format before sending strings out for translation. They may have a specific file format in mind for use with translation memory tools. Some use Microsoft Word and prefer their import format for creating a table.

If there's no preference (or you just can't wait), here's an acceptable text format that is both human and machine readable:

project:
copyright:
date:
version:
charset:

id: MESSAGE_ID1
en: english text
xx: translated text
note: Notes.
-
id: MESSAGE_ID2
en: english text
xx: translated text
note: Notes.
-

Here's a first pass at an XML string resource format that should be compatible with current translation tools. See http://www.opentag.com/ for more details.

<?xml version="1.0" encoding="UTF-8"?>
<strings>
   <version>1.0</version>
   <string id="test">
      <en>test</en>
      <fr>l'examen</fr>
      <note>noun</note>
   </string>
</strings>

Here's a parser:


use XML::Parser;
use XML::Simple;

my $file = './strings.xml';
my $xs1 = XML::Simple->new();

my $doc = $xs1->XMLin($file);

foreach my $key (keys (%{$doc->{string}})){
   print "string=$key\n";
   print "   en=", $doc->{string}->{$key}{en}, "\n";
   print "   fr=", $doc->{string}->{$key}{fr}, "\n";
   print "   note=", $doc->{string}->{$key}{note}, "\n";
}

<?xml version="1.0" encoding="UTF-8"?>
<strings>
   <version>1.0</version>
   <string id="test" xml:lang="en">test</string>
   <string id="test" xml:lang="fr">l'examen</string>
</strings>

A30. Both transfer and character set encodings must be considered.

SMTP is a 7-bit protocol that supports US-ASCII. To support 8-bit data over 7-bits, many transfer encodings were developed:

• quoted-printable - quoted-printable encodes bytes greater than 127 as hex values. It is mostly human-readable for Western European languages.
• base64 - base64 encodes 3 bytes as 4 bytes.
• uuencode - similar to base64, but with control information. See man uuencode.
• binhex - Macintosh format
• xxencode - obsolete alternative to uuencode, very similar algorithm.

Mail may be mangled such that multiple transfer encodings are used simultaneously (most commonly observed in webmail products).

Also, MIME encoding is often used now.

Here are standard and common character set encodings used in email:

• English: US-ASCII
• Latin languages: ISO-8859-1 or Windows code pages
• Non-latin languages: ISO-8859-x or Windows code pages
• Japanese: Standard - ISO-2022-JP, Common in US - S-JIS, Common in US but not recommended - EUC-JP
• Korean: Standard - ISO-2022-KR and EUC-KR - Korean mail encodings are described here
• Simplified Chinese (Mainland PRC China, Singapore): GB2312
• Traditional Chinese (Taiwan, Hong Kong): Big5, EUC-TW

MIME allows multiple attachments, each in a different character set encoding.

What complicates email replies is that previous text is quoted in the body of the email. The quoted text is in a specific character set encoding that may not be what your MTA or MUA prefer, so transcoding may need to be done. An example would be replying to email in EUC-JP when you prefer S-JIS.

Here are some comments from a Japanese email user:

If you write Japanese email, it's strongly recommended
to use ISO-2022-JP. The contemporary way to send japanese email is,

  * Mime-Version is 1.0,
  * Content-Type is Text/Plain; charset=iso-2022-jp,
  * Content-Transfer-Encoding is 7bit (not Base64 nor quoted-printable)
   (some people/mailer don't attach CTE header. I don't know
    which is better.)

In early days, Japanese use ISO-2022-JP without MIME header. But
now, it's not recommended.

You should not use Shift_JIS and EUC-JP, even though RFC permit
to use these encodings, because there are many mailers in japan
not to handle these encodings still now.
In (near?) future, we may use UTF-8, but we should not use it now.

UTF-8 is not commonly used yet, but here's a sample header:

MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Mailer: Becky! ver. 2.00 (beta 10)
Content-Type: text/plain; charset="UTF-8"

Internet Mail Consortium: Implementing Internationalization in Internet Mail
GNU UUDeview and UUEnview Tools
Decoding Internet Attachments - A Tutorial

Q30b. What is happening with internationalized DNS?

A30b. The goal was to have a standard for hosting non-English domain names in local scripts by the end of 2000. For example, if a user wants to use Chinese ideograms as a domain name, that will be supported by both DNS resolvers and servers.

This will be accomplished by internationalizing the domain names, but not DNS itself, by storing the Unicode name as an encoded ASCII sequence in DNS. An example of an encoded ASCII sequence from the RACE algorithm of Japanese text is bq--gbqxg.com

The complications with internationalized DNS are:

• DNS is a critical Internet service that has to work reliably and uniquely
• BIND is a legacy product that is only slowly upgraded by hostmasters (if at all)
• some machines are still using DNS in ROM!
• hostmasters want human-readable (and English, too) logs generated for trouble-shooting.

Right now care must be taken when implementing iDNS since most the of standards documents are still in draft.

IETF IDN WG
draft-ietf-idn-race-03.txt - RACE: Row-based ASCII Compatible Encoding for IDN
Perl Sample Script for draft-ietf-idn-race-02.txt    Perl Module for draft-ietf-idn-race-02.txt

draft-ietf-idn-lace-01.txt - LACE: Length-based ASCII Compatible Encoding for IDN

Convert::RACE and Convert::LACE Perl modules by Tatsuhiko Miyagawa
Internet Mail Consortium nameprep test tool
i-DNS.net International
Center for Internet Research (CIRC) iDNS (Obsolete)

Q30c. How do I manage timezones in Perl?

Adding timezone support to an application your first time is a challenge.

Timezone boundaries are in fact dictated by law and politics, not geography. This means that a program needs a table by date and political entity (city or county level) that is updated on a monthly basis with GMT offsets to be of any use.

The generally accepted practice nowadays is to store the users' timezone setting preference as Region/City.

The city level is a fine enough granularity to define timezones accurately (so far). Of note is that city timezone policies are less subject to change than country borders, especially in politically fluid regions like Africa. Legacy alphabetic (short) identifiers like 'PST' are deprecated, or translated from and to Region/City as needed. Storing EST as a preference is obviously wrong for half the year, and am ambiguous, and storing a GMT offset is even worse since then you're not sure what the timezone is at all really.

Perl support for timezones has strengthened recently.

The most appropriate module for new applications is Perl Wrappers for ICU. It supports both Region/City and legacy short identifiers.

For legacy alphabetic identifier only support, Date::Manip and Time::Zone are available on CPAN.

Olson tz data (used in most Unix implementations) and other documents ftp://elsie.nci.nih.gov/pub/

A site that has interesting maps and info on timezones: http://www.worldtimezone.com/

More information: http://www.twinsun.com/tz/tz-link.htm

Just about any Unix install program these days uses a Region/City picker.

The Netmax Firewall product is notable in being a mod_perl application with very sophisticated timezone support based on a graphical image world map picker for Region/City.

A30c. Date::Manip can do timezone conversions between Standard timezones, not Daylight Savings timezones.

(If you are doing client-server programming, and the server stores dates in a common timezone (like PST or GMT), and the server's clock is set to Standard or Daylight time, and all your users also obey Standard or Daylight time, then Date::Manip will give you the correct timezone conversions. Otherwise the time will be offset 1 hour.)

ICU/picu is probably your best bet to do more flexible timezone conversions.

A Summary of the International Standard Date and Time Notation
Military World Time Zone Map

Q31. Any good references?

A31. Here's a list:

• perldoc perlunicode in perl 5.6+
• perldoc perllocale
• CJKV Information Processing, Ken Lunde, O'Reilly & Associates (Errata)
• The Unicode Standard : Version 3.0, The Unicode Consortium
• A Practical Guide to Software Localization, Bert Esselink, 1998
• Vancouver Web Pages is a site with cool stuff and has applied material on localization, translation memory tools and project management.
  • How to make a Multilingual Web server http://vancouver-webpages.com/multilingual/howto.html
  • Multilingual Resources http://vancouver-webpages.com/multilingual/resources.shtml
• IETF Draft - Internationalization of the Hypertext Markup Language
• Jim Breen's Japanese Page
• Ken Lunde's Home Page
• W3C - Internationalization / Localization Non-western Character sets, Languages, and Writing Systems
• W3C - Charlint - A Character Normalization Tool
• Notes on Internationalization by by A.J.Flavell
• View Perl5-Porters archives
• Subscribe to Perl-Unicode mailing list
• honyaku@onelist.com Japanese translation mailing list
  Archive
• nihongo-computing@msdi.co.jp Japanese computing list
  Archive
• The ASCII Consortium : Official Website (Humor)
• ON THE STATUS OF THE LATIN LETTER ÞORN AND OF ITS SORTING ORDER (More Humor?)
• Character Set Considered Harmful
• Unicode: A Primer, Tony Graham (Chapter Index)
• SimulTrans Book Club

Miscellaneous References

Ken Lunde has written a character set conversion C tool called jconv to convert Japanese JIS, SJIS and EUC encodings. Typical usage is
jconv -is -oe <sjis.txt >euc.txt

Some other transcoding utilities are Laux's Kakitori, Ichikawa's NKF (there is a NKF Perl wrapper also), Sato's QKC, Yano's RXJIS.

iconv can also be used (on Unix systems): iconv -f UJIS -t ISO-2022-JP JIS.txt

Viewing and Inputting Japanese Text in Windows
Ruby and Perl Code
Communicator 4.7 Release Notes - International
Japanese Email Software
ISO C Standard

If you are looking for tools for chinese, you will find many things here
http://www.mandarintools.com/
and here for japanese :
 http://ftp.cc.monash.edu.au/pub/nihongo/00INDEX.html

This free editor for Windows (95/98/NT/CE) is able to handle many different
encodings for japanese including EUC-JP and UTF-8.
 http://www.physics.ucla.edu/~grosenth/jwpce.html
Unicode resources on the net.


IBM Classes for Unicode (ICU), ex-Taligent, available in C++ and Java

http://czyborra.com/unicode/ucoverage lists the coverage of an *-iso10646-1 BDF font or a Unicode mapping file by Roman csyborra

The 19 × 21 × 28 = 11'172 Hangul syllables are all that are needed for Modern Hangul.

#!/usr/local/bin/perl
foreach $choseong (split(/;/,
"G;GG;N;D;DD;L;M;B;BB;S;SS;;J;JJ;C;K;T;P;H"))
{
  foreach $jungseong (split(/;/,
  "A;AE;YA;YAE;EO;E;YEO;YE;O;WA;WAE;OE;YO;U;WEO;WE;WI;YU;EU;YI;I"))
  {
    foreach $jongseong (split(/;/,
    ";G;GG;GS;N;NJ;NH;D;L;LG;LM;LB;LS;LT;LP;LH;M;B;BS;S;SS;NG;J;C;K;T;P;H"))
    {
      printf("U+%X:%s\n", 0xAC00 + $i++,
             "HANGUL SYLLABLE $choseong$jungseong$jongseong");
} } }

Q32. How do I convert US-ASCII to UTF-16 on Windows NT?

A32. Here's Peter Prymmer's code to convert US-ASCII to UTF-16 on Windows NT:

$unicode_string = "\0" . join("\0",split(//,$ascii_string));

Q33. How do I transform the name of a character encoding to the MIME charset name?

A33. Here's code to transform name of a encoding to the MIME charset name:

#!/usr/bin/perl
# Read the name of a character encoding from stdin and transform it into
# the corresponding standardized MIME charset name, as registered on
# (or pipelined for) ftp://ftp.isi.edu/in-notes/iana/assignments/character-sets
# Markus Kuhn  -- 2000-05-24

while () {
    tr/a-z/A-Z/;
    if (/8859[-_\/](\d+)(:.*)?$/) {
	print "ISO-8859-$1\n";
    } elsif (/LATIN[-_\/]?([1-4])$/) {
	print "ISO-8859-$1\n";
    } elsif (/LATIN[-_\/]?CYRILLIC/) {
	print "ISO-8859-5\n";
    } elsif (/LATIN[-_\/]?ARABIC/) {
	print "ISO-8859-6\n";
    } elsif (/LATIN[-_\/]?GREEK/) {
	print "ISO-8859-7\n";
    } elsif (/LATIN[-_\/]?HEBREW/) {
	print "ISO-8859-8\n";
    } elsif (/LATIN[-_\/]?5$/) {
	print "ISO-8859-9\n";
    } elsif (/LATIN[-_\/]?6$/) {
	print "ISO-8859-10\n";
    } elsif (/LATIN[-_\/]?7$/) {
	print "ISO-8859-13\n";
    } elsif (/LATIN[-_\/]?8$/) {
	print "ISO-8859-14\n";
    } elsif (/LATIN[-_\/]?9$/) {
	print "ISO-8859-15\n";
    } elsif (/LATIN[-_\/]?10$/) {
	print "ISO-8859-16\n";
    } elsif (/UTF[-_\/]?8/ || /UTF$/) {
	print "UTF-8\n";
    } elsif (/(WINDOWS|WIN|CP|DOS|IBM|MSDOS)[-_\/]?(\d+)$/) {
	print "windows-$2\n";
    } elsif (/ASCII/ || /X3\.4/ || /[^\d]646[\.-_\/]?IRV/) {
	print "US-ASCII\n";
    } elsif (/2022[-_\/]?([A-Z\d-]+)(:.*)?$/) {
	print "ISO-2022-$1\n";
    } elsif (/SHIFT[-_\/]?JIS/) {
	print "Shift_JIS\n";
    } elsif (/BIG[-_\/]?5$/) {
	print "Big5\n";
    } else {
	print;
    }
}