Internationalized Domain Names

Like much of the early infrastructure that today's Internet relies on, the domain name system was not conceived with the world's writing systems in mind. It was limited to the basic English alphabet, and its design reflects that. Since then, like other core technologies, it has been retrofitted to support new writing systems, and that support is now pervasive.

The mechanism is called Internationalized Domain Names in Applications (IDNA), and it allows internationalized domain names (IDNs) to be placed in the domain name system. IDNs remove the English-only limit, permitting domain names in essentially any written language — in scripts such as Arabic, Chinese, Cyrillic and Greek. IDNA builds on the Unicode standard, which provides a universal way of encoding the world's writing systems.

How it looks

At its heart, the DNS still supports only the limited 26-letter alphabet used for English. Rather than change this, which would involve effectively updating all DNS software in the world, IDNA works by having applications transform internationalized labels into a special ASCII form before they are looked up. A label like bücher becomes xn--bcher-kva, converted using an algorithm called Punycode, with the xn-- prefix marking it as an internationalized label. The DNS infrastructure carries only the ASCII form; software that understands IDNs converts between the two so users see the native script.

Making IDNs safe

Punycode solves the encoding problem, but it does not make the whole of Unicode safe to use in domain names as-is. Unicode contains many characters that look alike — sometimes similar, sometimes identical — and these could be used to create domain names that are visually indistinguishable yet resolve differently.

To successfully deploy IDNs we address this through several complementary strategies. The first is intrinsic to the IDNA standard: it limits the code points that may appear in a label to those actually used to write language, so symbols and punctuation are excluded by the core specification. Similarly, commingling writing forms that never appear together in real-life are prohibited. This is particularly important as some languages use virtually identical visual forms to represent different letters:

These limitations help reduce a whole class of security risks and sources of confusion.

Software Support

Most major programming languages have good support for IDNA. For Python, I've helped maintain an idna library, which keeps pace with the most recent versions of the technical standards, including the annual cadence of Unicode updates. The library has worked its way into the foundations of the Python ecosystem as a dependency of popular HTTP libraries — a large installed base that means IDNs can be used confidently with wide support.

Try it yourself

Here is a converter that runs the same code as the Python implementation, but in your web browser. Enter a domain name in any script, or in ASCII form beginning with xn--, and it is encoded or decoded as you type.

Adding rules tailored to specific languages

IDNA's ability to limit code points is not a complete solution to the security and usability challenges that IDNs present. In practice, IDN deployment is augmented with label generation rulesets (LGRs), a way to precisely identify which code points are needed to represent a given language or writing system, and denote the potential confusables within it. LGRs let registries offer domains suited to a specific community without opening the door to misleading look-alikes.

LGRs are also helpful for languages that have characters that are interchangeable in practice — the simplified and traditional Chinese forms of the same word. A registry will use LGRs as a tool to work out what can be registered within their zone, and when two or more labels should be treated as variants of one another.

LGR are machine-readable instruction sets that specifies the permitted repertoire for a zone, the variant relationships between characters, and the rules for how labels and their variants are handled. The work to define LGRs came from our experiences in IANA in handling their predecessor IDN tables, and resulted in RFC 7940.

For the root zone, panels of script community experts have developed rulesets for each writing system, integrated into a single Root Zone LGR that is used to evaluate proposed internationalized top-level domains. Individual registries publish their own rulesets for the zones they operate, many of which are collected in the IANA repository of IDN practices. Because the rules are increasingly expressed in a common machine-readable format, they can be reviewed, compared and applied consistently — allowing for automation and predictable outcomes in how these issues are handled.

* * *