Misplaced Pages

#488511

76-500: The limit of 17 planes is due to UTF-16 , which can encode 2 code points (16 planes) as pairs of words , plus the BMP as a single word. UTF-8 was designed with a much larger limit of 2 (2,147,483,648) code points (32,768 planes), and would still be able to encode 2 (2,097,152) code points (32 planes) even under the current limit of 4 bytes . The 17 planes can accommodate 1,114,112 code points. Of these, 2,048 are surrogates (used to make

152-1087: A pair of 16- bit codes: one High Surrogate and one Low Surrogate. A single surrogate code point will never be assigned a character. 65,520 of the 65,536 code points in this plane have been allocated to a Unicode block, leaving just 16 code points in a single unallocated range (2FE0..2FEF). As of Unicode 16.0, the BMP comprises the following 164 blocks: Plane 1 , the Supplementary Multilingual Plane ( SMP ), contains historic scripts (except CJK ideographic), and symbols and notation used within certain fields. Scripts include Linear B , Egyptian hieroglyphs , and cuneiform scripts. It also includes English reform orthographies like Shavian and Deseret , and some modern scripts like Osage , Warang Citi , Adlam , Wancho and Toto . Symbols and notations include historic and modern musical notation ; mathematical alphanumerics ; shorthands; Emoji and other pictographic sets; and game symbols for playing cards , mahjong , and dominoes . As of Unicode 16.0,

228-498: A block is always a multiple of 16, and is often a multiple of 128, but is otherwise arbitrary. Characters required for a given script may be spread out over several different, potentially disjunct blocks within the codespace. Each code point is assigned a classification, listed as the code point's General Category property. Here, at the uppermost level code points are categorized as one of Letter, Mark, Number, Punctuation, Symbol, Separator, or Other. Under each category, each code point

304-727: A calendar year and with rare cases where the scheduled release had to be postponed. For instance, in April 2020, a month after version 13.0 was published, the Unicode Consortium announced they had changed the intended release date for version 14.0, pushing it back six months to September 2021 due to the COVID-19 pandemic . Unicode 16.0, the latest version, was released on 10 September 2024. It added 5,185 characters and seven new scripts: Garay , Gurung Khema , Kirat Rai , Ol Onal , Sunuwar , Todhri , and Tulu-Tigalari . Thus far,

380-411: A code unit starts a character can be determined without examining earlier code units (i.e. the type of code unit can be determined by the ranges of values in which it falls). UTF-8 shares these advantages, but many earlier multi-byte encoding schemes (such as Shift JIS and other Asian multi-byte encodings) did not allow unambiguous searching and could only be synchronized by re-parsing from the start of

456-432: A comprehensive catalog of character properties, including those needed for supporting bidirectional text , as well as visual charts and reference data sets to aid implementers. Previously, The Unicode Standard was sold as a print volume containing the complete core specification, standard annexes, and code charts. However, version 5.0, published in 2006, was the last version printed this way. Starting with version 5.2, only

532-575: A full semantic duplicate of the Latin alphabet, because legacy CJK encodings contained both "fullwidth" (matching the width of CJK characters) and "halfwidth" (matching ordinary Latin script) characters. The Unicode Bulldog Award is given to people deemed to be influential in Unicode's development, with recipients including Tatsuo Kobayashi , Thomas Milo, Roozbeh Pournader , Ken Lunde , and Michael Everson . The origins of Unicode can be traced back to

608-429: A handful of scripts—often primarily between a given script and Latin characters —not between a large number of scripts, and not with all of the scripts supported being treated in a consistent manner. The philosophy that underpins Unicode seeks to encode the underlying characters— graphemes and grapheme-like units—rather than graphical distinctions considered mere variant glyphs thereof, that are instead best handled by

684-503: A hint to perform byte-swapping for the remaining values. If the BOM is missing, RFC 2781 recommends that big-endian (BE) encoding be assumed. In practice, due to Windows using little-endian (LE) order by default, many applications assume little-endian encoding. It is also reliable to detect endianness by looking for null bytes, on the assumption that characters less than U+0100 are very common. If more even bytes (starting at 0) are null, then it

760-467: A larger 31-bit space and an encoding ( UCS-4 ) that would require 4 bytes per character. This was resisted by the Unicode Consortium , both because 4 bytes per character wasted a lot of memory and disk space, and because some manufacturers were already heavily invested in 2-byte-per-character technology. The UTF-16 encoding scheme was developed as a compromise and introduced with version 2.0 of

836-534: A low-surrogate code point forms a surrogate pair in UTF-16 in order to represent code points greater than U+FFFF . In principle, these code points cannot otherwise be used, though in practice this rule is often ignored, especially when not using UTF-16. A small set of code points are guaranteed never to be assigned to characters, although third-parties may make independent use of them at their discretion. There are 66 of these noncharacters : U+FDD0 – U+FDEF and

SECTION 10

#1732765031489

912-568: A mix of UTF-16, UTF-8, and legacy byte encodings. While there's been some UTF-8 support for even Windows XP, it was improved (in particular the ability to name a file using UTF-8) in Windows 10 insider build 17035 and the May 2019 update. As of May 2019, Microsoft recommends software use UTF-8 , on Windows and Xbox , instead of other 8-bit encodings. It is unclear if they are recommending usage of UTF-8 over UTF-16, though they do state "UTF-16 [..]

988-535: A project run by Deborah Anderson at the University of California, Berkeley was founded in 2002 with the goal of funding proposals for scripts not yet encoded in the standard. The project has become a major source of proposed additions to the standard in recent years. The Unicode Consortium together with the ISO have developed a shared repertoire following the initial publication of The Unicode Standard : Unicode and

1064-399: A properly engineered design, 16 bits per character are more than sufficient for this purpose. This design decision was made based on the assumption that only scripts and characters in "modern" use would require encoding: Unicode gives higher priority to ensuring utility for the future than to preserving past antiquities. Unicode aims in the first instance at the characters published in

1140-558: A total of 168 scripts are included in the latest version of Unicode (covering alphabets , abugidas and syllabaries ), although there are still scripts that are not yet encoded, particularly those mainly used in historical, liturgical, and academic contexts. Further additions of characters to the already encoded scripts, as well as symbols, in particular for mathematics and music (in the form of notes and rhythmic symbols), also occur. The Unicode Roadmap Committee ( Michael Everson , Rick McGowan, Ken Whistler, V.S. Umamaheswaran) maintain

1216-654: A universal encoding than the original Unicode architecture envisioned. Version 1.0 of Microsoft's TrueType specification, published in 1992, used the name "Apple Unicode" instead of "Unicode" for the Platform ID in the naming table. The Unicode Consortium is a nonprofit organization that coordinates Unicode's development. Full members include most of the main computer software and hardware companies (and few others) with any interest in text-processing standards, including Adobe , Apple , Google , IBM , Meta (previously as Facebook), Microsoft , Netflix , and SAP . Over

1292-574: Is "constructed from a pair of Unicode scalar values" (and those values are outside the BMP and require 4 bytes each). UTF-16 in no way assists in "counting characters" or in "measuring the width of a string". UTF-16 is often claimed to be more space-efficient than UTF-8 for East Asian languages, since it uses two bytes for characters that take 3 bytes in UTF-8. Since real text contains many spaces, numbers, punctuation, markup (for e.g. web pages), and control characters, which take only one byte in UTF-8, this

1368-500: Is a character encoding method capable of encoding all 1,112,064 valid code points of Unicode. The encoding is variable-length as code points are encoded with one or two 16-bit code units . UTF-16 arose from an earlier obsolete fixed-width 16-bit encoding now known as UCS-2 (for 2-byte Universal Character Set), once it became clear that more than 2 (65,536) code points were needed, including most emoji and important CJK characters such as for personal and place names. UTF-16

1444-415: Is a text encoding standard maintained by the Unicode Consortium designed to support the use of text in all of the world's writing systems that can be digitized. Version 16.0 of the standard defines 154 998 characters and 168 scripts used in various ordinary, literary, academic, and technical contexts. Many common characters, including numerals, punctuation, and other symbols, are unified within

1520-733: Is a unique burden that Windows places on code that targets multiple platforms." The IBM i operating system designates CCSID ( code page ) 13488 for UCS-2 encoding and CCSID 1200 for UTF-16 encoding, though the system treats them both as UTF-16. UTF-16 is used by the Qualcomm BREW operating systems; the .NET environments; and the Qt cross-platform graphical widget toolkit . Symbian OS used in Nokia S60 handsets and Sony Ericsson UIQ handsets uses UCS-2. iPhone handsets use UTF-16 for Short Message Service instead of UCS-2 described in

1596-484: Is big-endian. The standard also allows the byte order to be stated explicitly by specifying UTF-16BE or UTF-16LE as the encoding type. When the byte order is specified explicitly this way, a BOM is specifically not supposed to be prepended to the text, and a U+FEFF at the beginning should be handled as a ZWNBSP character. Most applications ignore a BOM in all cases despite this rule. For Internet protocols, IANA has approved "UTF-16", "UTF-16BE", and "UTF-16LE" as

SECTION 20

#1732765031489

1672-694: Is encoded either as one or two 16-bit code units . Code points less than 2 ("in the BMP") are encoded with a single 16-bit code unit equal to the numerical value of the code point, as in the older UCS-2. Code points greater than or equal to 2 ("above the BMP") are encoded using two 16-bit code units. These two 16-bit code units are chosen from the UTF-16 surrogate range 0xD800–0xDFFF which had not previously been assigned to characters. Values in this range are not used as characters, and UTF-16 provides no legal way to code them as individual code points. A UTF-16 stream, therefore, consists of single 16-bit codes outside

1748-413: Is intended to suggest a unique, unified, universal encoding". In this document, entitled Unicode 88 , Becker outlined a scheme using 16-bit characters: Unicode is intended to address the need for a workable, reliable world text encoding. Unicode could be roughly described as "wide-body ASCII " that has been stretched to 16 bits to encompass the characters of all the world's living languages. In

1824-428: Is more than just a repertoire within which characters are assigned. To aid developers and designers, the standard also provides charts and reference data, as well as annexes explaining concepts germane to various scripts, providing guidance for their implementation. Topics covered by these annexes include character normalization , character composition and decomposition, collation , and directionality . Unicode text

1900-405: Is not padded. There are a total of 2 + (2 − 2 ) = 1 112 064 valid code points within the codespace. (This number arises from the limitations of the UTF-16 character encoding, which can encode the 2 code points in the range U+0000 through U+FFFF except for the 2 code points in the range U+D800 through U+DFFF , which are used as surrogate pairs to encode the 2 code points in

1976-510: Is only true for artificially constructed dense blocks of text. A more serious claim can be made for Devanagari and Bengali , which use multi-letter words and all the letters take 3 bytes in UTF-8 and only 2 in UTF-16. In addition the Chinese Unicode encoding standard GB 18030 always produces files the same size or smaller than UTF-16 for all languages, not just for Chinese (it does this by sacrificing self-synchronization). UTF-16

2052-417: Is processed and stored as binary data using one of several encodings , which define how to translate the standard's abstracted codes for characters into sequences of bytes. The Unicode Standard itself defines three encodings: UTF-8 , UTF-16 , and UTF-32 , though several others exist. Of these, UTF-8 is the most widely used by a large margin, in part due to its backwards-compatibility with ASCII . Unicode

2128-480: Is projected to include 4301 new unified CJK characters . The Unicode Standard defines a codespace : a sequence of integers called code points in the range from 0 to 1 114 111 , notated according to the standard as U+0000 – U+10FFFF . The codespace is a systematic, architecture-independent representation of The Unicode Standard ; actual text is processed as binary data via one of several Unicode encodings, such as UTF-8 . In this normative notation,

2204-410: Is rarely tested), has led to many bugs in software, including in Windows itself, the solution is usually adopting UTF-8 , as most software has done including (partially) Windows itself and Java and JavaScript. In the late 1980s, work began on developing a uniform encoding for a "Universal Character Set" ( UCS ) that would replace earlier language-specific encodings with one coordinated system. The goal

2280-415: Is still used. JavaScript may use UCS-2 or UTF-16. As of ES2015, string methods and regular expression flags have been added to the language that permit handling strings from an encoding-agnostic perspective. UEFI uses UTF-16 to encode strings by default. Swift , Apple's preferred application language, used UTF-16 to store strings until version 5 which switched to UTF-8. Quite a few languages make

2356-400: Is then further subcategorized. In most cases, other properties must be used to adequately describe all the characteristics of any given code point. The 1024 points in the range U+D800 – U+DBFF are known as high-surrogate code points, and code points in the range U+DC00 – U+DFFF ( 1024 code points) are known as low-surrogate code points. A high-surrogate code point followed by

Plane (Unicode) - Misplaced Pages Continue

2432-531: Is used by systems such as the Microsoft Windows API , the Java programming language and JavaScript /ECMAScript. It is also sometimes used for plain text and word-processing data files on Microsoft Windows. It is used by more modern implementations of SMS . UTF-16 is the only encoding (still) allowed on the web that is incompatible with 8-bit ASCII . However it has never gained popularity on

2508-478: Is used for text in the OS ; API of all currently supported versions of Microsoft Windows (and including at least all since Windows CE / 2000 / XP / 2003 / Vista / 7 ) including Windows 10 . In Windows XP, no code point above U+FFFF is included in any font delivered with Windows for European languages. Older Windows NT systems (prior to Windows 2000) only support UCS-2 . Files and network data tend to be

2584-457: The 3GPP TS 23.038 ( GSM ) and IS-637 ( CDMA ) standards. The Joliet file system , used in CD-ROM media, encodes file names using UCS-2BE (up to sixty-four Unicode characters per file name). Python version 2.0 officially only used UCS-2 internally, but the UTF-8 decoder to "Unicode" produced correct UTF-16. There was also the ability to compile Python so that it used UTF-32 internally, this

2660-588: The Basic Multilingual Plane ( BMP ), contains characters for almost all modern languages, and a large number of symbols . A primary objective for the BMP is to support the unification of prior character sets as well as characters for writing . Most of the assigned code points in the BMP are used to encode Chinese, Japanese, and Korean ( CJK ) characters. The High Surrogate ( U+D800–U+DBFF ) and Low Surrogate ( U+DC00–U+DFFF ) codes are reserved for encoding non-BMP characters in UTF-16 by using

2736-585: The Supplementary Special-purpose Plane ( SSP ). It comprises the following two blocks , as of Unicode 16.0: The two planes 15 and 16 (planes F and 10 in hexadecimal) each contain a " Private Use Area ". They contain blocks named Supplementary Private Use Area-A ( PUA-A ) and -B ( PUA-B ). The Private Use Areas are available for use by parties outside ISO and Unicode (private use character encoding). UTF-16 UTF-16 ( 16-bit Unicode Transformation Format)

2812-409: The Unicode Consortium , the latter representing mostly manufacturers of computing equipment. The two groups attempted to synchronize their character assignments so that the developing encodings would be mutually compatible. The early 2-byte encoding was originally called "Unicode", but is now called "UCS-2". When it became increasingly clear that 2 characters would not suffice, IEEE introduced

2888-401: The high surrogates ( 0xD800–0xDBFF ), low surrogates ( 0xDC00–0xDFFF ), and valid BMP characters (0x0000–0xD7FF, 0xE000–0xFFFF) are disjoint , it is not possible for a surrogate to match a BMP character, or for two adjacent code units to look like a legal surrogate pair . This simplifies searches a great deal. It also means that UTF-16 is self-synchronizing on 16-bit words: whether

2964-571: The typeface , through the use of markup , or by some other means. In particularly complex cases, such as the treatment of orthographical variants in Han characters , there is considerable disagreement regarding which differences justify their own encodings, and which are only graphical variants of other characters. At the most abstract level, Unicode assigns a unique number called a code point to each character. Many issues of visual representation—including size, shape, and style—are intended to be up to

3040-578: The 1980s, to a group of individuals with connections to Xerox 's Character Code Standard (XCCS). In 1987, Xerox employee Joe Becker , along with Apple employees Lee Collins and Mark Davis , started investigating the practicalities of creating a universal character set. With additional input from Peter Fenwick and Dave Opstad , Becker published a draft proposal for an "international/multilingual text character encoding system in August 1988, tentatively called Unicode". He explained that "the name 'Unicode'

3116-567: The ISO's Universal Coded Character Set (UCS) use identical character names and code points. However, the Unicode versions do differ from their ISO equivalents in two significant ways. While the UCS is a simple character map, Unicode specifies the rules, algorithms, and properties necessary to achieve interoperability between different platforms and languages. Thus, The Unicode Standard includes more information, covering in-depth topics such as bitwise encoding, collation , and rendering. It also provides

Plane (Unicode) - Misplaced Pages Continue

3192-663: The SMP comprises the following 161 blocks: Plane 2 , the Supplementary Ideographic Plane ( SIP ), is used for CJK Ideographs, mostly CJK Unified Ideographs , that were not included in earlier character encoding standards. As of Unicode 16.0, the SIP comprises the following seven blocks: Plane 3 is the Tertiary Ideographic Plane (TIP). CJK Unified Ideographs Extension G was added to

3268-501: The TIP in Unicode 13.0, released in March 2020. It also is tentatively allocated for Oracle Bone script and Small Seal Script . As of Unicode 16.0, the TIP comprises the following two blocks: Planes 4 to 13 (planes 4 to D in hexadecimal ): No characters have yet been assigned, or proposed for assignment, to Planes 4 through 13. Plane 14 ( E in hexadecimal) is designated as

3344-656: The Unicode standard in July 1996. It is fully specified in RFC 2781, published in 2000 by the IETF . UTF-16 is specified in the latest versions of both the international standard ISO/IEC 10646 and the Unicode Standard. "UCS-2 should now be considered obsolete. It no longer refers to an encoding form in either 10646 or the Unicode Standard." UTF-16 will never be extended to support a larger number of code points or to support

3420-479: The byte order of code units, UTF-16 allows a byte order mark (BOM), a code point with the value U+FEFF, to precede the first actual coded value. (U+FEFF is the invisible zero-width non-breaking space /ZWNBSP character). If the endian architecture of the decoder matches that of the encoder, the decoder detects the 0xFEFF value, but an opposite-endian decoder interprets the BOM as the noncharacter value U+FFFE reserved for this purpose. This incorrect result provides

3496-426: The code point are distributed among the UTF-16 bytes. Additional bits added by the UTF-16 encoding process are shown in black. UTF-16 and UCS-2 produce a sequence of 16-bit code units. Since most communication and storage protocols are defined for bytes, and each unit thus takes two 8-bit bytes, the order of the bytes may depend on the endianness (byte order) of the computer architecture. To assist in recognizing

3572-475: The code points that were replaced by surrogates, as this would violate the Unicode Stability Policy with respect to general category or surrogate code points. (Any scheme that remains a self-synchronizing code would require allocating at least one Basic Multilingual Plane (BMP) code point to start a sequence. Changing the purpose of a code point is disallowed.) Each Unicode code point

3648-496: The core specification, published as a print-on-demand paperback, may be purchased. The full text, on the other hand, is published as a free PDF on the Unicode website. A practical reason for this publication method highlights the second significant difference between the UCS and Unicode—the frequency with which updated versions are released and new characters added. The Unicode Standard has regularly released annual expanded versions, occasionally with more than one version released in

3724-475: The discretion of the software actually rendering the text, such as a web browser or word processor . However, partially with the intent of encouraging rapid adoption, the simplicity of this original model has become somewhat more elaborate over time, and various pragmatic concessions have been made over the course of the standard's development. The first 256 code points mirror the ISO/IEC 8859-1 standard, with

3800-551: The encoding part of the string object, and thus store and support a large set of encodings including UTF-16. Most consider UTF-16 and UCS-2 to be different encodings. Examples are the PHP language and MySQL . A method to determine what encoding a system is using internally is to ask for the "length" of string containing a single non-BMP character. If the length is 2 then UTF-16 is being used. 4 indicates UTF-8. 3 or 6 may indicate CESU-8 . 1 may indicate UTF-32, but more likely indicates

3876-1752: The entirety of planes 15 and 16). For future usage, ranges of characters have been tentatively mapped out for most known current and ancient writing systems. 0000–​0FFF 1000–​1FFF 2000–​2FFF 3000–​3FFF 4000–​4FFF 5000–​5FFF 6000–​6FFF 7000–​7FFF 8000–​8FFF 9000–​9FFF A000–​AFFF B000–​BFFF C000–​CFFF D000–​DFFF E000–​EFFF F000–​FFFF 10000–​10FFF 11000–​11FFF 12000–​12FFF 13000–​13FFF 14000–​14FFF 16000–​16FFF 17000–​17FFF 18000–​18FFF 1A000–​1AFFF 1B000–​1BFFF 1C000–​1CFFF 1D000–​1DFFF 1E000–​1EFFF 1F000–​1FFFF 20000–​20FFF 21000–​21FFF 22000–​22FFF 23000–​23FFF 24000–​24FFF 25000–​25FFF 26000–​26FFF 27000–​27FFF 28000–​28FFF 29000–​29FFF 2A000–​2AFFF 2B000–​2BFFF 2C000–​2CFFF 2D000–​2DFFF 2E000–​2EFFF 2F000–​2FFFF 30000–​30FFF 31000–​31FFF 32000–​32FFF E0000–​E0FFF 15: SPUA-A F0000–​FFFFF 16: SPUA-B 100000–​10FFFF The first plane, plane 0 ,

SECTION 50

#1732765031489

3952-401: The following versions of The Unicode Standard have been published. Update versions, which do not include any changes to character repertoire, are signified by the third number (e.g., "version 4.0.1") and are omitted in the table below. The Unicode Consortium normally releases a new version of The Unicode Standard once a year. Version 17.0, the next major version,

4028-526: The group. By the end of 1990, most of the work of remapping existing standards had been completed, and a final review draft of Unicode was ready. The Unicode Consortium was incorporated in California on 3 January 1991, and the first volume of The Unicode Standard was published that October. The second volume, now adding Han ideographs, was published in June 1992. In 1996, a surrogate character mechanism

4104-562: The intent of trivializing the conversion of text already written in Western European scripts. To preserve the distinctions made by different legacy encodings, therefore allowing for conversion between them and Unicode without any loss of information, many characters nearly identical to others , in both appearance and intended function, were given distinct code points. For example, the Halfwidth and Fullwidth Forms block encompasses

4180-490: The language decodes the string to code points before measuring the "length". In many languages, quoted strings need a new syntax for quoting non-BMP characters, as the C-style "\uXXXX" syntax explicitly limits itself to 4 hex digits. The following examples illustrate the syntax for the non-BMP character U+1D11E 𝄞 MUSICAL SYMBOL G CLEF : Unicode Unicode , formally The Unicode Standard ,

4256-403: The last two code points in each of the 17 planes (e.g. U+FFFE , U+FFFF , U+1FFFE , U+1FFFF , ..., U+10FFFE , U+10FFFF ). The set of noncharacters is stable, and no new noncharacters will ever be defined. Like surrogates, the rule that these cannot be used is often ignored, although the operation of the byte order mark assumes that U+FFFE will never be the first code point in

4332-637: The list of scripts that are candidates or potential candidates for encoding and their tentative code block assignments on the Unicode Roadmap page of the Unicode Consortium website. For some scripts on the Roadmap, such as Jurchen and Khitan large script , encoding proposals have been made and they are working their way through the approval process. For other scripts, such as Numidian and Rongorongo , no proposal has yet been made, and they await agreement on character repertoire and other details from

4408-675: The modern text (e.g. in the union of all newspapers and magazines printed in the world in 1988), whose number is undoubtedly far below 2 = 16,384. Beyond those modern-use characters, all others may be defined to be obsolete or rare; these are better candidates for private-use registration than for congesting the public list of generally useful Unicode. In early 1989, the Unicode working group expanded to include Ken Whistler and Mike Kernaghan of Metaphor, Karen Smith-Yoshimura and Joan Aliprand of Research Libraries Group , and Glenn Wright of Sun Microsystems . In 1990, Michel Suignard and Asmus Freytag of Microsoft and NeXT 's Rick McGowan had also joined

4484-490: The names for these encodings (the names are case insensitive). The aliases UTF_16 or UTF16 may be meaningful in some programming languages or software applications, but they are not standard names in Internet protocols. Similar designations, UCS-2BE and UCS-2LE , are used to show versions of UCS-2 . A "character" may use any number of Unicode code points. For instance an emoji flag character takes 8 bytes, since it

4560-402: The other planes are encoded as two 16-bit code units called a surrogate pair . The first code unit is a high surrogate and the second is a low surrogate (These are also known as "leading" and "trailing" surrogates, respectively, analogous to the leading and trailing bytes of UTF-8. ): Illustrated visually, the distribution of U' between W1 and W2 looks like: Since the ranges for

4636-530: The pairs in UTF-16), 66 are non-characters , and 137,468 are reserved for private use , leaving 974,530 for public assignment. Planes are further subdivided into Unicode blocks , which, unlike planes, do not have a fixed size. The 338 blocks defined in Unicode 16.0 cover 27% of the possible code point space, and range in size from a minimum of 16 code points (sixteen blocks) to a maximum of 65,536 code points (Supplementary Private Use Area-A and -B, which constitute

SECTION 60

#1732765031489

4712-503: The previous environment of a myriad of incompatible character sets , each used within different locales and on different computer architectures. Unicode is used to encode the vast majority of text on the Internet, including most web pages , and relevant Unicode support has become a common consideration in contemporary software development. The Unicode character repertoire is synchronized with ISO/IEC 10646 , each being code-for-code identical with one another. However, The Unicode Standard

4788-651: The range U+10000 through U+10FFFF .) The Unicode codespace is divided into 17 planes , numbered 0 to 16. Plane 0 is the Basic Multilingual Plane (BMP), and contains the most commonly used characters. All code points in the BMP are accessed as a single code unit in UTF-16 encoding and can be encoded in one, two or three bytes in UTF-8. Code points in planes 1 through 16 (the supplementary planes ) are accessed as surrogate pairs in UTF-16 and encoded in four bytes in UTF-8 . Within each plane, characters are allocated within named blocks of related characters. The size of

4864-499: The standard and are not treated as specific to any given writing system. Unicode encodes 3790 emoji , with the continued development thereof conducted by the Consortium as a part of the standard. Moreover, the widespread adoption of Unicode was in large part responsible for the initial popularization of emoji outside of Japan. Unicode is ultimately capable of encoding more than 1.1 million characters. Unicode has largely supplanted

4940-629: The standard states that such arrangements should be treated as encoding errors. It is possible to unambiguously encode an unpaired surrogate (a high surrogate code point not followed by a low one, or a low one not preceded by a high one) in the format of UTF-16 by using a code unit equal to the code point. The result is not valid UTF-16, but the majority of UTF-16 encoder and decoder implementations do this when translating between encodings. To encode U+10437 (𐐷) to UTF-16: To decode U+10437 (𐐷) from UTF-16: The following table summarizes this conversion, as well as others. The colors indicate how bits from

5016-478: The string. UTF-16 is not self-synchronizing if one byte is lost or if traversal starts at a random byte. Because the most commonly used characters are all in the BMP, handling of surrogate pairs is often not thoroughly tested. This leads to persistent bugs and potential security holes, even in popular and well-reviewed application software (e.g. CVE - 2008-2938 , CVE- 2012-2135 ). The official Unicode standard says that no UTF forms, including UTF-16, can encode

5092-446: The surrogate code points. Since these will never be assigned a character, there should be no reason to encode them. However, Windows allows unpaired surrogates in filenames and other places, which generally means they have to be supported by software in spite of their exclusion from the Unicode standard. UCS-2, UTF-8, and UTF-32 can encode these code points in trivial and obvious ways, and a large amount of software does so, even though

5168-613: The surrogate range, and pairs of 16-bit values that are within the surrogate range. Both UTF-16 and UCS-2 encode code points in this range as single 16-bit code units that are numerically equal to the corresponding code points. These code points in the Basic Multilingual Plane (BMP) are the only code points that can be represented in UCS-2. As of Unicode 9.0, some modern non-Latin Asian, Middle-Eastern, and African scripts fall outside this range, as do most emoji characters. Code points from

5244-418: The two-character prefix U+ always precedes a written code point, and the code points themselves are written as hexadecimal numbers. At least four hexadecimal digits are always written, with leading zeros prepended as needed. For example, the code point U+00F7 ÷ DIVISION SIGN is padded with two leading zeros, but U+13254 𓉔 EGYPTIAN HIEROGLYPH O004 ( [REDACTED] )

5320-618: The user communities involved. Some modern invented scripts which have not yet been included in Unicode (e.g., Tengwar ) or which do not qualify for inclusion in Unicode due to lack of real-world use (e.g., Klingon ) are listed in the ConScript Unicode Registry , along with unofficial but widely used Private Use Areas code assignments. There is also a Medieval Unicode Font Initiative focused on special Latin medieval characters. Part of these proposals has been already included in Unicode. The Script Encoding Initiative,

5396-462: The web, where it is declared by under 0.003% of public web pages. UTF-8 , by comparison, accounts for over 98% of all web pages. The Web Hypertext Application Technology Working Group (WHATWG) considers UTF-8 "the mandatory encoding for all [text]" and that for security reasons browser applications should not use UTF-16. The variable length character of UTF-16, combined with the fact that most characters are not variable length (so variable length

5472-640: The years several countries or government agencies have been members of the Unicode Consortium. Presently only the Ministry of Endowments and Religious Affairs (Oman) is a full member with voting rights. The Consortium has the ambitious goal of eventually replacing existing character encoding schemes with Unicode and its standard Unicode Transformation Format (UTF) schemes, as many of the existing schemes are limited in size and scope and are incompatible with multilingual environments. Unicode currently covers most major writing systems in use today. As of 2024 ,

5548-491: Was implemented in Unicode 2.0, so that Unicode was no longer restricted to 16 bits. This increased the Unicode codespace to over a million code points, which allowed for the encoding of many historic scripts, such as Egyptian hieroglyphs , and thousands of rarely used or obsolete characters that had not been anticipated for inclusion in the standard. Among these characters are various rarely used CJK characters—many mainly being used in proper names, making them far more necessary for

5624-483: Was originally designed with the intent of transcending limitations present in all text encodings designed up to that point: each encoding was relied upon for use in its own context, but with no particular expectation of compatibility with any other. Indeed, any two encodings chosen were often totally unworkable when used together, with text encoded in one interpreted as garbage characters by the other. Most encodings had only been designed to facilitate interoperation between

5700-536: Was sometimes done on Unix. Python 3.3 switched internal storage to use one of ISO-8859-1 , UCS-2, or UTF-32 depending on the largest code point in the string. Python 3.12 drops some functionality (for CPython extensions) to make it easier to migrate to UTF-8 for all strings. Java originally used UCS-2, and added UTF-16 supplementary character support in J2SE 5.0 . Recently they have encouraged dumping support for any 8-bit encoding other than UTF-8 but internally UTF-16

5776-426: Was to include all required characters from most of the world's languages, as well as symbols from technical domains such as science, mathematics, and music. The original idea was to replace the typical 256-character encodings, which required 1 byte per character, with an encoding using 65,536 (2 ) values, which would require 2 bytes (16 bits) per character. Two groups worked on this in parallel, ISO/IEC JTC 1/SC 2 and

#488511