31.5. Encodings
Prev | Chapter 31. Platform Independent Extensions | Next

## 31.5. Encodings

### 31.5.1. Introduction

An “encoding” describes the correspondence between `CHARACTER`s and raw bytes during input/output via `STREAM`s with `STREAM-ELEMENT-TYPE` `CHARACTER`.

An `EXT:ENCODING` is an object composed of the following facets:

character set
  This denotes both the set of `CHARACTER`s that can be represented and passed through the I/O channel, and the way these characters translate into raw bytes, i.e., the map between sequences of `CHARACTER` and `(`UNSIGNED-BYTE` 8)` in the form of `STRING`s and `(`VECTOR` (`UNSIGNED-BYTE` 8))` as well as character and byte `STREAM`s. In this context, for example, `CHARSET:UTF-8` and `CHARSET:UCS-4` are considered different, although they can represent the same set of characters.
line terminator mode
  This denotes the way newline characters are represented.

`EXT:ENCODING`s are also `TYPE`s. As such, they represent the set of characters encodable in the character set. In this context, the way characters are translated into raw bytes is ignored, and the line terminator mode is ignored as well. `TYPEP` and `SUBTYPEP` can be used on encodings:

    (SUBTYPEP CHARSET:UTF-8 CHARSET:UTF-16)
    ⇒ T ;
    ⇒ T
    (SUBTYPEP CHARSET:UTF-16 CHARSET:UTF-8)
    ⇒ T ;
    ⇒ T
    (SUBTYPEP CHARSET:ASCII CHARSET:ISO-8859-1)
    ⇒ T ;
    ⇒ T
    (SUBTYPEP CHARSET:ISO-8859-1 CHARSET:ASCII)
    ⇒ NIL ;
    ⇒ T

### 31.5.2. Character Sets

Platform Dependent: Only in CLISP built without compile-time flag UNICODE
  Only one character set is understood: the platform's native (8-bit) character set. See Chapter 13, Characters .
Platform Dependent: Only in CLISP built with compile-time flag UNICODE

  The following character sets are supported, as values of the corresponding (constant) symbol in the “CHARSET” package:

  Symbols in package “CHARSET”

  1. `UCS-2` ≡ `UNICODE-16` ≡ `UNICODE-16-BIG-ENDIAN`, the 16-bit basic multilingual plane of the UNICODE character set. Every character is represented as two bytes.
  2. `UNICODE-16-LITTLE-ENDIAN`
  3. `UCS-4` ≡ `UNICODE-32` ≡ `UNICODE-32-BIG-ENDIAN`, the 21-bit UNICODE character set. Every character is represented as four bytes. This encoding is used by CLISP internally.
  4. `UNICODE-32-LITTLE-ENDIAN`
  5. `UTF-8`, the 21-bit UNICODE character set. Every character is represented as one to four bytes. ASCII characters represent themselves and need one byte per character. Most Latin/Greek/Cyrillic/Hebrew characters need two bytes per character. Most other characters need three bytes per character, and the rarely used remaining characters need four bytes per character. This is therefore, in general, the most space-efficient encoding of all of Unicode.
  6. `UTF-16`, the 21-bit UNICODE character set. Every character in the 16-bit basic multilingual plane is represented as two bytes, and the rarely used remaining characters need four bytes per character. This character set is only available on platforms with GNU libc or GNU libiconv.
  7. `UTF-7`, the 21-bit UNICODE character set. This is a stateful 7-bit encoding. Not all ASCII characters represent themselves. This character set is only available on platforms with GNU libc or GNU libiconv.
  8. `JAVA`, the 21-bit UNICODE character set. ASCII characters represent themselves and need one byte per character. All other characters of the basic multilingual plane are represented by `\u`nnnn`` sequences (`nnnn` a hexadecimal number) and need 6 bytes per character. The remaining characters are represented by `\u`xxxx`\u`yyyy`` and need 12 bytes per character. While this encoding is very comfortable for editing Unicode files using only ASCII-aware tools and editors, it cannot faithfully represent all UNICODE text. Only text which does not contain `\u` (backslash followed by lowercase Latin u) can be faithfully represented by this encoding.
  9. `ASCII`, the well-known US-centric 7-bit character set (American Standard Code for Information Interchange - ASCII).

  10. `ISO-8859-1`, an extension of the ASCII character set, suitable for the Afrikaans, Albanian, Basque, Breton, Catalan, Cornish, Danish, Dutch, English, Færoese, Finnish, French, Frisian, Galician, German, Greenlandic, Icelandic, Irish, Italian, Latin, Luxemburgish, Norwegian, Portuguese, Ræto-Romanic, Scottish, Spanish, and Swedish languages.

      This encoding has the nice property that

          (LOOP :for i :from 0 :to CHAR-CODE-LIMIT :for c = (CODE-CHAR i)
            :always (OR (NOT (TYPEP c CHARSET:ISO-8859-1))
                        (EQUALP (EXT:CONVERT-STRING-TO-BYTES (STRING c) CHARSET:ISO-8859-1)
                                (VECTOR i))))
          ⇒ T

      i.e., it is compatible with CLISP `CODE-CHAR`/`CHAR-CODE` in its own domain.

  11. `ISO-8859-2`, an extension of the ASCII character set, suitable for the Croatian, Czech, German, Hungarian, Polish, Slovak, Slovenian, and Sorbian languages.
  12. `ISO-8859-3`, an extension of the ASCII character set, suitable for the Esperanto and Maltese languages.
  13. `ISO-8859-4`, an extension of the ASCII character set, suitable for the Estonian, Latvian, Lithuanian and Sami (Lappish) languages.
  14. `ISO-8859-5`, an extension of the ASCII character set, suitable for the Bulgarian, Byelorussian, Macedonian, Russian, Serbian, and Ukrainian languages.
  15. `ISO-8859-6`, suitable for the Arabic language.
  16. `ISO-8859-7`, an extension of the ASCII character set, suitable for the Greek language.
  17. `ISO-8859-8`, an extension of the ASCII character set, suitable for the Hebrew language (without punctuation).
  18. `ISO-8859-9`, an extension of the ASCII character set, suitable for the Turkish language.
  19. `ISO-8859-10`, an extension of the ASCII character set, suitable for the Estonian, Icelandic, Inuit (Greenlandic), Latvian, Lithuanian, and Sami (Lappish) languages.
  20. `ISO-8859-13`, an extension of the ASCII character set, suitable for the Estonian, Latvian, Lithuanian, Polish and Sami (Lappish) languages.
  21. `ISO-8859-14`, an extension of the ASCII character set, suitable for the Irish Gælic, Manx Gælic, Scottish Gælic, and Welsh languages.
  22. `ISO-8859-15`, an extension of the ASCII character set, suitable for the ISO-8859-1 languages, with improvements for French, Finnish and the Euro.
  23. `ISO-8859-16` an extension of the ASCII character set, suitable for the Rumanian language.
  24. `KOI8-R`, an extension of the ASCII character set, suitable for the Russian language (very popular, especially on the internet).
  25. `KOI8-U`, an extension of the ASCII character set, suitable for the Ukrainian language (very popular, especially on the internet).
  26. `KOI8-RU`, an extension of the ASCII character set, suitable for the Russian language. This character set is only available on platforms with GNU libiconv.
  27. `JIS_X0201`, a character set for the Japanese language.
  28. `MAC-ARABIC`, a platform specific extension of the ASCII character set.
  29. `MAC-CENTRAL-EUROPE`, a platform specific extension of the ASCII character set.
  30. `MAC-CROATIAN`, a platform specific extension of the ASCII character set.
  31. `MAC-CYRILLIC`, a platform specific extension of the ASCII character set.
  32. `MAC-DINGBAT`, a platform specific character set.
  33. `MAC-GREEK`, a platform specific extension of the ASCII character set.
  34. `MAC-HEBREW`, a platform specific extension of the ASCII character set.
  35. `MAC-ICELAND`, a platform specific extension of the ASCII character set.
  36. `MAC-ROMAN` ≡ `MACINTOSH`, a platform specific extension of the ASCII character set.
  37. `MAC-ROMANIA`, a platform specific extension of the ASCII character set.
  38. `MAC-SYMBOL`, a platform specific character set.
  39. `MAC-THAI`, a platform specific extension of the ASCII character set.
  40. `MAC-TURKISH`, a platform specific extension of the ASCII character set.
  41. `MAC-UKRAINE`, a platform specific extension of the ASCII character set.
  42. `CP437`, a DOS oldie, a platform specific extension of the ASCII character set.
  43. `CP437-IBM`, an IBM variant of `CP437`.
  44. `CP737`, a DOS oldie, a platform specific extension of the ASCII character set, meant to be suitable for the Greek language.
  45. `CP775`, a DOS oldie, a platform specific extension of the ASCII character set, meant to be suitable for some Baltic languages.
  46. `CP850`, a DOS oldie, a platform specific extension of the ASCII character set.
  47. `CP852`, a DOS oldie, a platform specific extension of the ASCII character set.
  48. `CP852-IBM`, an IBM variant of `CP852`.
  49. `CP855`, a DOS oldie, a platform specific extension of the ASCII character set, meant to be suitable for the Russian language.
  50. `CP857`, a DOS oldie, a platform specific extension of the ASCII character set, meant to be suitable for the Turkish language.
  51. `CP860`, a DOS oldie, a platform specific extension of the ASCII character set, meant to be suitable for the Portuguese language.
  52. `CP860-IBM`, an IBM variant of `CP860`.
  53. `CP861`, a DOS oldie, a platform specific extension of the ASCII character set, meant to be suitable for the Icelandic language.
  54. `CP861-IBM`, an IBM variant of `CP861`.
  55. `CP862`, a DOS oldie, a platform specific extension of the ASCII character set, meant to be suitable for the Hebrew language.
  56. `CP862-IBM`, an IBM variant of `CP862`.
  57. `CP863`, a DOS oldie, a platform specific extension of the ASCII character set.
  58. `CP863-IBM`, an IBM variant of `CP863`.
  59. `CP864`, a DOS oldie, meant to be suitable for the Arabic language.
  60. `CP864-IBM`, an IBM variant of `CP864`.
  61. `CP865`, a DOS oldie, a platform specific extension of the ASCII character set, meant to be suitable for some Nordic languages.
  62. `CP865-IBM`, an IBM variant of `CP865`.
  63. `CP866`, a DOS oldie, a platform specific extension of the ASCII character set, meant to be suitable for the Russian language.
  64. `CP869`, a DOS oldie, a platform specific extension of the ASCII character set, meant to be suitable for the Greek language.
  65. `CP869-IBM`, an IBM variant of `CP869`.
  66. `CP874`, a DOS oldie, a platform specific extension of the ASCII character set, meant to be suitable for the Thai language.
  67. `CP874-IBM`, an IBM variant of `CP874`.
  68. `WINDOWS-1250` ≡ `CP1250`, a platform specific extension of the ASCII character set, heavily incompatible with ISO-8859-2.
  69. `WINDOWS-1251` ≡ `CP1251`, a platform specific extension of the ASCII character set, heavily incompatible with ISO-8859-5, meant to be suitable for the Russian language.
  70. `WINDOWS-1252` ≡ `CP1252`, a platform specific extension of the ISO-8859-1 character set.
  71. `WINDOWS-1253` ≡ `CP1253`, a platform specific extension of the ASCII character set, gratuitously incompatible with ISO-8859-7, meant to be suitable for the Greek language.
  72. `WINDOWS-1254` ≡ `CP1254`, a platform specific extension of the ISO-8859-9 character set.
  73. `WINDOWS-1255` ≡ `CP1255`, a platform specific extension of the ASCII character set, gratuitously incompatible with ISO-8859-8, suitable for the Hebrew language. This character set is only available on platforms with GNU libc or GNU libiconv.
  74. `WINDOWS-1256` ≡ `CP1256`, a platform specific extension of the ASCII character set, meant to be suitable for the Arabic language.
  75. `WINDOWS-1257` ≡ `CP1257`, a platform specific extension of the ASCII character set.
  76. `WINDOWS-1258` ≡ `CP1258`, a platform specific extension of the ASCII character set, meant to be suitable for the Vietnamese language. This character set is only available on platforms with GNU libc or GNU libiconv.
  77. `HP-ROMAN8`, a platform specific extension of the ASCII character set.
  78. `NEXTSTEP`, a platform specific extension of the ASCII character set.
  79. `EUC-JP`, a multibyte character set for the Japanese language. This character set is only available on platforms with GNU libc or GNU libiconv.
  80. `SHIFT-JIS`, a multibyte character set for the Japanese language. This character set is only available on platforms with GNU libc or GNU libiconv.
  81. `CP932`, a Microsoft variant of `SHIFT-JIS`. This character set is only available on platforms with GNU libc or GNU libiconv.
  82. `ISO-2022-JP`, a stateful 7-bit multibyte character set for the Japanese language. This character set is only available on platforms with GNU libc or GNU libiconv.
  83. `ISO-2022-JP-2`, a stateful 7-bit multibyte character set for the Japanese language. This character set is only available on platforms with GNU libc 2.3 or newer or GNU libiconv.
  84. `ISO-2022-JP-1`, a stateful 7-bit multibyte character set for the Japanese language. This character set is only available on platforms with GNU libiconv.
  85. `EUC-CN`, a multibyte character set for simplified Chinese. This character set is only available on platforms with GNU libc or GNU libiconv.
  86. `HZ`, a stateful 7-bit multibyte character set for simplified Chinese. This character set is only available on platforms with GNU libiconv.
  87. `GBK`, a multibyte character set for Chinese, This character set is only available on platforms with GNU libc or GNU libiconv.
  88. `CP936`, a Microsoft variant of `GBK`. This character set is only available on platforms with GNU libc or GNU libiconv.
  89. `GB18030`, a multibyte character set for Chinese, This character set is only available on platforms with GNU libc or GNU libiconv.
  90. `EUC-TW`, a multibyte character set for traditional Chinese. This character set is only available on platforms with GNU libc or GNU libiconv.
  91. `BIG5`, a multibyte character set for traditional Chinese. This character set is only available on platforms with GNU libc or GNU libiconv.
  92. `CP950`, a Microsoft variant of `BIG5`. This character set is only available on platforms with GNU libc or GNU libiconv.
  93. `BIG5-HKSCS`, a multibyte character set for traditional Chinese. This character set is only available on platforms with GNU libc or GNU libiconv.
  94. `ISO-2022-CN`, a stateful 7-bit multibyte character set for Chinese. This character set is only available on platforms with GNU libc or GNU libiconv.
  95. `ISO-2022-CN-EXT`, a stateful 7-bit multibyte character set for Chinese. This character set is only available on platforms with GNU libc or GNU libiconv.
  96. `EUC-KR`, a multibyte character set for Korean. This character set is only available on platforms with GNU libc or GNU libiconv.
  97. `CP949`, a Microsoft variant of `EUC-KR`. This character set is only available on platforms with GNU libc or GNU libiconv.
  98. `ISO-2022-KR`, a stateful 7-bit multibyte character set for Korean. This character set is only available on platforms with GNU libc or GNU libiconv.
  99. `JOHAB`, a multibyte character set for Korean used mostly on DOS. This character set is only available on platforms with GNU libc or GNU libiconv.
  100. `ARMSCII-8`, an extension of the ASCII character set, suitable for the Armenian. This character set is only available on platforms with GNU libc or GNU libiconv.
  101. `GEORGIAN-ACADEMY`, an extension of the ASCII character set, suitable for the Georgian. This character set is only available on platforms with GNU libc or GNU libiconv.
  102. `GEORGIAN-PS`, an extension of the ASCII character set, suitable for the Georgian. This character set is only available on platforms with GNU libc or GNU libiconv.
  103. `TIS-620`, an extension of the ASCII character set, suitable for the Thai. This character set is only available on platforms with GNU libc or GNU libiconv.
  104. `MULELAO-1`, an extension of the ASCII character set, suitable for the Laotian. This character set is only available on platforms with GNU libiconv.
  105. `CP1133`, an extension of the ASCII character set, suitable for the Laotian. This character set is only available on platforms with GNU libc or GNU libiconv.
  106. `VISCII`, an extension of the ASCII character set, suitable for the Vietnamese. This character set is only available on platforms with GNU libc or GNU libiconv.
  107. `TCVN`, an extension of the ASCII character set, suitable for the Vietnamese. This character set is only available on platforms with GNU libc or GNU libiconv.

  108. `BASE64`, encodes arbitrary byte sequences with 64 ASCII characters

       ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/

       as specifined by `MIME`; 3 bytes are encoded with 4 characters, line breaks are inserted after every 76 characters.

       While this is not a traditional character set (i.e., it does not map a set of characters in a natural language into bytes), it does define a map between arbitrary byte sequences and certain character sequences, so it falls naturally into the `EXT:ENCODING` class.

  Platform Dependent: Only on GNU systems with GNU libc 2.2 or better and other systems (UNIX and Win32) on which the GNU libiconv C library has been installed

    The character sets provided by the library function `iconv` can also be used as encodings. To create such an encoding, call `EXT:MAKE-ENCODING` with the character set name (a string) as the `:CHARSET` argument.

    When an `EXT:ENCODING` is available both as a built-in and through `iconv`, the built-in is used, because it is more efficient and available across all platforms.

    These encodings are not assigned to global variables, since there is no portable way to get the list of all character sets supported by `iconv`.

    On standard-compliant UNIX systems (e.g., GNU systems, such as GNU/Linux and GNU/Hurd) and on systems with GNU libiconv you get this list by calling the program: iconv -l.

    The reason we use only GNU libc 2.2 or GNU libiconv is that the other `iconv` implementations are broken in various ways and we do not want to deal with random CLISP crashes caused by those bugs. If your system supplies an `iconv` implementation which passes the GNU libiconv's test suite, please report that to clisp-list and a future CLISP version will use `iconv` on your system.

### 31.5.3. Line Terminators

The line terminator mode can be one of the following three keywords:

`:UNIX`
  Newline is represented by the ASCII LF character (U000A).
`:MAC`
  Newline is represented by the ASCII CR character (U000D).
`:DOS`
  Newline is represented by the ASCII CR followed by the ASCII LF.

Windows programs typically use the `:DOS` line terminator, sometimes they also accept `:UNIX` line terminators or produce `:MAC` line terminators.

The HTTP protocol also requires `:DOS` line terminators.

The line terminator mode is relevant only for output (writing to a file/pipe/socket `STREAM`). During input, all three kinds of line terminators are recognized. See also Section 13.10, “Treatment of Newline during Input and Output ”.

### 31.5.4. Function `EXT:MAKE-ENCODING`

The function `(`EXT:MAKE-ENCODING` `&KEY` :CHARSET :LINE-TERMINATOR :INPUT-ERROR-ACTION :OUTPUT-ERROR-ACTION)` returns an `EXT:ENCODING`. The `:CHARSET` argument may be an encoding, a string, or `:DEFAULT`. The possible values for the line terminator argument are the keywords `:UNIX`, `:MAC`, `:DOS`.

The `:INPUT-ERROR-ACTION` argument specifies what happens when an invalid byte sequence is encountered while converting bytes to characters. Its value can be `:ERROR`, `:IGNORE` or a character to be used instead. The UNICODE character #\uFFFD is typically used to indicate an error in the input sequence.

The `:OUTPUT-ERROR-ACTION` argument specifies what happens when an invalid character is encountered while converting characters to bytes. Its value can be `:ERROR`, `:IGNORE`, a byte to be used instead, or a character to be used instead. The UNICODE character #\uFFFD can be used here only if it is encodable in the character set.

### 31.5.5. Function `EXT:ENCODING-CHARSET`

#### Platform Dependent: Only in CLISP built with compile-time flag UNICODE

The function `(`EXT:ENCODING-CHARSET` `encoding`)` returns the charset of the `encoding`, as a `SYMBOL` or a `STRING`.

### Warning

`(`STRING` (`EXT:ENCODING-CHARSET` `encoding`))` is not necessarily a valid `MIME` name.

### 31.5.6. Default encodings

Besides every file/pipe/socket `STREAM` containing an encoding, the following `SYMBOL-MACRO` places contain global `EXT:ENCODING`s:

`SYMBOL-MACRO` `CUSTOM:*DEFAULT-FILE-ENCODING*`. The `SYMBOL-MACRO` place `CUSTOM:*DEFAULT-FILE-ENCODING*` is the encoding used for new file/pipe/socket `STREAM`, when no `:EXTERNAL-FORMAT` argument was specified.

Platform Dependent: Only in CLISP built with compile-time flag UNICODE

  The following are `SYMBOL-MACRO` places.

  `CUSTOM:*PATHNAME-ENCODING*`
    is the encoding used for pathnames in the file system. Normally, this should be a 1:1 encoding. Its line terminator mode is ignored.
  `CUSTOM:*TERMINAL-ENCODING*`
    is the encoding used for communication with the terminal, in particular by `*TERMINAL-IO*`.
  `CUSTOM:*MISC-ENCODING*`
    is the encoding used for access to environment variables, command line options, and the like. Its line terminator mode is ignored.
  `CUSTOM:*FOREIGN-ENCODING*`
    is the encoding for strings passed through the “FFI” (some platforms only). If it is a 1:1 encoding, i.e. an encoding in which every character is represented by one byte, it is also used for passing characters through the “FFI”.

The default encoding objects are initialized according to ``-E`domain` `encoding```.

### Reminder

You have to use `EXT:LETF`/`EXT:LETF*` for `SYMBOL-MACRO`s; `LET`/`LET*` will not work!

#### 31.5.6.1. Default line terminator

The line terminator facet of the above `EXT:ENCODING`s is determined by the following logic: since CLISP understands all possible line terminators on input (see Section 13.10, “Treatment of Newline during Input and Output ”), all that matters is what line terminator do most other programs expect?

Platform Dependent: UNIX platform only.
  If a non-0 `O_BINARY` cpp constant is defined, we assume that the OS distinguishes between text and binary files, and, since the encodings are relevant only for text files, we thus use `:DOS`; otherwise the default is `:UNIX`.
Platform Dependent: Win32 platform only.
  Since most Win32 programs expect CRLF, the default line terminator is `:DOS`.

This boils down to the following code in `src/encoding.d`:

     #if defined(WIN32) || (defined(UNIX) && (O_BINARY != 0))

### Default line terminator on Cygwin

Both of the above tests pass on Cygwin, so the default line terminator is `:DOS`. If you so desire, you can change it in your RC file.

### 31.5.7. Converting between strings and byte vectors

Encodings can also be used to convert directly between strings and their corresponding byte vector representation according to that encoding.

`(`EXT:CONVERT-STRING-FROM-BYTES` `vector` `encoding` `&KEY` `:START` `:END`)`
  converts the subsequence of `vector` (a `(`VECTOR` (`UNSIGNED-BYTE` 8))`) from `start` to `end` to a `STRING`, according to the given `encoding`, and returns the resulting string.
`(`EXT:CONVERT-STRING-TO-BYTES` `string` `encoding` `&KEY` `:START` `:END`)`
  converts the subsequence of `string` from `start` to `end` to a `(`VECTOR` (`UNSIGNED-BYTE` 8))`, according to the given `encoding`, and returns the resulting byte vector.

| These notes document CLISP version 2.45 | Last modified: 2008-05-14

Enter your search termsSubmit search form
Webclisp.cons.orgclisp.podval.orgwww.lisp.org

Prev | Up | Next
31.4. Internationalization of CLISP | Home | 31.6. Generic streams