Unicode Objects and Codecs

Unicode Objects

These are the basic Unicode object types used for the Unicode implementation in Python:

Note that UCS2 and UCS4 Python builds are not binary compatible. Please keep this in mind when writing extensions or interfaces.

The following APIs are really C macros and can be used to do fast checks and to access internal read-only data of Unicode objects:

Unicode provides many different character properties. The most often needed ones are available through these macros which are mapped to C functions depending on the Python configuration.

These APIs can be used for fast direct character conversions:

To create Unicode objects and access their basic sequence properties, use these APIs:

If the platform supports :ctype:`wchar_t` and provides a header file wchar.h, Python can interface directly to this type using the following functions. Support is optimized if Python’s own :ctype:`Py_UNICODE` type is identical to the system’s :ctype:`wchar_t`.

Built-in Codecs

Python provides a set of builtin codecs which are written in C for speed. All of these codecs are directly usable via the following functions.

Many of the following APIs take two arguments encoding and errors. These parameters encoding and errors have the same semantics as the ones of the builtin unicode() Unicode object constructor.

Setting encoding to NULL causes the default encoding to be used which is ASCII. The file system calls should use :cdata:`Py_FileSystemDefaultEncoding` as the encoding for file names. This variable should be treated as read-only: On some systems, it will be a pointer to a static string, on others, it will change at run-time (such as when the application invokes setlocale).

Error handling is set by errors which may also be set to NULL meaning to use the default handling defined for the codec. Default error handling for all builtin codecs is 「strict」 (ValueError is raised).

The codecs all use a similar interface. Only deviation from the following generic ones are documented for simplicity.

These are the generic codec APIs:

These are the UTF-8 codec APIs:

These are the UTF-32 codec APIs:

These are the UTF-16 codec APIs:

These are the 「Unicode Escape」 codec APIs:

These are the 「Raw Unicode Escape」 codec APIs:

These are the Latin-1 codec APIs: Latin-1 corresponds to the first 256 Unicode ordinals and only these are accepted by the codecs during encoding.

These are the ASCII codec APIs. Only 7-bit ASCII data is accepted. All other codes generate errors.

These are the mapping codec APIs:

This codec is special in that it can be used to implement many different codecs (and this is in fact what was done to obtain most of the standard codecs included in the encodings package). The codec uses mapping to encode and decode characters.

Decoding mappings must map single string characters to single Unicode characters, integers (which are then interpreted as Unicode ordinals) or None (meaning 「undefined mapping」 and causing an error).

Encoding mappings must map single Unicode characters to single string characters, integers (which are then interpreted as Latin-1 ordinals) or None (meaning 「undefined mapping」 and causing an error).

The mapping objects provided must only support the __getitem__ mapping interface.

If a character lookup fails with a LookupError, the character is copied as-is meaning that its ordinal value will be interpreted as Unicode or Latin-1 ordinal resp. Because of this, mappings only need to contain those mappings which map characters to different code points.

The following codec API is special in that maps Unicode to Unicode.

These are the MBCS codec APIs. They are currently only available on Windows and use the Win32 MBCS converters to implement the conversions. Note that MBCS (or DBCS) is a class of encodings, not just one. The target encoding is defined by the user settings on the machine running the codec.

Methods and Slot Functions

The following APIs are capable of handling Unicode objects and strings on input (we refer to them as strings in the descriptions) and return Unicode objects or integers as appropriate.

They all return NULL or -1 if an exception occurs.