Python 3 Binary Data Services

This page covers the struct and codecs binary services available in Python 3.

struct — Interpret bytes as packed binary data

The struct module performs conversions between Python values and C structs represented as Python bytes objects. This can be used in handling binary data stored in files or from network connections, among other sources. It uses Format Strings as compact descriptions of the layout of the C structs and the intended conversion to/from Python values.

Struct functions and exceptions

The module defines the following exception and functions:

exception struct.error

struct.pack(fmt,  v1,  v2, ...)

struct.pack_into(fmt,  buffer,  offset,  v1,  v2, ...)

struct.unpack(fmt, buffer)

struct.unpack_from(fmt,  buffer,  offset=0)

struct.iter_unpack(fmt, buffer)

struct.calcsize(fmt)

Struct format strings

Format strings are the mechanism used to specify the expected layout when packing and unpacking data. They are built up from Format Characters, which specify the type of data being packed/unpacked. Also, there are special characters for controlling the Byte Order, Size, and Alignment.

Struct byte order, size, and alignment

By default, C types are represented in the machine's native format and byte order, and properly aligned by skipping pad bytes if necessary (according to the rules used by the C compiler).

Alternatively, the first character of the format string can indicate the byte order, size, and alignment of the packed data, according to the following table:

If the first character is not one of these, '@' is assumed.

Native byte order is big-endian or little-endian, depending on the host system. For example, Intel x86 and AMD64 (x86-64) are little-endian; Motorola 68000 and PowerPC G5 are big-endian; ARM and Intel Itanium feature switchable endianness (bi-endian). Use sys.byteorder to check the endianness of your system.

Native size and alignment are determined using the C compiler's sizeof expression. This is always combined with native byte order.

Standard size depends only on the format character; see the table in the Format Characters section.

Note the difference between '@' and '=': both use native byte order, but the size and alignment of the latter is standardized.

The form '!' is available for those poor souls who claim they can’t remember whether network byte order is big-endian or little-endian.

There is no way to indicate non-native byte order (force byte-swapping); use the appropriate choice of '<' or '>'. Notes:

• Padding is only automatically added between successive structure members. No padding is added at the beginning or the end of the encoded struct.
• No padding is added when using non-native size and alignment, e.g., with ‘<’, ‘>’, ‘=’, and ‘!’.
• To align the end of a structure to the alignment requirement of a particular type, end the format with the code for that type with a repeat count of zero. See examples.

Struct format characters

Format characters have the following meaning; the conversion between C and Python values should be obvious given their types. The ‘Standard size’ column refers to the size of the packed value in bytes when using standard size; that is, when the format string starts with one of '<', '>', '!' or '='. When using native size, the size of the packed value is platform-dependent.

Notes:

• The '?' conversion code corresponds to the _Bool type defined by C99. If this type is not available, it is simulated using a char. In standard mode, it is always represented by one byte.
• The 'q' and 'Q' conversion codes are available in native mode only if the platform C compiler supports C long long, or, on Windows, __int64. They are always available in standard modes.
• When attempting to pack a non-integer using any of the integer conversion codes, if the non-integer has a __index__() method then that method is called to convert the argument to an integer before packing.
• Changed in version 3.2: Use of the __index__() method for non-integers is new in 3.2.
• The 'n' and 'N' conversion codes are only available for the native size (selected as the default or with the '@' byte order character).
• For the standard size, you can use whichever of the other integer formats fits your application.
• For the 'f' and 'd' conversion codes, the packed representation uses the IEEE 754 binary32 (for 'f') or binary64 (for 'd') format, regardless of the floating-point format used by the platform.
• The 'P' format character is only available for the native byte ordering (selected as the default or with the '@' byte order character). The byte order character '=' chooses to use little-endian or big-endian ordering based on the host system. The struct module does not interpret this as native ordering, so the 'P' format is not available.

A format character may be preceded by an integral repeat count. For example, the format string '4h' means the same as 'hhhh'.

Whitespace characters between formats are ignored; a count and its format must not contain whitespace though.

For the 's' format character, the count is interpreted as the length of the bytes, not a repeat count like for the other format characters; for example, '10s' means a single 10-byte string, while '10c' means 10 characters. If a count is not given, it defaults to 1. For packing, the string is truncated or padded with null bytes as appropriate to make it fit. For unpacking, the resulting bytes object always has exactly the specified number of bytes. As a special case, '0s' means a single, empty string (while '0c' means 0 characters).

When packing a value x using one of the integer formats ('b', 'B', 'h', 'H', 'i', 'I', 'l', 'L', 'q', 'Q'), if x is outside the valid range for that format then struct.error is raised.

Changed in version 3.1: In 3.0, some of the integer formats wrapped out-of-range values and raised DeprecationWarning instead of struct.error.

The 'p' format character encodes a “Pascal string”, meaning a short variable-length string stored in a fixed number of bytes, given by the count. The first byte stored is the length of the string, or 255, whichever is smaller. The bytes of the string follow. If the string passed in to pack() is too long (longer than the count minus 1), only the leading count-1 bytes of the string are stored. If the string is shorter than count-1, it is padded with null bytes so that exactly count bytes in all are used. Note that for unpack(), the 'p' format character consumes count bytes, but that the string returned can never contain more than 255 bytes.

For the '?' format character, the return value is either True or False. When packing, the truth value of the argument object is used. Either 0 or 1 in the native or standard bool representation is packed, and any non-zero value is True when unpacking.

Examples

A basic example of packing/unpacking three integers:

>>> from struct import *
>>> pack('hhl', 1, 2, 3)
b'\x00\x01\x00\x02\x00\x00\x00\x03'
>>> unpack('hhl', b'\x00\x01\x00\x02\x00\x00\x00\x03')
(1, 2, 3)
>>> calcsize('hhl')
8

Unpacked fields can be named by assigning them to variables or by wrapping the result in a named tuple:

>>> record = b'raymond   \x32\x12\x08\x01\x08'
>>> name, serialnum, school, gradelevel = unpack('<10sHHb', record)
>>> from collections import namedtuple
>>> Student = namedtuple('Student', 'name serialnum school gradelevel')
>>> Student._make(unpack('<10sHHb', record))
Student(name=b'raymond   ', serialnum=4658, school=264, gradelevel=8)

The ordering of format characters may have an impact on size since the padding needed to satisfy alignment requirements is different:

>>> pack('ci', b'*', 0x12131415)
b'*\x00\x00\x00\x12\x13\x14\x15'
>>> pack('ic', 0x12131415, b'*')
b'\x12\x13\x14\x15*'
>>> calcsize('ci')
8
>>> calcsize('ic')
5

The following format 'llh0l' specifies two pad bytes at the end, assuming longs are aligned on 4-byte boundaries:

>>> pack('llh0l', 1, 2, 3)
b'\x00\x00\x00\x01\x00\x00\x00\x02\x00\x03\x00\x00'

This only works when native size and alignment are in effect; standard size and alignment does not enforce any alignment.

Struct classes

The struct module also defines the following type:

class struct.Struct(format)

Compiled Struct objects support the following methods and attributes:

pack(v1,  v2, ...)

pack_into(buffer,  offset,  v1,  v2, ...)

unpack(buffer)

unpack_from(buffer, offset=0)

iter_unpack(buffer)

format

size

Codecs module

The codecs module handles the compression and decompression of some standard data formats.

Codecs: registry and base classes

This module defines base classes for standard Python codecs (encoders and decoders) and provides access to the internal Python codec registry which manages the codec and error handling look up process.

It defines the following functions:

codecs.encode(obj [, encoding [, errors]])

codecs.decode(obj [, encoding [, errors]])

codecs.register(search_function)

factory(errors='strict')

factory(stream, errors='strict')

codecs.lookup(encoding)

To simplify access to the various codecs, the module provides these additional functions which use lookup() for the codec look up:

codecs.getencoder(encoding)

codecs.getdecoder(encoding)

codecs.getincrementalencoder(encoding)

codecs.getincrementaldecoder(encoding)

codecs.getreader(encoding)

codecs.getwriter(encoding)

codecs.register_error(name,  error_handler)

codecs.lookup_error(name)

codecs.strict_errors(exception)

codecs.replace_errors(exception)

codecs.ignore_errors(exception)

codecs.xmlcharrefreplace_errors(exception)

codecs.backslashreplace_errors(exception)

To simplify working with encoded files or stream, the module also defines these utility functions:

codecs.open(filename,  mode [, encoding [, errors [, buffering]]])

codecs.EncodedFile(file,  data_encoding,  file_encoding=None,  errors='strict')

codecs.iterencode(iterator,  encoding,  errors='strict',  **kwargs)

codecs.iterdecode(iterator,  encoding,  errors='strict',  **kwargs)

The module also provides the following constants that are useful for reading and writing to platform dependent files:

codecs.BOMcodecs.BOM_BEcodecs.BOM_LEcodecs.BOM_UTF8codecs.BOM_UTF16codecs.BOM_UTF16_BEcodecs.BOM_UTF16_LEcodecs.BOM_UTF32codecs.BOM_UTF32_BEcodecs.BOM_UTF32_LE

Codecs base classes

The codecs module defines a set of base classes which define the interface and can also be used to easily write codecs for use in Python.

Each codec has to define four interfaces to make it usable as codec in Python: stateless encoder, stateless decoder, stream reader and stream writer. The stream reader and writers often reuse the stateless encoder/decoder to implement the file protocols.

The Codec class defines the interface for stateless encoders/decoders.

To simplify and standardize error handling, the encode() and decode() methods may implement different error handling schemes by providing the errors string argument. The following string values are defined and implemented by all standard Python codecs:

Also, the following error handlers are specific to Unicode encoding schemes:

The set of allowed values can be extended via register_error().

Codec objects

The Codec class defines these methods that also define the function interfaces of the stateless encoder and decoder:

Codec.encode(input [, errors])

Codec.decode(input[,  errors])

The IncrementalEncoder and IncrementalDecoder classes provide the basic interface for incremental encoding and decoding. Encoding/decoding the input isn’t done with one call to the stateless encoder/decoder function, but with multiple calls to the encode()/decode() method of the incremental encoder/decoder. The incremental encoder/decoder keeps track of the encoding/decoding process during method calls.

The joined output of calls to the encode()/decode() method is the same as if all the single inputs were joined into one, and this input was encoded/decoded with the stateless encoder/decoder.

Codecs IncrementalEncoder objects

The IncrementalEncoder class is used for encoding an input in multiple steps. It defines the following methods which every incremental encoder must define to be compatible with the Python codec registry.

class codecs.IncrementalEncoder([errors])

encode(object [, final])

reset()

IncrementalEncoder.getstate()

IncrementalEncoder.setstate(state)

Codecs IncrementalDecoder objects

The IncrementalDecoder class is used for decoding an input in multiple steps. It defines the following methods which every incremental decoder must define to be compatible with the Python codec registry.

class codecs.IncrementalDecoder([errors])

decode(object [, final])

reset()

getstate()

setstate(state)

The StreamWriter and StreamReader classes provide generic working interfaces which can implement new encoding submodules very easily. See encodings.utf_8 for an example of how this is done.

Codecs StreamWriter objects

The StreamWriter class is a subclass of Codec and defines the following methods which every stream writer must define to be compatible with the Python codec registry.

class codecs.StreamWriter(stream [, errors])

write(object)

writelines(list)

reset()

Codecs StreamReader objects

The StreamReader class is a subclass of Codec and defines the following methods which every stream reader must define to be compatible with the Python codec registry.

class codecs.StreamReader(stream [, errors])

read([size [, chars [, firstline]]])

readline([size [, keepends]])

readlines([sizehint [, keepends]])

reset()

Codecs StreamReaderWriter objects

The StreamReaderWriter allows wrapping streams which work in both read and write modes.

The design is such that one can use the factory functions returned by the lookup() function to construct the instance.

class codecs.StreamReaderWriter(stream,  Reader,  Writer,  errors)

StreamReaderWriter instances define the combined interfaces of StreamReader and StreamWriter classes. They inherit all other methods and attributes from the underlying stream.

Codecs StreamRecoder objects

The StreamRecoder provide a frontend - backend view of encoding data that is sometimes useful when dealing with different encoding environments.

The design is such that one can use the factory functions returned by the lookup() function to construct the instance.

class codecs.StreamRecoder(stream,  encode,  decode,  Reader,  Writer,  errors)

StreamRecoder instances define the combined interfaces of StreamReader and StreamWriter classes. They inherit all other methods and attributes from the underlying stream.

Codecs encodings and unicode

Strings are stored internally as sequences of codepoints in range 0 - 10FFFF (see PEP 393 for more details about the implementation). Once a string object is used outside of CPU and memory, CPU endianness and how these arrays are stored as bytes become an issue. Transforming a string object into a sequence of bytes is called encoding and recreating the string object from the sequence of bytes is known as decoding. There are many different methods for how this transformation can be done (these methods are also called encodings). The simplest method is to map the codepoints 0-255 to the bytes 0x0-0xff. This means that a string object containing codepoints above U+00FF can’t be encoded with this method (which is called 'latin-1' or 'iso-8859-1'). str.encode() raises a UnicodeEncodeError that looks like this:

UnicodeEncodeError: 'latin-1' codec can't encode character 
'\u1234' in position 3: ordinal not in range(256).

There's another group of encodings (the so called charmap encodings) that choose a different subset of all Unicode code points and how these codepoints are mapped to the bytes 0x0-0xff. To see how this is done open e.g., encodings/cp1252.py (which is an encoding that is used primarily on Windows). There's a string constant with 256 characters that shows you which character is mapped to each byte value.

All of these encodings can only encode 256 of the 1114112 codepoints defined in Unicode. A simple and straightforward way that can store each Unicode code point, is to store each codepoint as four consecutive bytes. There are two possibilities: store the bytes in big endian or in little endian order. These two encodings are called UTF-32-BE and UTF-32-LE respectively. Their disadvantage is that if e.g., you use UTF-32-BE on a little endian machine, you always have to swap bytes on encoding and decoding. UTF-32 avoids this problem: bytes is always in natural endianness. When these bytes are read by a CPU with a different endianness, then bytes have to be swapped though. To be able to detect the endianness of a UTF-16 or UTF-32 byte sequence, there's the so called BOM (“Byte Order Mark”). This is the Unicode character U+FEFF. This character can be prepended to every UTF-16 or UTF-32 byte sequence. The byte swapped version of this character (0xFFFE) is an illegal character that may not appear in a Unicode text. So when the first character in an UTF-16 or UTF-32 byte sequence appears to be a U+FFFE the bytes have to be swapped on decoding. Unfortunately, the character U+FEFF had a second purpose as a ZERO WIDTH NO-BREAK SPACE: a character with no width and doesn't allow a word to be split. It can e.g., be used to give hints to a ligature algorithm. With Unicode 4.0, using U+FEFF as a ZERO WIDTH NO-BREAK SPACE is deprecated (with U+2060 (WORD JOINER) assuming this role). Nevertheless Unicode software still must be able to handle U+FEFF in both roles: as a BOM it's a device to determine the storage layout of the encoded bytes, and vanishes once the byte sequence is decoded into a string; as a ZERO WIDTH NO-BREAK SPACE it's a normal character that is decoded like any other.

There's another encoding that can encode the full range of Unicode characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two parts: marker bits (the most significant bits) and payload bits. The marker bits are a sequence of zero to four 1 bits followed by a 0 bit. Unicode characters are encoded like this (with x being payload bits, which when concatenated give the Unicode character):

The least significant bit of the Unicode character is the rightmost x bit.

As UTF-8 is an 8-bit encoding no BOM is required and any U+FEFF character in the decoded string (even if it's the first character) is treated as a ZERO WIDTH NO-BREAK SPACE.

Without external information it's impossible to reliably determine which encoding was used for encoding a string. Each charmap encoding can decode any random byte sequence. However, that's not possible with UTF-8, as UTF-8 byte sequences have a structure that doesn’t allow arbitrary byte sequences. To increase the reliability with which a UTF-8 encoding can be detected, Microsoft invented a variant of UTF-8 (that Python 2.5 calls "utf-8-sig") for its Notepad program: Before any of the Unicode characters is written to the file, a UTF-8 encoded BOM (which looks like this as a byte sequence: 0xef, 0xbb, 0xbf) is written. As it's rather improbable that any charmap encoded file starts with these byte values (which would e.g., map to

LATIN SMALL LETTER I WITH DIAERESIS
RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
INVERTED QUESTION MARK

In iso-8859-1), this increases the probability that a utf-8-sig encoding can be correctly guessed from the byte sequence. So here the BOM is not used to be able to determine the byte order used for generating the byte sequence, but as a signature that helps in guessing the encoding. On encoding the utf-8-sig codec writes 0xef, 0xbb, 0xbf as the first three bytes to the file. On decoding, utf-8-sig skips those three bytes if they appear as the first three bytes in the file. In UTF-8, the use of the BOM is discouraged and should generally be avoided.

Additional Python Reference

• Python 3 programming language
• The Python standard library
• Data types in Python
• Python text processing modules
• Python numeric and mathematical modules
• Python operating system services
• Functional programming models in Python
• The Python HTML module