Numbers and Meaning - Binary, Text and Unicode

7.0 What this chapter gives you#

1. You will be able to explain why computers count in twos, and convert any number to binary and back by two different methods.
2. You will be able to read hexadecimal on sight, and say where you meet it in real work: memory addresses, colour codes, MAC addresses, error codes.
3. You will be able to say what a byte is, why “octet” exists, and what “64-bit CPU” means about registers and address space.
4. You will be able to explain two’s complement, give the exact range of any signed integer size, and name three real disasters caused by overflow.
5. You will be able to encode a decimal fraction into IEEE 754 float32 bits by hand, decode a pattern back, and say why 0.1 plus 0.2 is not 0.3.
6. You will be able to trace text encoding from Morse code through ASCII to Unicode, and write the UTF-8 bytes for an emoji by hand.
7. You will be able to explain endianness, parity, checksums, CRC32 and ECC memory, and say what each one can and cannot catch.
8. You will be able to build a Huffman tree, explain LZ77, and state Shannon entropy in one formula.
9. You will understand the deepest idea here: a pattern of bits has no meaning until something decides how to read it.

7.1 Why base 2#

PLAIN7.1.1 in simple words#

1. A computer is made of switches. A switch is either on or off. There is no “half on” a machine can read back reliably.
2. So a computer can store only two marks. Call them 0 and 1.
3. To store bigger numbers, line up many switches in a row.
4. Two switches give four patterns: 00, 01, 10, 11. Three give eight. Each new switch doubles the count.
5. Counting with only two marks is called binary, or base 2.
6. Our normal counting uses ten marks, 0 to 9. That is base 10, or decimal, and the only reason for ten is that people have ten fingers.
7. A machine has two voltage levels, so it uses two marks.

PLAIN7.1.2 a picture in your head#

1. Picture a row of eight light switches on a wall.
2. Each has a price tag above it. From the right: 1, 2, 4, 8, 16, 32, 64, 128. Each tag is double the one to its right.
3. To show a number, flip up the switches whose tags add to it. For 5, flip up 4 and 1. That is 00000101. For 156, flip up 128, 16, 8 and 4.
4. Every number from 0 to 255 has exactly one pattern, and no number has two.
5. Reading the wall is just adding the tags of the switches that are up.

Where this comparison breaks

1. Real memory does not hold a switch position. It holds a small charge or a magnetic direction that fades and must be refreshed or rewritten.
2. Real switches are also not exactly on or off. They sit above or below a voltage threshold, and the circuit rounds them to a clean 0 or 1.

PLAIN7.1.3 a worked example#

Counting from 0 to 32 in binary, five bits then six.

1. Notice 32 needs a sixth bit. Five bits stop at 31.
2. N bits hold 2 to the power N patterns, from 0 to (2 to the N) minus 1.

The place values in one byte:

Method one, decimal to binary by repeated division. Convert 156.

156 / 2 = 78  remainder 0   <- lowest bit
 78 / 2 = 39  remainder 0
 39 / 2 = 19  remainder 1
 19 / 2 =  9  remainder 1
  9 / 2 =  4  remainder 1
  4 / 2 =  2  remainder 0
  2 / 2 =  1  remainder 0
  1 / 2 =  0  remainder 1   <- highest bit

Read the remainders bottom to top: 1 0 0 1 1 1 0 0
156 decimal = 10011100 binary

Method two, decimal to binary by subtraction. Same number, 156.

Largest place value not bigger than 156 is 128. Use it.
  156 - 128 =  28    bit 7 = 1
Is 64 <= 28? No.    bit 6 = 0
Is 32 <= 28? No.    bit 5 = 0
Is 16 <= 28? Yes.  28 - 16 = 12   bit 4 = 1
Is  8 <= 12? Yes.  12 -  8 =  4   bit 3 = 1
Is  4 <=  4? Yes.   4 -  4 =  0   bit 2 = 1
Is  2 <=  0? No.    bit 1 = 0
Is  1 <=  0? No.    bit 0 = 0

Result: 10011100

Binary back to decimal is easier still. Take 10011100 and add the place values of the bits that are 1: 128 + 16 + 8 + 4 = 156.

1. Both directions agree, as they must. Division scales better to large numbers; subtraction is easier to do in your head for one byte.

PLAIN7.1.4 what is really happening inside#

1. Inside a chip a 1 is a wire held near the supply voltage and a 0 is a wire held near ground. Modern logic cores run at roughly 0.7 to 1.2 volts, so the whole gap is under one volt.
2. The circuit never measures the exact voltage. It compares it with a threshold and snaps the answer to a clean 0 or 1.
3. That snapping is why binary survives noise. A wire at 0.9 volts and one at 1.05 volts are both read as 1, and both come out identical.
4. A base-10 circuit would need ten bands in the same one volt gap, about 0.1 volts each, and noise would ruin it.
5. So base 2 is not a preference. It is the widest safety margin one wire can give you.
6. The honest version: some devices do store more than two levels. NAND flash stores 3 or 4 bits per cell using 8 or 16 charge levels. It works because a memory cell is read slowly and carefully, not billions of times a second.

TECHNICAL7.1.5 the engineer’s version#

1. Binary is a positional numeral system with radix 2. The digit at position i carries weight 2^i, counted from 0 at the least significant bit. An n-bit unsigned field represents integers in [0, 2^n - 1] inclusive.
2. Repeated division produces the radix-2 expansion least significant digit first. The subtraction method is greedy expansion against descending powers of 2; it is correct because binary place values are super-increasing, each power exceeding the sum of all lower powers.
3. Gottfried Wilhelm Leibniz described binary arithmetic in a 1703 paper for the Paris Academy, long before any machine used it.
4. Claude Shannon’s 1937 MIT master’s thesis, “A Symbolic Analysis of Relay and Switching Circuits”, first tied Boolean algebra to switching hardware.
5. Noise margin is the real engineering reason for radix 2. With two levels the margin is roughly half the supply rail; with ten levels it is one eighteenth, before any process variation is counted.
6. Multi-level storage exists where read latency may be large: NAND flash SLC (1 bit per cell), MLC (2), TLC (3), QLC (4).

7. Observe it with python3 -c "print(bin(156))" or echo 'obase=2; 156' | bc.

WORDS7.1.6 remember these#

1. Bit — one on-or-off mark — a binary digit, the smallest unit of information, one element of the set {0, 1}.
2. Binary — counting with only 0 and 1 — base 2 positional notation with place weights 2^i.
3. Base or radix — how many different digits a system uses — the number of distinct symbols per position in a positional numeral system.
4. Place value — the worth of a digit’s slot — the weight r^i applied to the digit at position i in radix r.
5. Most significant bit — the leftmost, biggest-value bit — the bit with the highest positional weight, often written MSB.
6. Least significant bit — the rightmost, worth 1 — the bit of weight 2^0, often written LSB.

7.2 Hexadecimal and octal#

PLAIN7.2.1 in simple words#

1. Binary is correct but painful to read. 11010110 is easy to mistype.
2. Engineers group the bits in fours and give each group one short name.
3. Four bits have sixteen patterns, so we need sixteen digit names: 0 to 9 and then A, B, C, D, E, F. A means ten, F means fifteen.
4. This is hexadecimal, usually shortened to hex. So 11010110 becomes 1101 and 0110, which is D and 6, written 0xD6.
5. The 0x in front is just a label saying “the rest is hex”.
6. Hex is not a different kind of number. It is a shorter way of writing the same bits.
7. There is also octal, which groups bits in threes and uses digits 0 to 7. It is mostly historical, but it survives in file permissions on Linux and macOS.

PLAIN7.2.2 a picture in your head#

1. Think of a long phone number with no spaces: 919876543210. Hard to read, easy to lose your place, easy to copy wrong.
2. Now group it: 91 98765 43210. Same number, far easier to handle.
3. Hex does exactly this to bits, but the grouping is fixed at four and each group gets a single character instead of a space.
4. One hex digit is always exactly four bits. Two hex digits are always exactly one byte. That rule never bends.

Where this comparison breaks

1. Phone number grouping is a display habit and can vary by country.
2. Hex grouping is arithmetic. Because 16 is 2 to the fourth power, each hex digit maps to four bits with no carrying between groups. Decimal grouping has no such property, which is why decimal to binary needs real division.

PLAIN7.2.3 a worked example#

The complete conversion table. Learn this and hex becomes automatic.

Worked conversion, hex to binary to decimal. Take 0xB4.

0xB4  ->  B    4
      -> 1011 0100
      -> 10110100

Now add the place values:
  128 + 0 + 32 + 16 + 0 + 4 + 0 + 0 = 180

0xB4 = 10110100 binary = 180 decimal

Worked conversion, decimal to hex. Take 3000.

3000 / 16 = 187 remainder  8  -> digit 8
 187 / 16 =  11 remainder 11  -> digit B
  11 / 16 =   0 remainder 11  -> digit B

Read bottom to top: 0xBB8
Check: 11 x 256 + 11 x 16 + 8 = 2816 + 176 + 8 = 3000

Octal, three bits per digit, for Unix permissions.

chmod 755 script.sh

7 = 111 = read, write, execute   (owner)
5 = 101 = read, no write, execute (group)
5 = 101 = read, no write, execute (everyone else)

1. The three bits are, in order, read (4), write (2), execute (1).
2. So 6 is read plus write, 4 is read only, 0 is nothing.

PLAIN7.2.4 what is really happening inside#

1. The machine never stores hex. Memory holds bits and nothing else. Hex exists only where a human reads or types a value.
2. A debugger, a hex editor and an error message all convert bits to hex just before printing, and hex back to bits just after you type.
3. Hex won over decimal for this job because of alignment. Byte, nibble and word boundaries all fall on hex digit boundaries.
4. A 32-bit value is always exactly 8 hex digits and a 64-bit value always exactly 16, so you can count the digits and know the size. In decimal the same value takes 1 to 10 digits, so the width is invisible.
5. Octal survives from machines whose word sizes divided by three, such as the 12-bit PDP-8 and the 36-bit PDP-10, where four bits per digit did not fit.

TECHNICAL7.2.5 the engineer’s version#

1. Hexadecimal is radix 16, octal is radix 8. Both are power-of-two radices, so conversion to and from binary is pure regrouping, never division.
2. A 4-bit group is a nibble (also spelled nybble). One hex digit is one nibble. Two nibbles form one octet.
3. Notation is language-specific: 0x1F in C, C++, Java, Python, Go, Rust and JavaScript; 1Fh or $1F in assemblers; 16#1F# in Ada; #x1F in Lisp. Octal is 0o755 in Python 3 and Rust, and a bare leading 0 in C, which is a well-known source of bugs.
4. Where you meet hex in real work:

5. File magic numbers, the first bytes that identify a format:

6. JPEG files start FF D8 FF. GIF files start 47 49 46 38, which reads “GIF8”. The file command on Unix works mainly from a database of these patterns, held in /usr/share/file/magic or similar.
7. In a MAC address the first three octets are the Organizationally Unique Identifier assigned to a vendor by the IEEE, and the last three are chosen by that vendor.
8. Tools that show hex: xxd, hexdump -C, od -t x1, objdump -d, and the x/16xb command inside GDB.

WORDS7.2.6 remember these#

1. Hexadecimal — base 16, digits 0 to F — radix-16 positional notation, four bits per digit, standard for displaying binary data.
2. Nibble — half a byte — a 4-bit group, exactly one hexadecimal digit.
3. Octal — base 8, digits 0 to 7 — radix-8 notation, three bits per digit, used for Unix permission modes and legacy word sizes.
4. Magic number — the fingerprint at the start of a file — a fixed byte sequence at a known offset used to identify a file format.
5. Prefix 0x — a label meaning “hex follows” — a lexical marker for hexadecimal integer literals in C-family languages.

7.3 Bits, bytes and words#

PLAIN7.3.1 in simple words#

1. One bit is one 0 or 1, and almost useless alone. Eight bits make one byte, which holds 256 different values.
2. A byte is the smallest chunk memory hands out. You cannot ask for “bit number 5” directly; you ask for the byte and pick the bit yourself.
3. A byte is enough for one English letter, or a number from 0 to 255.
4. A word is how many bits the processor works on at once. Old processors used 8-bit words, then 16, then 32. Today it is 64.
5. “A 64-bit computer” means the processor handles 64 bits at a time and can point at a huge amount of memory.
6. Networking people say octet instead of byte, because “byte” has not always meant exactly eight bits.

PLAIN7.3.2 a picture in your head#

1. Think of a warehouse of identical boxes on numbered shelves. Each box is a byte and each shelf number is a memory address.
2. The forklift is the processor, and its fork width is the word size. A 64-bit forklift picks up eight boxes in one trip; a 32-bit one picks up four.
3. The address space is how many shelf labels the building can print. With 32-bit labels you get about 4.29 billion; with 64-bit labels, 18.4 billion billion.

Where this comparison breaks

1. A real forklift wastes no effort carrying fewer boxes. A real 64-bit CPU moving one byte still uses a full 64-bit path, and pointers get bigger, which uses more cache. Wider is not free.
2. Real memory addresses are not all backed by RAM. Large regions are mapped to devices, or to nothing at all.

PLAIN7.3.3 a worked example#

The bits inside one byte, with their weights.

   bit 7   6   5   4   3   2   1   0
        |   |   |   |   |   |   |   |
value 128  64  32  16   8   4   2   1

Byte  1   0   1   1   0   1   1   0   = 0xB6
      128     32  16      4   2       = 182

1. The largest value in one byte is 11111111, which is 255, or 0xFF.
2. Two bytes side by side make 16 bits, holding 0 to 65535.
3. Four bytes make 32 bits, holding 0 to 4,294,967,295.

Why a 32-bit machine stopped near 4 GB:

2^32 = 4,294,967,296 distinct addresses
Each address names one byte
So the total space named = 4,294,967,296 bytes
                         = 4 GiB exactly

1. That 4 GiB is not only RAM. The graphics card aperture, the BIOS ROM and other device windows are carved out of the same range.
2. That is why 32-bit Windows machines with 4 GB installed typically reported about 3.2 to 3.5 GB usable. The rest of the addresses were spoken for.

PLAIN7.3.4 what is really happening inside#

1. Inside the processor are small named storage slots called registers. On a 64-bit CPU each general register is 64 bits wide, and so is the adder, so one add instruction handles 64 bits in one step.
2. Separately, the address the CPU puts on the memory path decides how much memory it can name. These two widths are usually equal but need not be: some 8-bit chips had 16-bit addresses, some 32-bit chips 36-bit ones.
3. A pointer is a plain integer holding an address, so on a 64-bit system it occupies 8 bytes. Moving to 64-bit doubles every pointer, and programs full of pointers use noticeably more memory afterwards.
4. The honest version: no shipping x86-64 chip wires all 64 address bits. Current designs use 48, giving 256 TiB; newer server parts support 57, giving 128 PiB. The unused top bits must copy the highest used bit, a rule called canonical form.

TECHNICAL7.3.5 the engineer’s version#

1. The term byte was coined by Werner Buchholz at IBM in July 1956 during the Stretch project, which shipped as the IBM 7030. The spelling was deliberately changed from “bite” so it could not be misread as “bit”.
2. A byte was not always 8 bits. The CDC 6600 used 60-bit words with 6-bit characters. The PDP-10 used 36-bit words with programmable byte sizes of 6, 7, 8 or 9 bits. Some early machines used 9-bit bytes.
3. Because of that history, IETF and ITU specifications say octet when they mean exactly 8 bits. RFC documents use octet throughout for this reason.
4. The 8-bit byte became universal with the IBM System/360, announced 7 April 1964, which fixed 8-bit bytes and 32-bit words.
5. Word size across the generations:

6. Address space arithmetic:

7. Physical Address Extension (PAE), introduced with the Intel Pentium Pro in 1995, widened physical addresses to 36 bits, allowing 64 GiB of RAM on a 32-bit machine. Each individual process was still limited to a 4 GiB virtual address space, so it helped servers running many processes and did little for a single large application.
8. x86-64 canonical addressing sign-extends bit 47 into bits 48 to 63. Intel 5-level paging, shipped in Ice Lake server parts from 2019 and supported in Linux since kernel 4.14, extends this to 57 bits.
9. Observe with getconf LONG_BIT, uname -m, lscpu (look for “Address sizes”), and cat /proc/cpuinfo | grep "address sizes" on Linux.

WORDS7.3.6 remember these#

1. Byte — eight bits — the smallest individually addressable unit of memory on virtually all modern architectures.
2. Octet — a guaranteed eight bits — the term used in IETF and ITU standards because “byte” was historically machine-dependent.
3. Word — the chunk size a CPU works on naturally — the native register and datapath width of an architecture.
4. Register — a slot inside the processor — the fastest storage in the machine, named rather than addressed, typically 64 bits wide today.
5. Address space — the set of memory locations that can be named — the range of addresses expressible in the architecture’s pointer width.
6. Canonical address — a valid pointer shape on x86-64 — an address whose bits 63 to 48 are all copies of bit 47.

7.4 Negative numbers#

PLAIN7.4.1 in simple words#

1. A byte holds a pattern of 0s and 1s. Nothing in it says “minus”, so we must agree on a rule reserving some patterns for negatives.
2. The first idea was to use the leftmost bit as a minus sign. That is sign-magnitude. It gives two zeros, plus zero and minus zero, and needs special rules for adding mixed signs.
3. The second idea was to flip every bit to negate. That is one’s complement. It also gives two zeros and needs an extra carry step.
4. The third idea won everything: flip every bit, then add 1. That is two’s complement. There is exactly one zero, and normal addition just works, because the circuit never needs to know the signs.
5. Every processor you will use today stores signed integers this way.
6. The cost is a lopsided range. In one byte you get -128 to +127, one more negative than positive.

PLAIN7.4.2 a picture in your head#

1. Picture a clock face with 256 marks instead of 12. Going past the top wraps to the start, so 255 plus 1 is 0.
2. Now agree that marks past halfway are read as negatives. The mark we would call 255 we call -1; 254 becomes -2.
3. Under that reading, “add -1” and “add 255” are the same movement. The hand lands in the same place either way.
4. That is the whole trick: subtracting is adding, if you go far enough forward around the dial. The circuit never thinks about signs. It steps around and throws away whatever falls off the top.

Where this comparison breaks

1. A clock hand moving past 12 is harmless. A number wrapping past its top is often a serious bug, because the program keeps running with a value that is wildly wrong rather than stopping.
2. Also, a clock has no “unsigned versus signed” reading. In a computer, the same bits are read one way or the other depending on the declared type, and mixing the two in one expression causes real defects.

PLAIN7.4.3 a worked example#

Making -5 in eight bits, by all three methods.

Start with +5:            00000101

SIGN-MAGNITUDE
  set the top bit:        10000101   (reads as "minus five")
  problem: 10000000 is "minus zero", a second zero

ONE'S COMPLEMENT
  flip every bit:         11111010
  problem: 11111111 is "minus zero", a second zero

TWO'S COMPLEMENT
  flip every bit:         11111010
  add one:                11111011   <- this is -5
  as an unsigned byte that pattern is 251

Now check that plain addition works with no special cases.

      00000101      (+5)
    + 11111011      (-5)
    ----------
    1 00000000

The 1 falls off the left end and is discarded.
What remains is 00000000, which is zero. Correct.

Wraparound at the top of the range.

      01111111      (+127, the largest signed byte)
    + 00000001      (+1)
    ----------
      10000000      which as signed is -128, not +128

1. Adding 1 to the biggest positive number gives the most negative number.
2. This is overflow. The answer did not fit, so it wrapped.

The exact ranges you should know by heart:

1. The 64-bit unsigned maximum written out is 18,446,744,073,709,551,615.
2. The 64-bit signed minimum written out is -9,223,372,036,854,775,808.

PLAIN7.4.4 what is really happening inside#

1. The adder inside a CPU adds two n-bit patterns and produces an n-bit result plus a carry-out bit, the same way every time.
2. To compute A minus B, the CPU negates B and adds. Many designs feed the inverted B into the adder with carry-in set to 1, performing “flip and add one” in a single pass.
3. So one adder does both jobs. That is the real reason two’s complement won: fewer transistors.
4. The CPU sets status flags after each arithmetic instruction, and there are two separate overflow signals:
   1. The carry flag says the result did not fit as an unsigned number.
   2. The overflow flag says it did not fit as a signed number.
5. The processor does not know which reading you intended. It sets both and leaves the program to check. Most programs never check, and the wrong value continues silently.
6. To test whether a pattern is negative, look at the top bit alone. To read the magnitude of a negative, apply the same operation again: flip and add one to 11111011 gives 00000101, which is 5.

TECHNICAL7.4.5 the engineer’s version#

1. Two’s complement represents value v in n bits as the residue v mod 2^n. Interpretation: the MSB carries weight -2^(n-1), all other bits carry the usual positive weights.
2. Range is [-2^(n-1), 2^(n-1) - 1]. The asymmetry is unavoidable: 2^n patterns cannot be split evenly around zero if zero takes one pattern.
3. Consequence to remember: negating the minimum value overflows. In 32-bit arithmetic, -INT_MIN is INT_MIN, and abs(-2147483648) returns -2147483648. This is a genuine source of security bugs.
4. In C and C++, signed integer overflow is undefined behaviour. Unsigned overflow is defined and wraps modulo 2^n. Compilers exploit the undefined case for optimization, so an overflowing signed loop can be transformed in ways that surprise you. In Java, signed overflow is defined to wrap. In Rust, it panics in debug builds and wraps in release builds by default.
5. Historical use: the EDSAC (1949) used two’s complement. The IBM System/360
   1964. used it for binary integers and made it the mainstream choice. One’s complement persisted in the CDC 6600 (1964) and UNIVAC 1100 series; sign-magnitude persisted in the IBM 7090 series.

Three real overflow failures.

6. Year 2038. Unix time is seconds since 00:00:00 UTC on 1 January 1970, traditionally stored in a signed 32-bit time_t. The last representable second is 2,147,483,647, which is 03:14:07 UTC on 19 January 2038. The next increment wraps to -2,147,483,648, which reads as 20:45:52 UTC on 13 December 1901. The fix is a 64-bit time_t; Linux has supported 64-bit time_t on 32-bit architectures since kernel 5.6 (2020). The bug already bites early: AOLserver crashed in May 2006 on a timeout one billion seconds in the future, and a Microsoft Exchange antimalware update failed in January 2022 when a version number was read as a Unix timestamp.
7. YouTube, 2 December 2014. The view counter for the video “Gangnam Style” reached 2,147,483,647, the maximum signed 32-bit integer. Google’s public statement said the company “never thought a video would be watched in numbers greater than a 32-bit integer”. The counter was moved to 64-bit, which raises the ceiling to 9,223,372,036,854,775,807. Note the honest caveat: press coverage described the site as “breaking”, but what the public saw was largely a display Easter egg, and reports differ on exactly when the underlying storage was widened.
8. Ariane 5 Flight 501, 4 June 1996. About 37 seconds after main engine ignition, both Inertial Reference Systems failed within milliseconds of each other. The cause was a conversion of the horizontal bias value, BH, from a 64-bit floating point number to a 16-bit signed integer. The value exceeded 32,767 and raised an Operand Error that was not handled. The code was Ariane 4 alignment software kept running after liftoff “for commonality reasons” although it served no purpose on Ariane 5. The launcher broke up and was destroyed about 39 seconds after ignition. The inquiry board was chaired by Professor Jacques-Louis Lions and reported on 19 July 1996. The financial loss is widely reported as roughly 370 million US dollars; the inquiry report itself states no figure, so treat that number as approximate.
9. Detection tools: -fsanitize=signed-integer-overflow and -fsanitize=unsigned-integer-overflow in Clang and GCC, __builtin_add_overflow in GCC and Clang, checked_add in Rust, Math.addExact in Java.

WORDS7.4.6 remember these#

1. Two’s complement — flip the bits and add one to negate — the standard signed integer encoding where the MSB carries weight -2^(n-1).
2. One’s complement — flip the bits to negate — an older encoding with two representations of zero and end-around carry addition.
3. Sign-magnitude — a separate sign bit plus the plain value — an encoding with two zeros, still used for the sign bit in IEEE 754 floats.
4. Overflow — the answer was too big to fit — the condition where an arithmetic result falls outside the representable range of the type.
5. Wraparound — the number rolls over the top back to the bottom — modular reduction of the result by 2^n, defined for unsigned types in C.
6. Unix time — seconds counted since the start of 1970 — POSIX time_t, seconds since the epoch 1970-01-01T00:00:00Z, ignoring leap seconds.

7.5 Fractions: fixed point and floating point#

PLAIN7.5.1 in simple words#

1. Bits count whole things. To store 3.75 you need an agreement about where the point goes.
2. The simple way is fixed point: decide once that the last two digits are after the point, and never change it. Storing money in paise or cents is fixed point. You store 1075 and everyone knows it means 10.75.
3. Fixed point is exact and fast, but the range is narrow. One setting must cover both tiny and huge values, and it cannot.
4. The other way is floating point: store the digits, and store separately where the point sits. That is scientific notation, like 6.02 times 10 to the 23. Two small numbers describe one enormous one.
5. Floating point gives a gigantic range at a price. Most values are stored slightly wrong, rounded to the nearest one the format can hold.
6. The rounding is tiny but it accumulates, and it is why 0.1 plus 0.2 does not come out as exactly 0.3.

PLAIN7.5.2 a picture in your head#

1. Imagine a metre ruler marked every millimetre. That is fixed point: every measurement is a whole number of millimetres, every gap the same size. But you cannot measure a bacterium and you cannot measure a road.
2. Now imagine an instrument that reports “3 point something, times ten to the power something”.
3. Near 1 it can tell 1.001 from 1.002. Near a million it can only tell 1,000,000 from 1,000,001,000. The steps grow with the number.
4. That is floating point. Precision is relative, not absolute. You get about the same number of significant digits at every scale.

Where this comparison breaks

1. A ruler has no equivalent of infinity, of “not a number”, or of a negative zero. Floating point has all three, and they behave in specific ways.
2. And a ruler’s marks are evenly spaced everywhere. Floating point steps are even only within one power of two, and double at every power boundary.

PLAIN7.5.3 a worked example#

The layout of a 32-bit float, called binary32 or float32.

 bit 31   bits 30..23        bits 22..0
 +-----+---------------+----------------------------+
 |  S  |   exponent    |         fraction           |
 +-----+---------------+----------------------------+
   1 bit    8 bits              23 bits

 value = (-1)^S  x  1.fraction  x  2^(exponent - 127)

1. S is the sign. 0 means positive, 1 means negative.
2. The exponent field stores the real exponent plus 127. That offset is called the bias, and it lets the field stay an unsigned number.
3. The fraction stores only the digits after the point. The leading 1 is not stored, because a normalized binary number always starts with 1. That free bit is called the hidden bit or implicit leading one.

Encoding 0.15625 step by step.

Step 1: convert 0.15625 to binary by repeated doubling.
  0.15625 x 2 = 0.3125  -> digit 0
  0.3125  x 2 = 0.625   -> digit 0
  0.625   x 2 = 1.25    -> digit 1, keep 0.25
  0.25    x 2 = 0.5     -> digit 0
  0.5     x 2 = 1.0     -> digit 1, keep 0.0, stop

  0.15625 = 0.00101 in binary

Step 2: normalize to 1.something
  0.00101 = 1.01 x 2^-3

Step 3: bias the exponent
  -3 + 127 = 124 = 01111100

Step 4: take the fraction bits after the leading 1
  01 followed by 21 zeros
  = 01000000000000000000000

Step 5: assemble
  S=0  exp=01111100  frac=01000000000000000000000

  0 01111100 01000000000000000000000
  = 0011 1110 0010 0000 0000 0000 0000 0000
  = 0x3E200000

Encoding -6.25 step by step.

Step 1: 6 = 110 in binary. 0.25 = 0.01 in binary.
  6.25 = 110.01

Step 2: normalize
  110.01 = 1.1001 x 2^2

Step 3: bias
  2 + 127 = 129 = 10000001

Step 4: fraction after the leading 1
  1001 followed by 19 zeros
  = 10010000000000000000000

Step 5: sign is 1 because the number is negative
  1 10000001 10010000000000000000000
  = 1100 0000 1100 1000 0000 0000 0000 0000
  = 0xC0C80000

Decoding 0x41200000 back to a number.

0x41200000 = 0100 0001 0010 0000 0000 0000 0000 0000

  sign     = 0                      -> positive
  exponent = 10000010 = 130         -> 130 - 127 = 3
  fraction = 01000000000000000000000
           = 0.25 (only the 2^-2 bit is set)

  value = +1.25 x 2^3 = 1.25 x 8 = 10.0

1. All three results can be confirmed in one line of Python: import struct; struct.pack('>f', 0.15625).hex() prints 3e200000.

PLAIN7.5.4 what is really happening inside#

1. When the exponent field is neither all zeros nor all ones, the value is normal and the hidden leading 1 applies. That is the ordinary case.
2. When the exponent field is all zeros, the hidden bit is 0 instead of 1 and the exponent is treated as -126. These are subnormal or denormal numbers, and they fill the tiny gap between the smallest normal number and zero. Without them, the gap around zero would be larger than the gap between the smallest two normal numbers, which breaks the rule that “a minus b is zero only if a equals b”.
3. When the exponent field is all zeros and the fraction is also all zeros, the value is zero. The sign bit still applies, so there is a positive zero and a negative zero. They compare as equal, but 1 divided by +0 gives positive infinity and 1 divided by -0 gives negative infinity.
4. When the exponent field is all ones and the fraction is all zeros, the value is infinity, positive or negative by the sign bit.
5. When the exponent field is all ones and the fraction is not zero, the value is NaN, meaning not a number. It is produced by 0 divided by 0, by infinity minus infinity, and by the square root of a negative number.
6. NaN has a strange and useful property: it is not equal to anything, including itself. x != x is a valid test for NaN, and many libraries implement isnan exactly that way.
7. Now the famous problem. In binary, 0.1 is a repeating fraction, exactly the way 1/3 is repeating in decimal. It is 0.0001100110011001100… forever.
8. The format has to stop somewhere, so it stores the nearest value it can. The stored value is very slightly too big.
9. 0.2 is likewise stored slightly too big. Their sum, rounded again, lands on a value slightly above the nearest stored value of 0.3.
10. So the comparison fails, not because the machine is broken, but because it is doing exactly what a finite binary format must do.

TECHNICAL7.5.5 the engineer’s version#

1. IEEE 754 was approved in 1985. William Kahan of UC Berkeley led the effort; the initial proposal was drafted in 1978 by Kahan, Jerome Coonen and Harold Stone. Kahan received the ACM Turing Award in 1989. The Intel 8087 coprocessor shipped in 1980 implementing the draft, five years before approval. The standard was revised as IEEE 754-2008 and IEEE 754-2019, and adopted internationally as ISO/IEC 60559.
2. Format parameters:

3. Ranges and step sizes:

4. Machine epsilon is the distance from 1.0 to the next representable value above it. It equals 2^-p where p is the number of stored fraction bits. It is the correct constant to use in a relative tolerance test, not an arbitrary value like 0.000001.
5. The exact stored values behind the famous example, in binary64:

0.1 stores as
  0.1000000000000000055511151231257827021181583404541015625
0.2 stores as
  0.200000000000000011102230246251565404236316680908203125
their sum rounds to
  0.3000000000000000444089209850062616169452667236328125
but the nearest double to 0.3 is
  0.299999999999999988897769753748434595763683319091796875

so (0.1 + 0.2) == 0.3 is false, and 0.1 + 0.2 prints as
0.30000000000000004

6. Rounding modes defined by the standard: round to nearest with ties to even (the default), round toward zero, round toward positive infinity, round toward negative infinity, and round to nearest with ties away from zero. Ties-to-even avoids the small upward bias that ties-away introduces.
7. Money must not use binary floating point. Reasons: 0.01 is not exactly representable, sums are order-dependent, and financial rules require specific decimal rounding. Use one of:
   1. Integers in minor units, such as paise or cents, with explicit scaling.
   2. A decimal type: decimal.Decimal in Python, BigDecimal in Java, System.Decimal in .NET, NUMERIC or DECIMAL in SQL.
   3. The IEEE 754-2008 decimal formats decimal32, decimal64 and decimal128, which are hardware-supported on IBM POWER and z/Architecture.
8. bfloat16 was developed at Google Brain and used in Tensor Processing Units from TPU v2 onward. It is binary32 with the low 16 fraction bits removed, so conversion to and from float32 is a truncate or a zero-fill. Established fact: it keeps float32’s exponent range, so it does not underflow on the very small gradient values that appear in deep networks, which is where binary16 struggles and needs loss scaling. Hardware support is now broad, including Intel Cooper Lake (2020), Armv8.6-A, and NVIDIA GPUs from the Ampere generation (2020).
9. Active research rather than settled practice: 8-bit formats. The E4M3 and E5M2 shapes proposed jointly by NVIDIA, Arm and Intel in 2022, and now carried in the Open Compute Project microscaling specifications, are in production use for inference and increasingly for training, but best practice is still moving. Treat any specific claim about FP8 training quality as version-dependent and dated.
10. Inspect bit patterns with struct.pack in Python, printf "%a" in C (which prints hexadecimal floating point), frexp and ldexp in the C library, or numpy.float32(x).view(numpy.uint32).

WORDS7.5.6 remember these#

1. Fixed point — the point never moves — an integer with an implicit constant scale factor, usually a power of two or ten.
2. Floating point — the point moves with the exponent — a sign, an exponent and a significand encoding a value as significand times base^exponent.
3. Mantissa or significand — the digits of the number — the fractional field plus the implicit leading bit for normal values.
4. Bias — a fixed offset added to the exponent — the constant 2^(k-1) - 1 that lets a k-bit exponent field stay unsigned.
5. Subnormal — very small numbers below the normal range — values with a zero exponent field and no implicit leading one, giving gradual underflow.
6. NaN — a result that is not a number — an exponent field of all ones with a non-zero significand, unequal to every value including itself.
7. Machine epsilon — the smallest step above 1.0 — 2^-p for p stored fraction bits; 1.19e-7 for binary32 and 2.22e-16 for binary64.

7.6 Text before Unicode#

PLAIN7.6.1 in simple words#

1. A computer stores numbers. To store letters, someone must publish a table saying which number means which letter.
2. That table is called a character encoding. It is only an agreement.
3. The first such agreements were for telegraphs, not computers.
4. Morse code gave each letter a pattern of short and long signals, with the common letters kept short. E is one dot.
5. Baudot code gave every letter exactly five on-off marks, so machines could handle it without a human ear.
6. IBM built its own table, EBCDIC, for its big machines.
7. Then in 1963 a committee published ASCII, which used 7 bits and 128 codes, and it slowly won.
8. ASCII covered English and nothing else. There was no space for accents, no Devanagari, no Chinese, no Arabic.
9. So every country invented its own extension using the spare eighth bit. Those extensions were called code pages.
10. A file did not say which code page it used. Open it with the wrong one and you got garbage. That mess is what Unicode was created to end.

PLAIN7.6.2 a picture in your head#

1. Imagine a phone keypad where each button number stands for a letter, and two friends each keep their own written key.
2. If both keys are identical, messages arrive fine.
3. If one friend swaps a few entries, most of the message still reads, but some words come out as nonsense.
4. Now imagine a hundred friends with a hundred slightly different keys, and no way to write on the envelope which key you used.
5. That is exactly the code page era. Text arrived as bare numbers with no label saying how to read them.

Where this comparison breaks

1. In the keypad picture, both sides know a key exists. In the real problem, most software silently assumed its own local default and never asked.
2. And guessing was often close enough to look right, which was worse than failing loudly, because the corruption was saved back to disk.

PLAIN7.6.3 a worked example#

The clever parts of the ASCII layout.

1. The digits sit at 0x30 to 0x39, so the low four bits are the digit value. To turn the character ‘7’ into the number 7, mask with 0x0F.
2. ‘A’ is 65 and ‘a’ is 97. The difference is 32, which is a single bit, bit 5.

'A' = 1000001
'a' = 1100001
       ^
       this one bit is the whole difference

To lowercase:  c OR  0x20
To uppercase:  c AND 0xDF
To flip case:  c XOR 0x20

3. That is why case conversion on ASCII is one instruction, not a lookup.
4. Control characters occupy 0 to 31. They line up with letters too. The Ctrl key clears bits 6 and 5, so Ctrl-I is 9 (tab), Ctrl-M is 13 (carriage return), Ctrl-J is 10 (line feed), Ctrl-C is 3 (end of text).
5. DEL is 127, all seven bits set. On paper tape you deleted a character by punching every hole, which no other code could be mistaken for.
6. Uppercase comes before lowercase, and digits before both, so sorting by byte value roughly matches alphabetical order for plain English.

PLAIN7.6.4 what is really happening inside#

1. A text file is a sequence of bytes. There is no letter anywhere in it.
2. When you open the file, a program picks a decoding table, looks up each byte, and asks a font for a shape to draw.
3. Three separate things are involved and people confuse them constantly:
   1. The code, meaning which number stands for which character.
   2. The encoding, meaning how that number is packed into bytes.
   3. The font, meaning what the shape looks like on screen.
4. For 7-bit ASCII the code and the encoding are almost the same thing, because every character fits in one byte with a spare top bit.
5. That spare top bit is where the trouble started. It doubled the table from 128 to 256 entries, and nobody agreed on the second half.
6. Russian text encoded in KOI8-R and read as Windows-1252 produces Latin letters and punctuation. Nothing crashes. The bytes are all valid. Only the meaning is lost.
7. The honest version: some code pages were also stateful, using shift codes to switch between banks of characters. Baudot did this with LTRS and FIGS, and some Asian encodings did it with escape sequences. In a stateful encoding you cannot decode a byte without knowing the state, so you cannot start reading in the middle of a file.

TECHNICAL7.6.5 the engineer’s version#

1. Morse code was developed by Samuel Morse and Alfred Vail through the late 1830s; the famous line “What hath God wrought” was sent on 24 May 1844. It is a variable-length code with frequency-weighted symbols, an early instance of the idea Huffman later formalized. It is not binary: it has dot, dash, and three distinct gap lengths.
2. Baudot code was devised by Emile Baudot around 1870 and patented in 1874. Five bits give 32 combinations, extended by LTRS and FIGS shift codes. Donald Murray revised it in 1901; the result became International Telegraph Alphabet No. 2, standardized by the CCITT in 1930. The unit “baud” is named after Baudot.
3. EBCDIC (Extended Binary Coded Decimal Interchange Code) was created by IBM for the System/360, announced 7 April 1964. It is 8-bit and derived from punched-card BCD. Its letters are not contiguous: A to I occupy 0xC1 to 0xC9, J to R occupy 0xD1 to 0xD9, S to Z occupy 0xE2 to 0xE9. Code that tests c >= 'A' && c <= 'Z' is wrong on EBCDIC.
4. ASCII was published as ASA X3.4-1963 on 17 June 1963 by the American Standards Association. Work began at the first meeting of the X3.2 subcommittee on 6 October 1960. Bob Bemer was a leading contributor. Revisions followed as USAS X3.4-1967 and ANSI X3.4-1986. The international equivalents are ECMA-6 and ISO/IEC 646.
5. Common 8-bit code pages:

6. ISO 8859-15 arrived in 1999 mainly to add the euro sign, which ISO 8859-1 had no room for. That single missing character forced a new standard, which tells you how badly the fixed 256-slot model had run out.
7. Windows-1252 is not ISO 8859-1, although it is often labelled as such. It places printable characters in the C1 control range 0x80 to 0x9F, including the curly quotes and the em dash. Mislabelled Windows-1252 content is the single most common source of stray question marks on the web.
8. Tools: iconv -f WINDOWS-1252 -t UTF-8, file -i, chardet and charset-normalizer in Python, enca.

WORDS7.6.6 remember these#

1. Character encoding — the table linking numbers to letters — a mapping from a character repertoire to byte sequences.
2. ASCII — the 1963 English 7-bit table — ASA X3.4-1963, 128 code positions, 33 control characters and 95 printable characters.
3. Code page — a national extension of the top 128 byte values — a vendor or national single-byte character set covering 0x80 to 0xFF.
4. Control character — a code that means an action, not a shape — ASCII 0x00 to 0x1F plus 0x7F, used for framing, flow control and terminals.
5. EBCDIC — IBM’s rival table — an 8-bit punched-card-derived encoding used on IBM mainframes, with non-contiguous alphabetic ranges.
6. Stateful encoding — you must know what mode you are in — an encoding whose byte meanings depend on preceding shift or escape sequences.

7.7 Unicode and UTF-8#

PLAIN7.7.1 in simple words#

1. Unicode’s idea is simple: give every character in every writing system its own permanent number, once, for everyone.
2. That number is called a code point. It is written as U+ then hex, for example U+0041 for the letter A.
3. Unicode assigns the numbers. It does not say how to store them in bytes. That is a separate decision.
4. There are three main storage schemes: UTF-8, UTF-16 and UTF-32.
5. UTF-8 uses one to four bytes per character. English text takes one byte per letter, exactly like ASCII.
6. That backward compatibility is why UTF-8 took over the world. Every old ASCII file was already a valid UTF-8 file, unchanged.
7. UTF-16 uses two bytes for most characters and four for the rest.
8. UTF-32 uses four bytes for everything, always.
9. Once you use these, the old question “which code page is this file?” disappears. There is one table for the whole planet.

PLAIN7.7.2 a picture in your head#

1. Think of a global postal system where every building on Earth gets one permanent number, and nobody ever reuses a number.
2. Unicode is that numbering scheme. U+0915 always means the Devanagari letter ka, in every country, forever.
3. Now think of how you write that number on an envelope. You could always use a fixed seven-digit box, wasting space on short numbers.
4. Or you could use a variable scheme where short numbers get short writing and long numbers get more digits, with a marker telling the reader how many digits to expect.
5. That second scheme is UTF-8. The first few bits of the first byte announce the length of the whole character.

Where this comparison breaks

1. Postal numbers name one thing each. A visible character on screen is often several code points glued together, and the glue is itself a code point.
2. And postal numbers are never reordered. Unicode has to handle text that runs right-to-left, and combining marks that attach to the character before them, so visual order and stored order differ.

PLAIN7.7.3 a worked example#

The UTF-8 rules in one table.

1. The full 4-byte template is 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx, which carries 3 + 6 + 6 + 6 = 21 bits.

Character one: the English letter A, U+0041.

0x41 = 65, which is under 128, so one byte.
Result: 41

Character two: the Devanagari letter ka, U+0915.

0x915 in binary, padded to 16 bits:
  0000 1001 0001 0101

U+0915 is in the 3-byte range, template 1110xxxx 10xxxxxx 10xxxxxx
That carries 4 + 6 + 6 = 16 bits, so use all 16.

Split 0000100100010101 as  0000 | 100100 | 010101

byte 1 = 1110 0000 = 0xE0
byte 2 = 10 100100 = 0xA4
byte 3 = 10 010101 = 0x95

Result: E0 A4 95

Character three: the grinning face emoji, U+1F600.

0x1F600 in binary, padded to 21 bits:
  0 0001 1111 0110 0000 0000
= 000011111011000000000

U+1F600 is in the 4-byte range, carrying 3 + 6 + 6 + 6 = 21 bits.

Split as  000 | 011111 | 011000 | 000000

byte 1 = 11110 000 = 0xF0
byte 2 = 10 011111 = 0x9F
byte 3 = 10 011000 = 0x98
byte 4 = 10 000000 = 0x80

Result: F0 9F 98 80

2. Confirm any of these with python3 -c "print('A'.encode('utf-8').hex())".

PLAIN7.7.4 what is really happening inside#

1. Look again at the templates. Every continuation byte starts with the bits
   10. No leading byte ever starts with 10.
2. So if you drop into the middle of a UTF-8 stream, you can find a character boundary by walking backwards until you hit a byte that does not start with
   10. You never scan back more than three bytes.
3. That property is called self-synchronising, and it is the reason UTF-8 beat the alternatives. A corrupted or truncated stream loses one character, not the rest of the file.
4. Second property: a byte below 0x80 is always a real ASCII character and never part of a multi-byte sequence.
5. That means old code searching for the byte 0x2F (a slash) or 0x00 (a string terminator) still works correctly on UTF-8 text without changes. This is what “file system safe” meant in the original name.
6. Third property: sorting UTF-8 byte strings gives the same order as sorting by code point. That is not true of UTF-16.
7. UTF-16 handles characters above U+FFFF with a pair of 16-bit units called a surrogate pair. Two thousand and forty-eight code points, U+D800 to U+DFFF, are permanently reserved for this and are never characters.
8. The arithmetic: subtract 0x10000 from the code point, leaving 20 bits. The top 10 bits go into 0xD800 plus that value, the bottom 10 into 0xDC00 plus that value.
9. For U+1F600: 0x1F600 minus 0x10000 is 0xF600. The high half is 0xD800 plus 0x3D, which is 0xD83D. The low half is 0xDC00 plus 0x200, which is 0xDE00.
10. This is why a JavaScript or Java string reports a length of 2 for a single emoji. Those languages count 16-bit units, not characters.

TECHNICAL7.7.5 the engineer’s version#

1. Unicode 1.0 was published in October 1991, following Joe Becker’s 1988 draft “Unicode 88”. The Unicode Consortium was incorporated in January 1991. Unicode 2.0, in July 1996, broke the original 16-bit assumption and added surrogates. Unicode 17.0 was released on 9 September 2025 and contains 159,801 assigned characters.
2. The code space is U+0000 to U+10FFFF: 1,114,112 code points in 17 planes of 65,536 each. Plane 0 is the Basic Multilingual Plane. Plane 1 is the Supplementary Multilingual Plane, holding emoji and historic scripts. Plane 2 holds CJK extensions. Planes 15 and 16 are private use.
3. The ceiling of U+10FFFF is not a natural limit. It exists because that is the largest value UTF-16 surrogate pairs can express. UTF-8’s own structure could have reached U+1FFFFF or, in the original 1992 design, six bytes and 31 bits. RFC 3629 cut it back to four bytes to match UTF-16.
4. UTF-8 was designed by Ken Thompson and Rob Pike in September 1992. Pike’s account states it was worked out “on a placemat in a New Jersey diner”, the scheme was described to the X/Open committee on 8 September 1992, and the whole of Plan 9 was converted within days. The original name was FSS-UTF, for File System Safe UCS Transformation Format. Pike and Thompson presented it at the USENIX Winter 1993 conference in San Diego.
5. The current specification is RFC 3629, published November 2003 by Francois Yergeau, which is also STD 63. It obsoletes RFC 2279 (1998), which obsoleted RFC 2044 (1996).
6. RFC 3629 forbids overlong encodings: a code point must use the shortest template that fits. Accepting overlongs was a real security hole, exploited against Microsoft IIS in 2001 to smuggle a directory-traversal slash past a filter that checked for 0x2F only.
7. Encoding sizes for one character:

8. Byte order marks. U+FEFF at the start of a stream signals encoding and byte order: FE FF for UTF-16BE, FF FE for UTF-16LE, 00 00 FE FF for UTF-32BE, FF FE 00 00 for UTF-32LE. In UTF-8 it appears as EF BB BF. The Unicode standard permits but does not recommend a UTF-8 BOM; it breaks shell scripts, PHP output, and many CSV parsers. Note the ambiguity: FF FE alone is UTF-16LE, but FF FE 00 00 is UTF-32LE, so a decoder must look further.
9. Combining characters and normalization, defined in Unicode Standard Annex #15. The character e-acute has two spellings:

precomposed:  U+00E9                      UTF-8: C3 A9
decomposed:   U+0065 U+0301               UTF-8: 65 CC 81

They render identically. They are not equal as byte strings
and not equal under a naive == comparison.

10. The four normalization forms are NFC (canonical composition), NFD (canonical decomposition), NFKC and NFKD (compatibility forms, which also fold the ligature fi to “fi” and the circled digit one to “1”). NFC is the web default; the W3C Character Model recommends it. Compatibility forms lose information and must never be used for round-tripping.
11. macOS HFS+ historically stored filenames in a variant of NFD, while Linux stores whatever bytes it is given. That mismatch broke real Git repositories containing accented or Indic filenames; git config core.precomposeunicode true exists for exactly this reason.
12. String length has at least four correct answers. For the four-person family emoji, built from four people joined by three zero-width joiners (U+200D):

13. Grapheme cluster rules live in Unicode Standard Annex #29 and change between versions. Unicode 15.1, released September 2023, added rule GB9c so that Indic conjunct sequences such as the Devanagari cluster in “namaste” group correctly. So even “how many characters is this” depends on which Unicode version your library implements. Python’s len counts code points, Go’s len counts bytes, Java’s String.length counts UTF-16 units, Swift’s count counts grapheme clusters. Four languages, four different answers, all defensible.
14. Tools: unicode and uniname on Linux, python3 -c "import unicodedata; print(unicodedata.name(ch))", iconv, hexdump -C, and the ICU library.

WORDS7.7.6 remember these#

1. Code point — the permanent number for a character — an integer in U+0000 to U+10FFFF assigned by the Unicode Standard.
2. UTF-8 — one to four bytes per character, ASCII-compatible — the variable width encoding defined by RFC 3629 and STD 63.
3. Surrogate pair — two halves standing for one big character — a high surrogate in U+D800 to U+DBFF followed by a low surrogate in U+DC00 to U+DFFF, encoding a code point above U+FFFF in UTF-16.
4. BOM — an invisible marker at the file start — U+FEFF used to signal encoding and byte order.
5. Mojibake — text turned to garbage by the wrong table — the result of decoding bytes with an encoding other than the one used to produce them.
6. Grapheme cluster — what a person calls one character — a sequence of code points treated as one unit by the rules in Unicode Annex #29.
7. Normalization — putting equal-looking text into one standard spelling — the NFC, NFD, NFKC and NFKD transformations of Unicode Annex #15.

7.8 Endianness#

PLAIN7.8.1 in simple words#

1. A number bigger than one byte must be split across several bytes.
2. There are two sensible orders to write them in, and both are in use.
3. Big-endian writes the biggest part first, the way we write numbers.
4. Little-endian writes the smallest part first, backwards from how we read.
5. Neither is better. They are two conventions that both work.
6. The trouble comes when a big-endian machine sends bytes to a little-endian machine and nobody converts. The number arrives scrambled.
7. So network protocols pick one order and everyone must obey it. They picked big-endian, and it is called network byte order.
8. Nearly every computer you own runs little-endian internally, so it converts on the way out and on the way in.

PLAIN7.8.2 a picture in your head#

1. Think of writing the date. Some people write 13/08/2026, day first. Others write 2026-08-13, year first.
2. Both write the same date. Both are unambiguous once you know the rule.
3. Give a form filled in one way to a reader expecting the other, and 08/13 becomes an impossible 13th month, or worse, a plausible wrong date.
4. Endianness is the same problem at the byte level, and computers do not notice that a date is impossible. They just use the wrong number.

Where this comparison breaks

1. Date formats are ambiguous forever. Byte order is fixed by the machine and by the protocol specification, so it is always knowable.
2. Also, little-endian is not simply “backwards”. It has a real advantage: the low byte sits at the lowest address, so reading a 32-bit value as an 8-bit value needs no address adjustment.

PLAIN7.8.3 a worked example#

Store the 32-bit value 0x12345678 at address 0x1000.

BIG-ENDIAN (most significant byte at the lowest address)

  address: 0x1000 0x1001 0x1002 0x1003
  byte:      12     34     56     78
             ^ biggest part first

LITTLE-ENDIAN (least significant byte at the lowest address)

  address: 0x1000 0x1001 0x1002 0x1003
  byte:      78     56     34     12
             ^ smallest part first

1. The number is identical in both. Only the layout in memory differs.
2. A real dump from a little-endian machine looks like this:

$ python3 -c "import sys; sys.stdout.buffer.write(
    (0x12345678).to_bytes(4,'little'))" | hexdump -C

00000000  78 56 34 12                    |xV4.|

3. If you send those four bytes to a big-endian reader without conversion, it reads 0x78563412, which is 2,018,915,346 instead of 305,419,896.

PLAIN7.8.4 what is really happening inside#

1. The CPU does not store a number “the wrong way round”. It stores it the only way its memory interface is wired.
2. Registers have no endianness. A 64-bit register is just 64 bits. Endianness only exists when a multi-byte value touches byte-addressed memory, a file, or a wire.
3. When a program writes a 32-bit integer, the store instruction splits it and places the bytes in the architecture’s fixed order.
4. When it reads it back on the same machine, the load reverses that split perfectly. So within one machine, endianness is invisible.
5. It becomes visible the moment bytes cross a boundary: a network socket, a file shared between machines, or a memory-mapped device.
6. That is why protocol code is full of conversion calls. On a little-endian host htonl swaps the bytes. On a big-endian host the same function does nothing at all and compiles away.
7. This is also why casting a byte buffer straight into a struct pointer is a portability bug, even though it usually works on the machine you tested on.

TECHNICAL7.8.5 the engineer’s version#

1. The terms come from Danny Cohen’s memo “On Holy Wars and a Plea for Peace”, Internet Experiment Note 137, dated 1 April 1980, later published in IEEE Computer in October 1981. Cohen borrowed Big-Endians and Little-Endians from Jonathan Swift’s Gulliver’s Travels of 1726, where the dispute is over which end of a boiled egg to open.
2. Architecture byte orders:

3. ARM, PowerPC, MIPS and RISC-V are bi-endian, meaning the mode is selectable. In practice ARM Linux, Android, iOS and macOS all run little-endian, and Linux on POWER moved to little-endian with the ppc64le port around POWER8 in 2014.
4. Network byte order is big-endian, fixed by the Internet Protocol specification, RFC 791 (September 1981), and restated in RFC 1700. The conversion functions are htons, htonl, ntohs, ntohl in <arpa/inet.h>, plus htobe32 and friends in <endian.h> on Linux.
5. File formats also pick a side. PNG, JPEG, Java class files and most internet formats are big-endian. ZIP, GIF, BMP and RIFF or WAV are little-endian. TIFF is explicitly both: the file starts with “II” for Intel little-endian or “MM” for Motorola big-endian.
6. There is also middle-endian, seen on the PDP-11, which stored 32-bit values as two 16-bit words in big-endian order with each word little-endian inside. It is why “PDP-endian” is a term.
7. Bit order within a byte is a separate question, handled by the hardware serializer rather than by software.
8. Observe with lscpu | grep -i "byte order", sysctl hw.byteorder on macOS, python3 -c "import sys; print(sys.byteorder)", and the C macro __BYTE_ORDER__ in GCC and Clang.

WORDS7.8.6 remember these#

1. Endianness — which end of a number goes first in memory — the byte ordering convention for multi-byte scalar values.
2. Big-endian — biggest byte at the lowest address — most significant byte first, the order used by network protocols.
3. Little-endian — smallest byte at the lowest address — least significant byte first, used by x86-64, ARM and RISC-V.
4. Network byte order — the order the internet insists on — big-endian, fixed by RFC 791 for all IP header fields.
5. Byte swap — reversing the order — the bswap instruction on x86 or the REV instruction on ARM, one cycle in hardware.

7.9 Checking that data is intact#

PLAIN7.9.1 in simple words#

1. Bits get flipped. Cables pick up noise, memory cells lose charge, disks develop weak spots, cosmic rays strike chips.
2. A flipped bit is silent. Nothing complains. The data is simply wrong.
3. So we send extra information alongside the data, computed from the data itself, and check that it still matches.
4. The simplest is a parity bit: one extra bit making the number of 1s even. If it comes out odd, something changed.
5. A checksum is slightly better: add all the bytes and send the total.
6. A CRC is much stronger. It uses division rather than addition, and it catches whole runs of damaged bits.
7. None of these stop a deliberate attacker, who can simply recompute the check value. For that you need a cryptographic hash, designed so that nobody can find a second message with the same value.
8. ECC memory goes further still. It stores enough extra bits to spot a single flipped bit and repair it on the fly.

PLAIN7.9.2 a picture in your head#

1. A shop counts a delivery of boxes and writes the total on the paperwork.
2. If one box goes missing, the count disagrees and the error is caught. That is a checksum.
3. If one box goes missing and someone adds a different box, the count still matches. The check passed and the delivery is still wrong.
4. Now suppose the paperwork records not just the count but the weight, the volume, and a code derived from the serial numbers in order.
5. Now swapping one box for another almost certainly disagrees somewhere. That is roughly what a CRC does.
6. And suppose there is enough redundant paperwork to work out exactly which box is missing and reorder it automatically. That is ECC.

Where this comparison breaks

1. A shop clerk can think and investigate. A CRC only ever says “matches” or “does not match”. It never says which bit changed or how to fix it.
2. And the paperwork analogy suggests deliberate fraud is hard. With a CRC it is trivially easy, because the formula is public and reversible.

PLAIN7.9.3 a worked example#

Parity on one byte.

Data byte:  1 0 1 1 0 1 0 0     number of 1s = 4, which is even
Even parity bit = 0   ->  sent as  10110100 0

Now flip one bit in transit:
Received:   1 0 1 1 0 1 1 0  0   number of 1s = 5, which is odd
The parity bit says it should be even -> error detected

Now flip two bits:
Received:   1 0 1 1 1 1 1 0  0   number of 1s = 6, still even
Parity says fine. The error is missed.

1. Parity detects any odd number of flipped bits and misses every even number.
2. It can never repair anything, because it does not know which bit moved.

A real CRC32 value you can reproduce:

$ python3 -c "import zlib; print(hex(zlib.crc32(b'123456789')))"
0xcbf43926

3. That value, 0xCBF43926 for the nine ASCII digits “123456789”, is the standard published check value for CRC-32. Any correct implementation produces it. It is the first thing to test when you write one.

PLAIN7.9.4 what is really happening inside#

1. A CRC treats the whole message as one enormous binary number.
2. It divides that number by a fixed constant and keeps only the remainder. The remainder is the CRC value.
3. The division is not ordinary division. It uses XOR instead of subtraction, so there are no carries and no borrows.
4. In practice you do it by shifting the message through a register, and every time a 1 falls off the top you XOR the register with the constant.
5. The constant is called the polynomial, because mathematicians describe the same operation as polynomial division over the field with two elements. The name is historical; you can implement it knowing only XOR and shift.
6. The strength comes from that division. Adding bytes lets errors cancel out. Division spreads every input bit’s influence across the whole remainder.
7. ECC memory works differently. It computes several overlapping parity bits, each covering a different subset of the data bits.
8. When a bit flips, the pattern of which parity checks fail points directly at the position of the flipped bit. Flip it back and the data is repaired.
9. This is a Hamming code. Adding one more overall parity bit lets it also detect, without correcting, a double flip. That combination is called SECDED, single error correct, double error detect.

TECHNICAL7.9.5 the engineer’s version#

1. Parity is used in UART serial framing, written as 8N1 for eight data bits, no parity, one stop bit, or 7E1 for seven data bits, even parity, one stop bit. Hamming distance 2: it detects one error, corrects none.
2. The Internet checksum, specified in RFC 1071, is the 16-bit one’s complement of the one’s complement sum of 16-bit words. It protects the IPv4 header, TCP and UDP. It is weak: it cannot detect a reordering of 16-bit words, and it misses many pairs of compensating errors.
3. The CRC idea was published by W. Wesley Peterson in 1961. CRC-32 as used in Ethernet (IEEE 802.3), PNG, gzip and ZIP uses the polynomial 0x04C11DB7 in normal notation, or 0xEDB88320 reflected, with an initial value of all ones and a final XOR of all ones.
4. Guaranteed properties of a 32-bit CRC with a well-chosen polynomial:
   1. Detects all single-bit and all double-bit errors within its bound.
   2. Detects any odd number of bit errors, if the polynomial includes the factor (x + 1).
   3. Detects all burst errors up to 32 bits long.
   4. Misses longer bursts with probability 2^-32, about one in 4.3 billion.
5. Common CRC variants:

6. CRC-32C has a dedicated x86 instruction, crc32, added with SSE4.2 in the Intel Nehalem generation (2008), which is why modern filesystems chose it.
7. Cryptographic hashes solve a different problem: collision resistance against an adversary. Named here and covered properly in the security chapter: MD5 (1992, collisions demonstrated in 2004), SHA-1 (1995, collision demonstrated by Google and CWI Amsterdam in February 2017, the SHAttered attack), the SHA-2 family including SHA-256 (2001), SHA-3 based on Keccak (standardized 2015), and BLAKE3 (2020). A CRC is not a hash and must never be used where tampering is possible.
8. ECC DRAM adds 8 check bits per 64 data bits, so a registered ECC DIMM is 72 bits wide rather than 64. Standard SECDED corrects one bit per 64-bit word and detects two. Chipkill, called Advanced ECC or SDDC by different vendors, arranges the code so that a whole failed DRAM device is correctable. The underlying theory is Richard Hamming’s 1950 paper “Error Detecting and Error Correcting Codes” in the Bell System Technical Journal.
9. DDR5, specified in JEDEC JESD79-5 published July 2020, mandates on-die ECC inside the DRAM chip. Be careful: on-die ECC protects the array, not the bus or the connector. It is not a substitute for full ECC, and marketing material sometimes blurs the two.
10. Real failure rates, from “DRAM Errors in the Wild: A Large-Scale Field Study” by Schroeder, Pinheiro and Weber, published at SIGMETRICS 2009, covering Google’s fleet from January 2006 to June 2008:

11. That study overturned the belief that memory errors were rare and caused mainly by cosmic rays. Errors correlated strongly with specific DIMMs, pointing at hard faults rather than random strikes.
12. Observe ECC events with edac-util -v, ras-mc-ctl --summary, dmesg | grep -i edac, or mcelog on Linux.

WORDS7.9.6 remember these#

1. Parity bit — one extra bit making the count of 1s even or odd — a distance-2 code detecting any odd number of bit errors.
2. Checksum — add everything up and compare — a weak integrity value, often a one’s complement sum as in RFC 1071.
3. CRC — a remainder from carry-less division — a cyclic redundancy check, the remainder of the message polynomial divided by a generator polynomial over GF(2).
4. ECC memory — memory that repairs single flipped bits — DRAM with SECDED Hamming coding, 8 check bits per 64 data bits.
5. SECDED — fix one error, spot two — single error correct, double error detect, the standard server DRAM protection level.
6. Collision resistance — nobody can forge a match — the property that distinguishes a cryptographic hash from a checksum or CRC.

7.10 Compression#

PLAIN7.10.1 in simple words#

1. Real data repeats itself. English text is full of “the”. Photographs have large areas of near-identical sky. Compression finds the repetition and writes it down once instead of many times.
2. It also exploits unevenness. In English ‘e’ is far more common than ‘z’, so giving both the same eight bits is wasteful.
3. Short codes for common things and long codes for rare things shrink the total. That is the second big idea.
4. Lossless compression returns the exact original bytes. ZIP, PNG and gzip are lossless.
5. Lossy compression throws away detail people are unlikely to notice and cannot give the original back. JPEG and MP3 are lossy.
6. Never use lossy compression on text, code or a bank record. One changed character can change the meaning completely.
7. And no scheme shrinks everything. Compressing an already compressed file usually makes it very slightly larger.

PLAIN7.10.2 a picture in your head#

1. Think of packing a suitcase. Folding clothes flat removes trapped air. The clothes are unchanged; you removed only the wasted space. That is lossless.
2. Now think of leaving three of your five shirts at home. The suitcase is far lighter, but those shirts are gone. That is lossy.
3. Now think of shorthand. A stenographer writes one squiggle for a whole common word and spells out the rare ones. That is entropy coding.
4. And think of a note saying “same as page 4, lines 2 to 9”. That is dictionary compression: pointing back instead of repeating.

Where this comparison breaks

1. A suitcase can always be packed a bit tighter with effort. Data has a hard mathematical floor, the entropy, below which no lossless scheme can go.
2. And unfolding a suitcase is easy either way. Decompression cost varies a great deal between schemes, which is often the deciding factor.

PLAIN7.10.3 a worked example#

Run-length encoding. Replace each run with a count and a value.

Input  (53 characters):
WWWWWWWWWWWWBWWWWWWWWWWWWBBBWWWWWWWWWWWWWWWWWWWWWWWWB

Output (15 characters):
12W1B12W3B24W1B

53 down to 15, a saving of about 72 percent.

1. But run-length encoding on text with no runs makes it twice as long, since every single character becomes a count plus a character.

Huffman coding, worked completely on the word “abracadabra”.

Symbol counts: a=5  b=2  r=2  c=1  d=1     total 11 symbols

Step 1: join the two smallest, c(1) and d(1)      -> node cd = 2
Step 2: smallest are now b(2), r(2), cd(2).
        join b(2) and r(2)                        -> node br = 4
Step 3: join cd(2) and br(4)                      -> node X = 6
Step 4: join a(5) and X(6)                        -> root  = 11

The finished tree, with 0 for left and 1 for right:

                 [11]
                /    \
             0 /      \ 1
          a(5)         [6]
                      /   \
                   0 /     \ 1
               cd[2]        br[4]
               /   \        /   \
            0 /     \ 1  0 /     \ 1
          c(1)     d(1) b(2)     r(2)

The code table read off the tree:

Encoding the whole word:

 a   b    r    a   c    a   d    a   b    r    a
 0  110  111   0  100   0  101   0  110  111   0

= 01101110100010101101110      (23 bits)

Plain ASCII would need 11 x 8 = 88 bits.
A fixed 3-bit code for 5 symbols would need 11 x 3 = 33 bits.
Huffman needs 23 bits.

2. No code is a prefix of any other code, so the decoder never needs separators. It walks down the tree, emits a symbol at each leaf, and returns to the root.

PLAIN7.10.4 what is really happening inside#

1. Huffman coding builds the tree from the bottom by always joining the two least frequent items. Rare symbols therefore end up deepest, with the longest codes. This is provably optimal for whole-bit codes.
2. LZ77 works on a completely different principle. It keeps a window of the recently seen bytes and, whenever the upcoming text has appeared before, it emits a back-reference instead of the text.
3. A back-reference is a pair: how far back to look, and how many bytes to copy. For example, “go back 3 bytes and copy 9”.
4. The length may exceed the distance. Copying is done one byte at a time, so “back 3, copy 9” on “abc” produces “abcabcabc”. That is how LZ77 also does run-length encoding for free.
5. DEFLATE is simply both together. First LZ77 replaces repeats with back-references. Then Huffman coding compresses the resulting stream of literals, lengths and distances.
6. That two-stage design is why gzip, ZIP and PNG all behave alike: they are the same algorithm with different containers around it.
7. The mathematical floor is entropy. For a source where symbol x has probability p(x), the entropy in bits per symbol is:

H = - sum over all x of  p(x) x log2 p(x)

8. In words: it is the average number of bits genuinely needed per symbol. No lossless method can average fewer than H bits per symbol over the long run.
9. For “abracadabra” the entropy is 2.04 bits per symbol, so 22.4 bits for the whole word. Huffman achieved 23 bits. It came within one bit of the theoretical floor, which is exactly the guarantee Huffman coding offers.

TECHNICAL7.10.5 the engineer’s version#

1. David A. Huffman published “A Method for the Construction of Minimum-Redundancy Codes” in the Proceedings of the IRE in September 1952, written as a term paper for Robert Fano’s information theory class at MIT. Huffman coding is optimal among prefix codes with integer code lengths; it is never worse than H + 1 bits per symbol.
2. Arithmetic coding and its modern relative asymmetric numeral systems, or ANS, published by Jarek Duda from 2009, break the whole-bit restriction and reach the entropy bound more closely. ANS is what makes Zstandard fast.
3. Jacob Ziv and Abraham Lempel published “A Universal Algorithm for Sequential Data Compression” in IEEE Transactions on Information Theory in May 1977, giving LZ77. Their 1978 follow-up gave LZ78, from which LZW derives.
4. DEFLATE was created by Phil Katz for PKZIP 2.0 in 1993 and specified by L. Peter Deutsch in RFC 1951, May 1996. The related containers are RFC 1950 for zlib and RFC 1952 for gzip. DEFLATE uses a 32 KiB sliding window and match lengths of 3 to 258 bytes.
5. Claude Shannon defined entropy in “A Mathematical Theory of Communication”, Bell System Technical Journal, July and October 1948. The source coding theorem in that paper is the statement that H is a hard lower bound.
6. Modern general-purpose compressors:

7. Brotli is specified in RFC 7932 (2016) and includes a built-in 120 KiB dictionary of common web strings. Zstandard, created by Yann Collet at Facebook, is specified in RFC 8878 (February 2021) and is now the default in Linux kernel modules, Btrfs and many package managers.
8. The counting argument for why universal compression is impossible: there are 2^n distinct inputs of length n and only 2^n - 1 possible outputs shorter than n bits in total across all shorter lengths. A lossless injective map cannot fit them all. Therefore any scheme that shrinks some inputs must expand others. DEFLATE’s stored block mode caps the expansion at about 5 bytes per 65,535-byte block.
9. Lossless versus lossy, with real formats: lossless are PNG, FLAC, ALAC, gzip, ZIP, TIFF with LZW; lossy are JPEG, WebP in lossy mode, MP3, AAC, Opus, H.264, H.265, AV1. Image and audio codecs are covered in Chapters 8 and 9; the compression machinery underneath them is what you have just read.
10. Measure with gzip -9 -c file | wc -c, zstd -19, xz -9e, and ent for a quick entropy estimate of a file.

WORDS7.10.6 remember these#

1. Lossless — you get the exact bytes back — a reversible encoding, as in DEFLATE, PNG and FLAC.
2. Lossy — some detail is discarded forever — an irreversible encoding tuned to human perception, as in JPEG and MP3.
3. Run-length encoding — write “twelve whites” instead of twelve whites — a code replacing each maximal run with a count and a value.
4. Huffman coding — short codes for common symbols — an optimal prefix code built bottom-up by repeatedly merging the two least frequent nodes.
5. Prefix code — no code is the start of another — a uniquely decodable code requiring no separators between symbols.
6. LZ77 — point back to text you already sent — sliding-window dictionary compression emitting (distance, length) back-references.
7. Entropy — the true information content — H = -sum p(x) log2 p(x) bits per symbol, the lower bound on lossless compression.

7.11 How a number becomes a thing#

PLAIN7.11.1 in simple words#

1. Here is the deepest idea in this chapter, and it is short: a pattern of bits means nothing at all by itself.
2. The same 32 bits can be a number, a fraction, four letters, a colour or a machine instruction.
3. Nothing in memory says which. There is no tag, no label, no type stored alongside the data.
4. The meaning comes entirely from what the program decides to do with it. If the program treats those bits as a number, they are a number. If it treats them as text, they are text.
5. That is why file formats begin with magic numbers, why network protocols have header fields, and why programming languages have types. Each is a way of writing down, somewhere, what the bits are supposed to mean.

PLAIN7.11.2 a picture in your head#

1. Think of the digits 0812. On a door they are a house number. On a form they are a date. On a receipt they are a price. On a lock they are a code.
2. The digits never change. Where you find them, and what you were expecting, decides what they mean.
3. Now remove all context. Someone hands you a slip of paper with 0812 and walks away. You genuinely cannot know.
4. Raw memory is that slip of paper. Every byte, every time.

Where this comparison breaks

1. A person seeing 0812 on a receipt can notice the context is wrong and ask.
2. A program cannot. It applies whatever interpretation it was written with, silently, and produces a confident wrong answer. That failure mode is called type confusion and it is a common security vulnerability class.

PLAIN7.11.3 a worked example#

One pattern, five readings. The bits are 0x41424344.

The 32 bits:
  0100 0001 0100 0010 0100 0011 0100 0100

Read as a 32-bit unsigned integer      -> 1,094,861,636
Read as a 32-bit signed integer        -> 1,094,861,636
Read as an IEEE 754 float32            -> 12.141422
Read as four ASCII bytes               -> A B C D
Read as R,G,B,A colour bytes           -> 65, 66, 67, 68
Read as two 16-bit big-endian integers -> 16706 and 17220

1. The float reading works out as sign 0, exponent 0x82 which is 130, giving 2^3, and a fraction of 0.5176778, so 1.5176778 times 8 is 12.141422.
2. The colour reading gives a very dark, slightly blue grey. In web notation the first three bytes are the colour #414243.
3. If the same four bytes are read on a little-endian machine as one 32-bit integer loaded from memory in the other order, the value becomes 0x44434241, which is 1,145,258,561. A sixth reading, from byte order alone.

You can reproduce all of it:

import struct
p = 0x41424344
b = struct.pack('>I', p)
print(struct.unpack('>I', b)[0])   # 1094861636
print(struct.unpack('>f', b)[0])   # 12.141422271728516
print(b.decode('ascii'))           # ABCD
print(list(b))                     # [65, 66, 67, 68]

PLAIN7.11.4 what is really happening inside#

1. Memory is a flat array of bytes with addresses. It carries no type information whatsoever.
2. The type lives in the compiled code, not in the data. When the compiler saw int x, it emitted an integer load instruction for every use of x.
3. If the same address is later read through a float pointer, the CPU issues a floating point load, and the floating point unit interprets the exponent and fraction fields of whatever is there.
4. Neither instruction checks anything. Both succeed. Only one is meaningful.
5. A cast in C changes nothing in memory. It changes which instruction the compiler emits.
6. A C union deliberately overlaps two types at one address, which is the controlled version of the same trick.
7. This is also why the same bytes can be executed. Jump to an address and the CPU fetches those bytes into the instruction decoder instead of a register. Attacks that redirect execution into data exploit exactly this.
8. The honest version: some systems do tag data in hardware. The Burroughs B5000 and Lisp machines tagged words; CHERI, developed at Cambridge and Arm and shipped in the Arm Morello prototype in 2022, tags pointers with capability metadata. These are real but exceptional. Mainstream hardware stores untagged bits.

TECHNICAL7.11.5 the engineer’s version#

1. Type is a static property of a program, not a runtime property of memory, on all mainstream hardware. C’s object model calls the intended reading the effective type of the storage.
2. Reading an object through a pointer of an incompatible type violates the strict aliasing rule in C and C++ and is undefined behaviour. The legal way to reinterpret bits is memcpy, or std::bit_cast in C++20, both of which compile to nothing on real targets.
3. -fno-strict-aliasing disables the compiler’s use of that rule. The Linux kernel builds with it, because too much existing code depends on aliasing.
4. Type confusion is CWE-843 and appears regularly in browser and interpreter vulnerabilities, where an object is allocated as one class and later accessed as another.
5. Because meaning is external, every serious data format carries it explicitly. Examples already met in this chapter: file magic numbers, the TIFF “II” or “MM” tag, HTTP Content-Type headers, and Unicode byte order marks.
6. The same principle explains why the encoding of a text file cannot be determined with certainty. Charset detectors such as charset-normalizer are statistical guesses, not measurements. Any file of bytes below 0x80 is simultaneously valid ASCII, valid UTF-8, valid Latin-1 and valid Windows-1252.
7. Inspect the same bytes several ways with xxd, od -t x1 -t c -t d4, objdump -D -b binary -m i386:x86-64, and numpy.ndarray.view.

WORDS7.11.6 remember these#

1. Interpretation — the decision about what bits mean — the effective type applied to a region of storage by the code that accesses it.
2. Cast — telling the compiler to read bits differently — a conversion that may reinterpret or may genuinely convert, depending on the language.
3. Type confusion — using data as the wrong kind of thing — CWE-843, accessing an object through an incompatible type, a common exploit class.
4. Magic number — a format’s fingerprint — a fixed byte signature at a known offset used to identify content when no external label exists.
5. Strict aliasing — the rule that objects of different types do not overlap — a C and C++ assumption that permits optimization and makes type punning through pointers undefined.

7.98 Common wrong ideas#

1. Wrong: computers use binary because it is more efficient. Right: they use it because a switch has two reliable states. Binary is the widest noise margin you can get from one wire, not the densest coding.
2. Wrong: hexadecimal is a different kind of number that computers understand. Right: hex is only a way of writing binary for humans. The machine stores bits; hex appears when something prints them.
3. Wrong: a byte has always been 8 bits. Right: the CDC 6600 used 6-bit characters and the PDP-10 used programmable 6, 7, 8 or 9-bit bytes. That is precisely why standards say “octet”.
4. Wrong: a 64-bit CPU can address 16 exbibytes of memory. Right: the architecture defines 64-bit pointers, but shipping x86-64 chips implement 48 address bits, giving 256 TiB, or 57 bits with 5-level paging.
5. Wrong: floating point numbers are approximate because computers are imprecise. Right: every float32 and float64 value is an exact binary number. The error comes from rounding a decimal input to the nearest representable binary value, which is a property of the format, not a hardware defect.
6. Wrong: use a small tolerance like 0.000001 to compare floats. Right: use a relative tolerance based on machine epsilon, 1.19e-7 for float32 and 2.22e-16 for float64, scaled to the magnitude being compared.
7. Wrong: UTF-8 is a 1-byte encoding and Unicode is a 2-byte encoding. Right: Unicode is a numbering scheme with no byte width at all. UTF-8 uses 1 to 4 bytes and UTF-16 uses 2 or 4.
8. Wrong: a Unicode character is one code point. Right: what a reader calls one character is a grapheme cluster, which may be many code points. The family emoji is 7 code points and 25 UTF-8 bytes.
9. Wrong: a CRC or a checksum proves the data was not tampered with. Right: it detects accidental corruption only. Anyone can alter data and recompute a CRC. Tamper resistance needs a cryptographic hash or a MAC.
10. Wrong: compression can shrink any file if the algorithm is clever enough. Right: by a simple counting argument, any lossless scheme that shrinks some inputs must expand others. Shannon entropy sets a hard floor.

7.99 Chapter summary in 20 lines#

1. Computers use base 2 because a wire is reliably above or below a threshold, and nothing in between can be read back safely.
2. N bits hold 2^N patterns, giving unsigned values from 0 to 2^N minus 1.
3. Convert decimal to binary by repeated division by 2, or by greedy subtraction of descending powers of 2. Both give the same answer.
4. Hexadecimal is binary written four bits at a time. Two hex digits are always exactly one byte, which is why engineers read memory in hex.
5. A byte is 8 bits today, but was not always, which is why standards say octet. A word is the CPU’s natural width, now 64 bits.
6. A 32-bit address space names 4,294,967,296 bytes, exactly 4 GiB, which is the wall 32-bit systems hit.
7. Two’s complement won because flipping the bits and adding one makes subtraction into addition, so one adder circuit does both jobs.
8. Signed 32-bit tops out at 2,147,483,647. That number caused the Year 2038 problem and the YouTube view counter overflow of 2 December 2014.
9. Ariane 5 Flight 501 was destroyed on 4 June 1996 because a 64-bit float was converted into a 16-bit signed integer that could not hold it.
10. IEEE 754, approved in 1985 under William Kahan, stores a sign bit, a biased exponent and a fraction with an implicit leading one.
11. In float32 the bias is 127, so 0.15625 encodes as 0x3E200000 and -6.25 encodes as 0xC0C80000.
12. All-ones and all-zeros exponents are reserved, giving zero, negative zero, subnormals, infinities and NaN, which is unequal to itself.
13. 0.1 plus 0.2 is not 0.3 because 0.1 is a repeating binary fraction, so both inputs and the sum are rounded to different nearby values. Money must use integer minor units or a decimal type.
14. ASCII, published as ASA X3.4-1963 on 17 June 1963, used 7 bits, put digits at 0x30 so masking gives their value, and separated ‘A’ from ‘a’ by one bit, bit 5.
15. The spare eighth bit produced the code page era, where the same bytes meant different characters in different countries, with no label in the file.
16. Unicode gives every character one permanent code point from U+0000 to U+10FFFF, across 17 planes; version 17.0 of 9 September 2025 holds 159,801 characters.
17. UTF-8, designed by Ken Thompson and Rob Pike in September 1992 and now specified by RFC 3629, is 1 to 4 bytes, ASCII-compatible and self-synchronising, which is why it won.
18. Endianness decides which byte of a multi-byte value sits at the lowest address; network byte order is big-endian, while x86-64 and ARM are little-endian.
19. Parity detects odd numbers of flipped bits, CRC32 catches all bursts up to 32 bits, ECC memory repairs one bit per 64 and detects two, and none of them resist a deliberate attacker.
20. Compression works because data repeats and is uneven; Huffman gives short codes to common symbols, LZ77 points back at earlier text, DEFLATE is both, and Shannon entropy is the floor none of them can pass.