Files, Formats, Extensions and the .exe

20.0 What this chapter gives you#

1. You will be able to say exactly what a file is, and why there is no such thing as a “picture file” at the level of the disk.
2. You will be able to explain what a file extension really does on Windows, on macOS and on Linux, and why the three answers are different.
3. You will be able to read the first bytes of any file and name the format, using a table of real magic numbers you will have memorized parts of.
4. You will be able to explain why a text file written on Windows shows strange marks on Linux, and fix it.
5. You will be able to open a DOCX, an XLSX, a JAR and an APK with a zip tool, because they are all zip files, and show the layout inside.
6. You will be able to describe what an .exe actually contains, section by section, including why it still says “This program cannot be run in DOS mode”.
7. You will be able to trace, in thirty numbered steps, everything that happens between your double-click and the first line of your program.
8. You will be able to explain why a .py or .sh file also runs, and what the two characters #! do at the level of the kernel.
9. You will be able to identify a file of an unknown format using only file, a hex dump, and strings.
10. You will be able to name what an installer does, why unsigned software warns you, and what a photo silently tells the world about where you were.

20.1 What a file actually is#

PLAIN20.1.1 in simple words#

1. A file is a run of numbered boxes. Each box holds one byte, a number from 0 to 255.
2. That is the whole of it. A file is a sequence of bytes with a length.
3. The file also has a name, so that a human and a program can find it again.
4. Around that sequence, the system keeps a few extra facts: how big it is, when it was last changed, who owns it, who is allowed to read it.
5. Those extra facts are called metadata, which just means “data about the data”.
6. Nothing in the bytes says “I am a picture”. Nothing in the metadata says it either, on most systems.
7. A file becomes a picture only when a program chooses to read those bytes using picture rules.
8. If a different program reads the same bytes using sound rules, it will produce noise. Nothing was broken. The bytes never claimed anything.
9. So a file is not a thing with a nature. It is raw material plus an agreement about how to read it.

PLAIN20.1.2 a picture in your head#

1. Think of a long shelf of numbered pigeonholes in a post room.
2. Each pigeonhole holds one small card with a number written on it, from 0 to
3. The shelf has a label on the front: “note.txt”. That is the file name.
4. A clipboard hangs beside the shelf. It records: 26 cards, filled on 13 August at 01:57, owned by the office manager, readable by everyone.
5. That clipboard is the metadata. The pigeonholes are the file contents.
6. Now, one clerk has been told “these cards spell out English letters”. Another clerk has been told “these cards are pixel colours”.
7. Both clerks read the exact same cards. They produce completely different results. Neither clerk is cheating.

Where this comparison breaks: real pigeonholes are physically next to each other, and a real file usually is not. The bytes of one file can be scattered across thousands of separate places on a disk, and the file system quietly stitches them into one continuous-looking sequence for you. The shelf is a fiction the file system maintains. Also, a real clipboard is attached to the shelf, whereas on most file systems the metadata lives somewhere else entirely, in a small record called an inode, and the file name is not even part of it.

PLAIN20.1.3 a worked example#

1. Here is a file with 26 bytes in it, viewed as raw numbers and then as characters.

$ stat note.txt
  File: note.txt
  Size: 26          Blocks: 8       IO Block: 4096  regular file
Device: 254,0    Inode: 794747      Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
Access: 2026-08-13 01:57:42.022877781 +0000
Modify: 2026-08-13 01:57:42.020159532 +0000
Change: 2026-08-13 01:57:42.020159532 +0000
 Birth: 2026-08-13 01:57:42.016211454 +0000

2. Size: 26 is the only fact about the contents. Twenty-six bytes. That is all the system knows about what is inside.
3. Inode: 794747 is the number of the little record that holds this metadata. The name note.txt is not stored there. It is stored in the folder.
4. Blocks: 8 means eight 512-byte units are actually reserved on disk, so 4096 bytes are used to store 26. Storage is handed out in whole blocks.
5. There are four timestamps, not one, and they mean different things:

6. Notice what is absent from every line above: any statement about what kind of file this is. The word “text” appears nowhere.

PLAIN20.1.4 what is really happening inside#

1. When you save a file, the file system does three separate jobs.
2. Job one: find free blocks on the storage device and write your bytes into them.
3. Job two: create or update a small fixed-size record holding the size, the owner, the permissions, the timestamps, and a list of which blocks hold the data. On Linux file systems this record is the inode.
4. Job three: add an entry to a directory. A directory is itself just a file whose contents are a list of pairs: a name, and an inode number.
5. This is why the name is not part of the file. The name lives in the folder, pointing at the file.
6. This is also why one file can have two names. Two directory entries can point at the same inode. That is a hard link.
7. Deleting a name does not delete the bytes. It removes one directory entry and reduces a counter. The bytes go only when the counter reaches zero.
8. The Links: 1 line in the output above is exactly that counter.
9. None of these three jobs asks or records what the bytes mean.

TECHNICAL20.1.5 the engineer’s version#

1. A regular file in POSIX terms is an unstructured byte stream addressed by offset, with no record structure imposed by the kernel.
2. The metadata returned by the stat(2) system call is defined in POSIX.1 and is filled from the file system’s own inode structure.
3. On ext4 the inode is 256 bytes by default and holds mode, uid, gid, size, four timestamps with nanosecond precision, link count, and either extents or block pointers.
4. st_size is the logical length in bytes. st_blocks is the allocated storage in 512-byte units and can be smaller than st_size for a sparse file or larger because of block rounding.
5. Directory entries on ext4 are stored in linux_dirent structures containing an inode number, a record length, a name length, a one-byte file type hint, and the name. There is no type field beyond regular, directory, symlink, FIFO, socket, block device and character device.
6. NTFS differs. Its Master File Table record holds a set of named attributes, and small files are stored resident inside the MFT record itself. NTFS also supports multiple named data streams per file, which is covered in 20.13.
7. Apple’s APFS, shipped in 2017 with macOS 10.13 High Sierra, stores extended attributes in the same B-tree as the file record.
8. Real figures for common file systems:

9. Tools that observe this layer: stat, ls -li, df -i, debugfs on ext4, fsutil file layout on Windows, diskutil and stat -f on macOS.

WORDS20.1.6 remember these#

1. File — a named run of bytes — an unstructured byte stream addressed by offset, with an associated metadata record.
2. Metadata — facts about a file that are not its contents — the struct stat fields: mode, uid, gid, size, timestamps, link count.
3. Inode — the small record holding a file’s facts — an on-disk structure indexed by number, containing everything except the name.
4. Directory — a folder — a file whose contents map names to inode numbers.
5. Hard link — a second name for the same file — an additional directory entry referencing the same inode, incrementing st_nlink.
6. Block — the smallest chunk of disk given out — the allocation unit, typically 4096 bytes on ext4 and NTFS.

20.2 File extensions: a convention, not a rule#

PLAIN20.2.1 in simple words#

1. The bit after the last dot in a file name is the extension: .txt, .jpg, .exe.
2. Say this once and remember it: on almost every system, the extension is part of the name and nothing else.
3. It is a convention. Everyone agreed to use it. It is not enforced by the disk, and mostly not enforced by the kernel either.
4. You can rename holiday.jpg to holiday.txt. Not one byte inside the file changes.
5. What changes is the guess that programs make about the file when they see the name.
6. On Windows, that guess is very strong. The extension decides which program opens the file, full stop.
7. On macOS, the extension matters too, but the system also keeps a richer type idea behind the scenes.
8. On Linux, the usual tools ignore the name and look at the bytes instead.
9. So the same rename does three different things on three systems, and none of them altered the file.

PLAIN20.2.2 a picture in your head#

1. Think of a hospital where every patient wears a wristband with a written label: “diabetes”, “fracture”, “asthma”.
2. The wristband is fast and convenient. A nurse can glance and act.
3. But the wristband is written by hand. It is not the patient. Swap two wristbands and the patients are unchanged.
4. A careful doctor does not trust the wristband. She examines the patient. That takes longer and is always right.
5. Windows is the nurse acting on the wristband. Linux is the doctor examining the patient.
6. macOS is a hospital that reads the wristband first but also keeps a proper file in the records office.

Where this comparison breaks: a real patient has an objective illness that exists whether or not anyone examines it. A file does not have an objective format in the same way. Its format only exists as an agreement between whoever wrote it and whoever reads it. Two different agreements can both fit the same bytes, which is why a valid file can genuinely be two formats at once, a fact attackers use.

PLAIN20.2.3 a worked example#

1. Here is a real one-pixel PNG image. We copy it to a .txt name and ask the system what it is.

$ cp tiny.png tiny.txt
$ file tiny.txt
tiny.txt: PNG image data, 1 x 1, 8-bit/color RGB, non-interlaced

$ file --mime-type tiny.txt tiny.png
tiny.txt:   image/png
tiny.png:   image/png

$ sha256sum tiny.png tiny.txt
b1ff9c8ea3a780bad09b346c423d2d0e46815926879b18e841d928376a946640
  tiny.png
b1ff9c8ea3a780bad09b346c423d2d0e46815926879b18e841d928376a946640
  tiny.txt

2. The two checksums are identical to the last character. The contents are the same file.
3. The Linux file command did not care about the name at all. It read the bytes and answered PNG both times.
4. On Windows, double-clicking tiny.txt would open Notepad, which would show a screenful of rubbish and then possibly corrupt the file if you saved it.
5. Nothing about the file was ever wrong. The name simply stopped matching the contents, and one operating system trusts names.

PLAIN20.2.4 what is really happening inside#

1. Windows keeps a large lookup table called the registry. One part of it, HKEY_CLASSES_ROOT, maps extensions to type identifiers.
2. .txt maps to a name like txtfile. txtfile then has a shell\open\command entry giving the exact command line to run.
3. Double-clicking runs ShellExecute, which walks that chain: extension, then program identifier, then command. The file’s contents are never consulted.
4. macOS uses Uniform Type Identifiers, reverse-domain strings such as public.png or com.adobe.pdf. Every application declares which identifiers it can open.
5. macOS decides a file’s identifier mostly from its extension, then registers the answer in a database called Launch Services.
6. Linux desktops use the freedesktop.org shared MIME database, which combines name patterns with content tests, and content tests usually win.
7. Command-line Linux tools such as file use a library called libmagic, which holds thousands of rules of the form “at offset X, expect these bytes, therefore this format”.
8. So there are two strategies in the world: trust the name, or read the bytes. Reading the bytes is slower and needs an open file. Trusting the name is instant and needs nothing.
9. That speed difference is the whole reason the extension convention survives.

TECHNICAL20.2.5 the engineer’s version#

1. The 8.3 filename with a three-character extension comes from CP/M in 1974 and was carried into MS-DOS in 1981 and into the FAT file system.
2. In FAT, the name and extension were stored as two fixed fields, 8 bytes and 3 bytes, with no dot stored on disk. The dot was purely a display convention.
3. Modern Windows stores the whole name including dots as one UTF-16 string. The extension is defined as the text after the final dot.
4. Windows Explorer hides extensions for registered types by default. This setting is HideFileExt under HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced.
5. That default is a documented security weakness. A file named invoice.pdf.exe displays as invoice.pdf with a PDF-looking icon supplied by the executable’s own resource section.
6. The ILOVEYOU worm of May 2000 used exactly this. Its attachment was named LOVE-LETTER-FOR-YOU.TXT.vbs and displayed as a .TXT file. It caused damage estimated in the billions of dollars.
7. Windows also supports right-to-left override characters in file names, which can display exe.txt when the real name ends in .exe. Modern mail gateways strip these.
8. Apple introduced Uniform Type Identifiers in Mac OS X 10.4 Tiger in 2005, and replaced the older four-character type and creator codes inherited from the 1984 Macintosh. A modern Swift framework, UniformTypeIdentifiers, arrived in macOS 11 in 2020.
9. The file command was written by Ian Darwin in 1986 and rewritten and maintained by Christos Zoulas since 1993. Its rule database is compiled into magic.mgc.
10. Behaviour by platform:

11. Web browsers are a fourth case. They obey the HTTP Content-Type header, with a defined sniffing algorithm in the WHATWG MIME Sniffing Standard, which servers can disable with X-Content-Type-Options: nosniff.

WORDS20.2.6 remember these#

1. Extension — the letters after the last dot — a filename suffix used as a type hint, not enforced by the file system.
2. Convention — everyone does it this way — behaviour that is customary but not specified or enforced.
3. Registry — the Windows settings database — a hive-structured key-value store; HKEY_CLASSES_ROOT maps extensions to ProgIDs.
4. UTI — Apple’s name for a file kind — Uniform Type Identifier, a reverse-DNS string in a declared inheritance hierarchy.
5. libmagic — the library that guesses a file’s type from its contents — a rules engine matching byte patterns at given offsets.
6. Double extension — a name with two dots used to deceive — an attack relying on HideFileExt and on icon resources embedded in the executable.

20.3 Magic numbers: the real identity of a file#

PLAIN20.3.1 in simple words#

1. Most file formats begin with a short, fixed pattern of bytes that says which format it is.
2. That pattern is called a magic number, or a signature.
3. It is at the very start, so a program can read a few bytes, decide, and stop if it is wrong.
4. A PNG image always begins with the byte 137 and then the letters P, N, G.
5. A ZIP archive always begins with the letters P and K, the initials of Phil Katz who invented the format.
6. A Linux program always begins with the byte 127 and then the letters E, L, F.
7. A Windows program always begins with the letters M and Z, the initials of Mark Zbikowski who designed the old DOS executable header.
8. This is the closest thing a file has to a real identity, and it is still only a convention, because nothing stops you writing those bytes yourself.
9. But it is a far better guess than the name, because the bytes were written by the program that made the file.

PLAIN20.3.2 a picture in your head#

1. Think of the first line of a formal letter.
2. “Dear Sir” tells you it is a letter. “Once upon a time” tells you it is a story. “Ingredients:” tells you it is a recipe.
3. You do not need to read the rest. Three or four words settle it.
4. Better still, some documents begin with a printed crest or a watermark, which is harder to produce by accident.
5. A magic number is that crest. It is at the top, it is short, and it is very unlikely to appear at the top of something else by chance.

Where this comparison breaks: a letter’s opening is meant for humans and can be argued about. A magic number is exact. Either the bytes are 89 50 4E 47 or they are not. There is no partial match and no interpretation. Also, unlike a crest, a magic number carries no proof of who wrote it. Anyone can type the same four bytes at the start of any file, which is exactly how polyglot files, files valid as two formats at once, are built.

PLAIN20.3.3 a worked example#

1. Here is the first sixteen bytes of a real PNG file we made, shown as hex on the left and as printable characters on the right.

$ xxd -l 16 tiny.png
00000000: 8950 4e47 0d0a 1a0a 0000 000d 4948 4452  .PNG........IHDR

2. Read it byte by byte. 89 is the number 137, chosen deliberately to be above 127 so that a program treating the file as text notices immediately.
3. 50 4e 47 is P, N, G in ASCII.
4. 0d 0a is carriage return and line feed. If a broken file transfer converted line endings, this pair would be damaged and the file would fail its own check.
5. 1a is the old DOS end-of-file character. If you type this file on DOS, it stops printing right there instead of spewing binary.
6. 0a is a lone line feed, catching the opposite conversion.
7. Then 00 00 00 0d, which is the number 13 in big-endian order: the length of the first chunk.
8. Then 49 48 44 52, which is IHDR, the image header chunk.
9. The whole eight-byte PNG signature was designed in the mid-1990s to survive damage by naive file transfer software, and it works.

PLAIN20.3.4 what is really happening inside#

1. A program that must identify a file opens it and reads the first block, usually the first 512 or 4096 bytes.
2. It then runs down a list of rules. Each rule says: at this offset, compare these bytes, and if they match, report this type.
3. Rules can be nested. Matching RIFF at offset 0 leads to a second test at offset 8, which decides between WAV, AVI and WebP.
4. Rules can also be negative. A ZIP that begins with the entry name mimetype is treated as an OpenDocument file rather than a plain archive.
5. Some formats have no fixed signature at all. Plain text, CSV and raw MP3 audio are examples, so the tool falls back to statistics: are all bytes printable, are there consistent separators.
6. When nothing matches, the answer is data, which honestly means “I do not know”.

TECHNICAL20.3.5 the engineer’s version#

1. The following signatures were all verified by hex-dumping real files on the machine used to write this chapter, except where the row says otherwise.

2. Notes on the trickier rows. Mach-O stores its magic as a 32-bit little-endian integer, so MH_MAGIC_64, whose value is 0xFEEDFACF, appears on disk as CF FA ED FE. The 32-bit form is 0xFEEDFACE.
3. A macOS universal binary starts with 0xCAFEBABE in big-endian order, which is byte-for-byte identical to a Java class file. Tools disambiguate by checking the next field: Java stores a version number below 100, a fat Mach-O stores an architecture count. This collision is real and long-standing.
4. Mach-O was not produced in this sandbox, which is Linux. Those two rows come from the loader.h header in Apple’s open-source XNU sources, not from a local dump.
5. Java class files carry a major version after the magic. Version 52 is Java 8, 55 is Java 11, 61 is Java 17, 65 is Java 21. A real dump taken here:

$ unzip -p commons-lang3-3.14.0.jar \
    org/apache/commons/lang3/AnnotationUtils\$1.class | xxd -l 16
00000000: cafe babe 0000 0034 00bc 0a00 0200 0307

0x34 is 52, so this class targets Java 8. 6. WebAssembly’s magic is the four bytes \0asm followed by a 32-bit version, currently 1. WebAssembly 1.0 became a W3C Recommendation in December 2019. 7. Raw MP3 has no header signature. An MPEG audio frame starts with an 11-bit sync pattern of all ones, so bytes such as FF FB. Most real files start with an ID3v2 tag instead, whose first three bytes are ID3. 8. The file command in action on the machine used here:

$ file sample.wav sample.db sample.wasm hello hello.exe
sample.wav:  RIFF (little-endian) data, WAVE audio,
             Microsoft PCM, 16 bit, mono 8000 Hz
sample.db:   SQLite 3.x database, last written using
             SQLite version 3045001, database pages 2
sample.wasm: WebAssembly (wasm) binary module version
             0x1 (MVP)
hello:       ELF 64-bit LSB pie executable, x86-64,
             dynamically linked, interpreter
             /lib64/ld-linux-x86-64.so.2, not stripped
hello.exe:   PE32+ executable (console) x86-64,
             for MS Windows, 19 sections

9. Note how much file extracted beyond the type: image dimensions, sample rate, SQLite version, section count, interpreter path. Those come from deeper rules in the same database.
10. The magic database ships as text in /usr/share/file/magic/ and is compiled into magic.mgc. You can add your own rules and point file at them with the -m option.

WORDS20.3.6 remember these#

1. Magic number — a fixed pattern at the start of a file naming its format — a signature matched at a specified offset by a type-detection rule.
2. Signature — another word for magic number — same thing; the term “signature” is also used for cryptographic signing, so context matters.
3. Polyglot file — one file that is validly two formats — a file crafted so that two parsers each find a complete valid structure.
4. Endianness — which end of a number comes first — byte order; little-endian stores the least significant byte at the lowest address.
5. Sniffing — guessing a type by reading contents — content-based type detection, as opposed to name-based.
6. libmagic rule — one line in the guessing database — an offset, a type, a test value and a message, optionally nested by indentation depth.

20.4 Text files versus binary files#

PLAIN20.4.1 in simple words#

1. People say a file is either “text” or “binary”. That split is useful but it is not a real property of the file.
2. Every file is binary. Every file is bytes. There is no text mode on the disk.
3. What people mean by “text” is: if you show each byte as a character, you get something a human can read.
4. So “text” is a statement about how the bytes happen to look, not about how they are stored.
5. Text files also have one extra habit: they are split into lines.
6. And here the trouble starts, because the world never agreed on how to mark the end of a line.
7. Linux and macOS end a line with one invisible byte. Windows ends it with two. Very old Macs used one different byte.
8. That is why a file written on Windows sometimes shows odd marks when opened on Linux, and why a Linux file sometimes appears as one giant line on Windows.
9. On top of that, there is the question of which byte means which letter, which is called the encoding, and that is a second, larger mess.

PLAIN20.4.2 a picture in your head#

1. Imagine an old mechanical typewriter with a paper carriage.
2. To start a new line you must do two separate physical things.
3. First, push the carriage back to the left edge. That is the carriage return.
4. Second, roll the paper up by one line. That is the line feed.
5. Two actions, two separate control codes, because the machine really did have two separate mechanisms.
6. Windows still sends both codes, faithfully imitating the typewriter.
7. Unix decided one code should mean “do both”, and sends only the line feed.
8. The old Macintosh decided the opposite, and sent only the carriage return.
9. Three reasonable choices, made in different decades, and we live with all three forever.

Where this comparison breaks: on a typewriter you could send carriage return without line feed on purpose, to type over a line and make bold text. Software no longer does that, so the two codes have collapsed into one meaning, “new line”, and only the spelling differs. Also, a typewriter had no concept of encoding. Which key made which shape was fixed by the metal. In a file, the mapping from byte to character is a separate choice on top of the line-ending choice.

PLAIN20.4.3 a worked example#

1. We made the same two lines of text in three different line-ending styles and dumped the raw bytes.

$ file unix.txt dos.txt mac9.txt
unix.txt: ASCII text
dos.txt:  ASCII text, with CRLF line terminators
mac9.txt: ASCII text, with CR line terminators

$ od -c unix.txt
l i n e   o n e \n l i n e   t w o \n

$ od -c dos.txt
l i n e   o n e \r \n l i n e   t w o \r \n

$ od -c mac9.txt
l i n e   o n e \r l i n e   t w o \r

$ wc -l unix.txt dos.txt mac9.txt
 2 unix.txt
 2 dos.txt
 0 mac9.txt

2. \n is byte 10, the line feed. \r is byte 13, the carriage return.
3. Notice the last line. The tool counted two lines in the Unix file, two in the Windows file, and zero in the old Mac file.
4. It is not broken. wc -l counts line feed bytes, and the old Mac file contains none at all.
5. This is exactly how a real file can be “empty” to one tool and full of text to another.
6. When a Windows-made file is read by a Linux tool that only strips \n, the \r stays on the end of every value. A version number 1.2.3 becomes 1.2.3\r, and comparisons silently fail.

PLAIN20.4.4 what is really happening inside#

1. There are three different layers where line endings can be changed, and confusing them causes most of the pain.
2. Layer one: the program that wrote the file chose what to put there.
3. Layer two: a transfer tool may translate on the way. Old FTP had an explicit text mode that rewrote line endings. Git can do the same on checkout.
4. Layer three: the program reading the file may quietly accept both, or may not.
5. Encoding is a separate question: which byte value stands for which character.
6. ASCII fixed 128 characters in the 1960s using values 0 to 127. That covers English and nothing else.
7. Everyone then invented their own meaning for values 128 to 255, producing dozens of incompatible sets, and the same bytes rendered differently in different countries.
8. Unicode solved the character question by giving every character in every script a number, and UTF-8 solved the byte question by encoding those numbers in one to four bytes.
9. UTF-8 was designed so that all the old ASCII values keep their old meaning. An English-only file is identical in ASCII and UTF-8.
10. Some tools put three special bytes at the very start of a UTF-8 file to announce the encoding. That is the byte order mark, and it causes its own set of problems.

TECHNICAL20.4.5 the engineer’s version#

1. Carriage return is US-ASCII 0x0D, decimal 13. Line feed is 0x0A, decimal 10. Both are in the C0 control set standardized in ASCII, published in 1963 and revised in 1967.
2. The two-code sequence comes from teleprinters such as the Teletype Model 33, introduced in 1963, which needed roughly 200 milliseconds to return the carriage. Sending two codes gave the mechanism time to finish.
3. Multics and then Unix reduced this to a single 0x0A in the file and let the terminal driver expand it, which is why the C standard library still has the notion of translating in text mode.
4. Line-ending conventions by system:

5. Several network protocols mandate CRLF regardless of platform, including HTTP/1.1 in RFC 9112, SMTP in RFC 5321, and FTP in RFC 959. This is a standard, not a convention, and sending a bare LF is a protocol violation.
6. Encodings and their byte order marks, all dumped from real files created for this chapter:

$ xxd -l 12 bom_utf8.txt
00000000: efbb bf68 656c 6c6f 0a                   ...hello.

$ xxd -l 12 utf16.txt
00000000: fffe 6800 6500 6c00 6c00 6f00            ..h.e.l.l.o.

$ xxd -l 6 latin1.txt
00000000: 6361 66e9 0a                             caf..

$ xxd -l 6 utf8.txt
00000000: 6361 66c3 a90a                           caf...

7. Read those carefully. The word café is five bytes in Latin-1 and six bytes in UTF-8, because the accented é needs one byte in the first and two in the second.
8. The UTF-8 byte order mark is EF BB BF. It encodes U+FEFF, a character that originally meant zero-width non-breaking space.
9. UTF-8 has no byte order to mark, so the mark is purely a flag. The Unicode standard permits it but does not recommend it, and POSIX tools do not expect it.
10. A UTF-8 byte order mark breaks a shebang line, breaks a shell script, breaks a JSON parser that follows RFC 8259 strictly, and appears as  when a file is misread as Latin-1. Microsoft tools add it by default; most others do not.
11. UTF-16 requires a mark or external knowledge, because fffe and feff mean opposite byte orders. The dump above shows fffe, which is little-endian.
12. Useful commands: file -i, iconv -f X -t Y, dos2unix, unix2dos, sed -i 's/\r$//', and in Git, core.autocrlf and a .gitattributes file with * text=auto.
13. The honest version: file reporting “ASCII text” is a guess based on statistics. It checks whether all bytes fall in printable and common control ranges. A binary file made only of printable bytes will be called text, and a UTF-8 file truncated mid-character will be called data.

WORDS20.4.6 remember these#

1. Text file — a file whose bytes read as readable characters — a byte stream interpretable under a character encoding and divided by line terminators.
2. CR — carriage return — US-ASCII 0x0D, historically returning the print head to column zero.
3. LF — line feed — US-ASCII 0x0A, historically advancing the paper one line.
4. CRLF — the Windows and network line ending — the two-byte sequence 0x0D 0x0A, mandated by HTTP, SMTP and FTP.
5. Encoding — the rule mapping bytes to characters — a character encoding scheme such as US-ASCII, ISO-8859-1 or UTF-8.
6. BOM — three or two marker bytes at the start of a text file — the byte order mark, U+FEFF, encoded as EF BB BF in UTF-8.
7. Mojibake — text that has turned into garbage symbols — the result of decoding bytes with a different encoding than they were written in.

20.5 Common formats and how they are built#

PLAIN20.5.1 in simple words#

1. Most real formats are built from two separate ideas: a container and the things inside it.
2. The container decides how to store several pieces, where each one starts, and what each one is called.
3. The pieces inside can each be squeezed or encoded in their own way.
4. A ZIP file is the most successful container ever made. It holds a list of named items, each optionally compressed.
5. That is why so many things you use every day are secretly ZIP files.
6. A Word document, an Excel sheet, a PowerPoint deck, a Java program and an Android app are all ZIP files with agreed contents inside.
7. A PDF is a different shape. It is a numbered list of objects with a directory at the end telling you where each object lives.
8. Then there are the small human-readable data formats, JSON, XML, YAML, CSV and TOML, each with a different set of traps.

PLAIN20.5.2 a picture in your head#

1. Think of a filing cabinet with a card index drawer at the bottom.
2. Each folder in the cabinet has its own label card stapled to the front, and the contents may be folded up tightly to save space.
3. The card index at the bottom lists every folder and says exactly how far in it sits.
4. To find one folder, you read the index, walk straight to that spot, and pull it out. You never touch the others.
5. To add a folder, you push it in at the end, then write a new index and put it after everything. The old index is simply ignored.
6. That is a ZIP file exactly: local labels on each item, and a central directory at the very end.

Where this comparison breaks: with a real cabinet you would throw the old index away. A ZIP file keeps it, sitting in the middle of the file, unreferenced. That leftover is not harmless: some tools read the file forwards and see the old listing, while correct tools read the central directory at the end and see the new one. Attackers have used this exact disagreement to smuggle content past scanners. It is known in the Android world as one of the master key bugs.

PLAIN20.5.3 a worked example#

1. A Microsoft Word document is a ZIP file. Here is a real one, created with a library, then opened with a plain unzip tool.

$ file report.docx
report.docx: Microsoft Word 2007+

$ xxd -l 16 report.docx
00000000: 504b 0304 1400 0000 0800 2e0f 0d5d ad52  PK...........].R

$ unzip -l report.docx
  Length      Date    Time    Name
---------  ---------- -----   ----
     1738  2026-08-13 01:57   [Content_Types].xml
      734  2026-08-13 01:57   _rels/.rels
      721  2026-08-13 01:57   docProps/core.xml
     1132  2026-08-13 01:57   docProps/app.xml
     1693  2026-08-13 01:57   word/document.xml
     1227  2026-08-13 01:57   word/_rels/document.xml.rels
   349458  2026-08-13 01:57   word/styles.xml
   438131  2026-08-13 01:57   word/stylesWithEffects.xml
     2535  2026-08-13 01:57   word/settings.xml
     2811  2026-08-13 01:57   word/fontTable.xml
    10939  2026-08-13 01:57   word/theme/theme1.xml
     5513  2026-08-13 01:57   word/numbering.xml
     8324  2026-08-13 01:57   docProps/thumbnail.jpeg
---------                     -------
   826305                     17 files

2. The first four bytes are 504b 0304, the ZIP local file header signature. It is a ZIP and nothing else.
3. Your actual words live in word/document.xml. Everything else is styles, relationships, fonts, and a preview image.
4. Look at the sizes. The text of a two-line document is 1693 bytes. The style definitions are 349458 bytes. That is why an empty Word file is not empty.
5. The docProps/thumbnail.jpeg entry is a real JPEG, which is why a document can leak a picture of an earlier version of itself.
6. You can extract, edit the XML with a text editor, and re-zip, and Word will open the result. This is how many document-generation tools work.

PLAIN20.5.4 what is really happening inside#

1. A ZIP file is stored as three kinds of record, in this order in the file.
2. First, for every item: a local file header giving the name, the compression method, the sizes and a checksum, immediately followed by the item’s bytes.
3. Then, after all the items: a central directory, one record per item, repeating the same information plus the exact offset where that item’s local header starts.
4. Last, a small end of central directory record saying how many entries there are and where the central directory begins.
5. A correct reader starts at the end, finds the end record, jumps to the central directory, and reads the list. It never scans forwards.
6. That design has three consequences worth knowing.
7. You can pull out one file from a five-gigabyte archive without reading the rest, because you know the exact offset.
8. You can add data before the archive without breaking it, because all offsets are found through the end record, which self-corrects. That is how self-extracting archives work: a program at the front, a ZIP at the back.
9. You can append junk after the archive and many tools still open it, because they search backwards for the end record.

TECHNICAL20.5.5 the engineer’s version#

1. ZIP was created by Phil Katz in 1989 for PKZIP. The specification is Katz’s APPNOTE.TXT, maintained by PKWARE, not by a standards body. It is a de facto standard.
2. Signature values, all confirmed by parsing a real archive built for this chapter: local header 0x04034B50, central directory 0x02014B50, end of central directory 0x06054B50. Stored little-endian, so PK\3\4 on disk.
3. The structure of a real three-entry archive, parsed with a short Python script written for this chapter:

total size: 556 bytes
LOCAL   @    0 name=note.txt  method=0 csize=  26 usize=  26
LOCAL   @   92 name=data.csv  method=0 csize=  12 usize=  12
LOCAL   @  170 name=tiny.png  method=8 csize=  64 usize=  69
EOCD    @  534 entries=3 cd_size=234 cd_offset=300
CENTRAL @  300 name=note.txt  local_hdr_at=   0
CENTRAL @  378 name=data.csv  local_hdr_at=  92
CENTRAL @  456 name=tiny.png  local_hdr_at= 170

4. Method 0 is Stored, meaning no compression. Method 8 is Deflate. The tool chose Stored for the tiny text files because compressing them made them larger.
5. Deflate is specified in RFC 1951, published in 1996 by Peter Deutsch. It combines LZ77 matching over a 32 KiB window with Huffman coding.
6. Appending to an archive was tested directly:

$ cp demo.zip appended.zip
$ printf 'EXTRA_TRAILING_DATA' >> appended.zip
$ unzip -l appended.zip
       26  note.txt
       12  data.csv
       69  tiny.png

The archive still lists correctly with 19 extra bytes glued on the end. 7. Prepending was also tested. Concatenating a Linux executable and a ZIP produced a file that file calls an ELF executable and unzip reads as a valid archive. Both tools are correct. 8. The original ZIP format uses 32-bit sizes and offsets and a 16-bit entry count, capping it at 4 GiB and 65535 entries. Zip64 extensions, added to APPNOTE around 2001, raise those to 64-bit fields. 9. ZIP-based formats and what identifies each:

10. OOXML, the DOCX family, was standardized as ECMA-376 in 2006 and ISO/IEC 29500 in 2008. OpenDocument was standardized by OASIS in 2005 and as ISO/IEC 26300 in 2006.
11. PDF has a completely different shape. Adobe released PDF 1.0 in 1993. It became an open ISO standard, ISO 32000-1, in 2008, and ISO 32000-2, known as PDF 2.0, in 2017.
12. A PDF is a header, then a body of numbered indirect objects, then a cross-reference table listing the byte offset of every object, then a trailer. Real tail of a 33-page PDF found on this machine:

0000683320 00000 n
0000683847 00000 n
trailer
<<
/Size 302
/Root 301 0 R
/Info 300 0 R
>>
startxref
683904
%%EOF

13. startxref gives the byte offset of the cross-reference table, and %%EOF ends the file. A reader also starts at the end here, exactly like ZIP.
14. /Root 301 0 R is an indirect reference to object 301, which is the document catalogue. The 0 is a generation number used by incremental updates.
15. PDF supports incremental update: append new objects and a new cross-reference table, leaving the old ones in place. This is why a redacted PDF can still contain the unredacted original.
16. The small data formats compared, with the trap that catches people:

17. Detail on each trap. JSON forbids comments and trailing commas by design, and RFC 8259 requires UTF-8 for network interchange. JSON numbers have no defined precision limit, so a 64-bit integer can lose accuracy in a parser that uses doubles.
18. XML supports entity definitions. A document defining nested entities that each expand ten times can expand to gigabytes from a few kilobytes. That is the billion laughs attack, first publicized in 2003. Parsers must disable external entity resolution, or an XML file can read local files off a server.
19. YAML 1.1 interpreted unquoted NO, ON, OFF and YES as booleans. In a country list, NO for Norway became false. YAML 1.2’s core schema fixed this in 2009, but many libraries still default to 1.1 behaviour. Check your library, not the specification.
20. CSV is not one format. RFC 4180 is informational and describes common practice: comma separator, CRLF line endings, double quotes doubled inside quoted fields. Real files use semicolons in European locales, tabs, and embedded newlines inside quoted fields.
21. TOML was created by Tom Preston-Werner in 2013 and reached version 1.0.0 in January 2021. It is used by Rust’s Cargo and by Python packaging in pyproject.toml.
22. Containers versus codecs, restated: a .mp4 file is a container defined by ISO/IEC 14496-12. It says nothing about how the video is encoded. Inside it can be H.264, H.265, AV1 or several others, plus AAC or Opus audio. “It is an MP4” tells you nothing about whether a device can play it.

WORDS20.5.6 remember these#

1. Container — a format that holds other pieces — a structure defining framing, naming and offsets for enclosed streams.
2. Codec — the rule for squeezing one stream — coder-decoder, the algorithm encoding a media stream inside a container.
3. Central directory — the index at the end of a ZIP — the authoritative list of entries with their local header offsets.
4. Deflate — the usual ZIP compression method — LZ77 with a 32 KiB window plus Huffman coding, specified in RFC 1951.
5. Indirect object — a numbered item in a PDF — an object addressed as “number generation R” and located through the cross-reference table.
6. Incremental update — adding to a PDF without rewriting it — appending new objects and a new xref section pointing back to the previous one.
7. Billion laughs — an XML file that explodes in size — a recursive entity expansion denial-of-service attack.

20.6 What an executable file really is#

PLAIN20.6.1 in simple words#

1. There is nothing magical about a program file. It is a file, like a photo or a letter.
2. What makes it special is what the bytes are, and a small label at the front that explains how to arrange them.
3. Most of the bytes are machine instructions: the exact numeric codes that the processor knows how to obey.
4. The rest is data the program needs: text it will print, tables it will read, space it will want.
5. The label at the front is called a header. It says things like “the code starts at this offset and should be placed at this address” and “start running at this point”.
6. The part of the operating system that reads that header and does what it says is called the loader.
7. So running a program is: read a header, copy or map the pieces into memory in the right places, then jump to the starting point.
8. Three families of headers dominate the world, one per operating system, and they do the same job in different words.
9. Windows uses PE. Linux uses ELF. macOS uses Mach-O.

PLAIN20.6.2 a picture in your head#

1. Think of a flat-pack wardrobe delivered in a box.
2. The box holds planks, screws and panels. None of it is a wardrobe yet.
3. On top sits an instruction sheet: put panel A here, panel B there, this side faces the wall, start with step one.
4. The planks are the machine instructions and data. The instruction sheet is the header.
5. The person following the sheet is the loader. It never invents anything. It only does what the sheet says.
6. A wardrobe from a different manufacturer comes with a different style of sheet. The planks might be almost identical. The sheet is not.
7. Give a Swedish sheet to a fitter trained on a different system and nothing gets built, even though the wood is fine.

Where this comparison breaks: a wardrobe is assembled once and then stands on its own. A program is re-assembled from the file every single time you run it, and usually most of the planks are never actually moved. The loader mostly just says “pretend these pages of the file are at these addresses” and the pages are read from disk only when first touched. It is less like building a wardrobe and more like handing someone a catalogue and promising to fetch each plank the moment they reach for it.

PLAIN20.6.3 a worked example#

1. We compiled the same six-line C program twice on the same Linux machine, once for Linux and once for Windows, and asked what each file is.

$ gcc -O2 -o hello hello.c
$ x86_64-w64-mingw32-gcc -O2 -o hello.exe hello.c

$ file hello
hello: ELF 64-bit LSB pie executable, x86-64, version 1
       (SYSV), dynamically linked, interpreter
       /lib64/ld-linux-x86-64.so.2, not stripped

$ file hello.exe
hello.exe: PE32+ executable (console) x86-64,
           for MS Windows, 19 sections

$ ls -l hello hello.exe
-rwxr-xr-x 1 root root  15960 hello
-rwxr-xr-x 1 root root 249671 hello.exe

2. Same source code. Same processor family, x86-64. Same instructions inside for the parts we wrote.
3. Yet the Linux file will not run on Windows and the Windows file will not run on Linux without a compatibility layer.
4. The reason is not the instructions. It is the header format and, more importantly, which library functions the file expects to find.
5. The Windows file is fifteen times larger here because the cross-compiler statically included more of its runtime and debug information.
6. Here is the first line of each file, side by side:

hello      7f45 4c46 0201 0100  .ELF....
hello.exe  4d5a 9000 0300 0000  MZ......

7. Four bytes and two bytes. That is the visible difference between an entire operating system’s idea of a program and another’s.

PLAIN20.6.4 what is really happening inside#

1. A processor does not know about files. It knows about memory addresses and instruction bytes.
2. Every program file therefore has to answer four questions for the loader.
3. Question one: which processor are these instructions for? An ARM chip cannot run x86 codes, so the header names the architecture.
4. Question two: which pieces of the file go where in memory? The header lists chunks, each with a file offset, a memory address, a length, and permissions.
5. Question three: what permissions does each chunk need? Code needs to be readable and executable but not writable. Data needs to be readable and writable but not executable.
6. Question four: where do I start? The header holds one address, the entry point.
7. There is a fifth question for most modern programs: which shared libraries do I need? The header lists them by name.
8. The loader answers these in order, builds a fresh empty memory space for the new program, maps in the chunks, and jumps to the entry point.
9. That is the entire trick. Everything else in an executable format is bookkeeping around those five answers.

TECHNICAL20.6.5 the engineer’s version#

1. The three formats and their lineage:

2. PE stands for Portable Executable. It extends COFF, the Common Object File Format from AT&T Unix System V, and was introduced with Windows NT 3.1 in
   1993. The specification is published by Microsoft as the PE Format document.
3. ELF stands for Executable and Linkable Format. It was defined by Unix System Laboratories for System V Release 4 and published in the System V Application Binary Interface. Linux moved from the older a.out format to ELF during 1995 and 1996.
4. Mach-O comes from the Mach microkernel project at Carnegie Mellon University, was used by NeXTSTEP from 1989, and was inherited by Mac OS X in 2001.
5. A structural comparison:

6. Mach-O uses .dylib for shared libraries, .a for static archives, load commands instead of a fixed table, LC_MAIN for the entry point and /usr/lib/dyld as the dynamic linker.
7. All three support position-independent code so that the operating system can place the image at a randomized base address. This is Address Space Layout Randomization, standard on Windows since Vista in 2007, on Linux since the mid-2000s, and on macOS since 10.5 in 2007.
8. UEFI firmware executables also use PE/COFF, with subsystem values 10 to 13. This is why a Linux bootloader signed for Secure Boot is a PE file even though Linux itself uses ELF.
9. There is genuine disagreement among engineers about whether a modern executable format should keep separate section and segment tables at all. ELF keeps both, one for the linker and one for the loader. Some newer formats such as WebAssembly use a single section list. The ELF side argues the split allows stripping without touching load behaviour; the critics argue it duplicates information and has caused parser confusion bugs.

WORDS20.6.6 remember these#

1. Executable — a file the system can run — an image containing machine code plus a header describing its memory layout and entry point.
2. Header — the label at the front of a program file — a fixed structure giving architecture, layout tables and the entry point.
3. Loader — the part of the system that sets a program up — kernel code that parses the image, creates an address space and maps segments.
4. Entry point — where execution begins — the virtual address stored in e_entry on ELF or AddressOfEntryPoint on PE.
5. Machine code — the numbers a processor obeys directly — encoded instructions for one instruction set architecture.
6. ASLR — placing a program at a random address each run — Address Space Layout Randomization, requiring position-independent code and relocations.

20.7 Anatomy of a PE file#

PLAIN20.7.1 in simple words#

1. Open any Windows .exe in a hex viewer and the first two letters are MZ.
2. Those are the initials of Mark Zbikowski, a Microsoft engineer who designed the executable header for MS-DOS in the early 1980s.
3. Right after that sits a tiny complete DOS program. Its only job is to print “This program cannot be run in DOS mode” and quit.
4. That stub is still there in 2026, in every Windows program you own, even though no one has run DOS in decades.
5. About 128 bytes in comes the letters PE followed by two zero bytes. That is where the modern file really starts.
6. After that come three tables: one describing the machine and how many pieces there are, one describing the whole image, and one listing the pieces.
7. Each piece is called a section, and each has a short name beginning with a dot: .text for code, .data for changeable data, .rdata for read-only data.
8. There is also a list of functions the program needs from other files, and if it is a library, a list of what it offers.
9. And one small number decides whether Windows opens a black console window for the program or not.

PLAIN20.7.2 a picture in your head#

1. Think of a very old building that has been refurbished many times.
2. At the front door there is still a Victorian gas lamp bracket. It has not carried gas since 1930. It is bolted to the wall and nobody removes it.
3. It costs almost nothing to leave, and taking it off risks damaging the wall.
4. The DOS stub is that gas lamp bracket. Sixty-four bytes of dead code plus a sentence.
5. Once you walk past it, the building inside is completely modern: a directory board in the lobby listing every floor, and every floor labelled by purpose.
6. The directory board is the section table. Each floor is a section.

Where this comparison breaks: the gas bracket really is useless, whereas the DOS stub has one live job left. The last four bytes before it, at offset 0x3C, hold the offset of the real PE header. Every Windows loader reads that number. So the stub region is not dead space; it contains the pointer that makes the whole format work. Removing the stub without fixing that pointer breaks the file.

PLAIN20.7.3 a worked example#

1. Here are the first 128 bytes of a real Windows executable we compiled for this chapter.

$ xxd -l 128 hello.exe
00000000: 4d5a 9000 0300 0000 0400 0000 ffff 0000  MZ..............
00000010: b800 0000 0000 0000 4000 0000 0000 0000  ........@.......
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 8000 0000  ................
00000040: 0e1f ba0e 00b4 09cd 21b8 014c cd21 5468  ........!..L.!Th
00000050: 6973 2070 726f 6772 616d 2063 616e 6e6f  is program canno
00000060: 7420 6265 2072 756e 2069 6e20 444f 5320  t be run in DOS 
00000070: 6d6f 6465 2e0d 0d0a 2400 0000 0000 0000  mode....$.......

2. Bytes 0 and 1 are 4d 5a, the letters MZ. That is the DOS header magic.
3. Look at offset 0x3c, which is on the fourth line: the four bytes 8000 0000. Little-endian, that is the number 0x00000080, decimal 128.
4. That is the promise: the real header begins at byte 128 of this file.
5. Between offset 0x40 and 0x78 sits the stub. The bytes 0e 1f ba 0e 00 b4 09 cd 21 are real 16-bit DOS instructions.
6. Decoded: push cs, pop ds, load the address of the message into dx, put 9 in ah, then int 21h which is the DOS system call to print a string.
7. Then b8 01 4c cd 21 is: put 0x4C01 in ax and call DOS again, which means terminate with exit code 1.
8. Then the message itself, ending with 0d 0d 0a 24. The 24 is a dollar sign, which is how DOS marked the end of a string.
9. Now jump to byte 128 and look:

$ xxd -s 128 -l 32 hello.exe
00000080: 5045 0000 6486 1300 5e24 7d6a 0032 0300  PE..d...^$}j.2..
00000090: 5307 0000 f000 2600 0b02 0229 006c 0000  S.....&....).l..

10. 5045 0000 is PE\0\0. The modern file starts here, exactly where the pointer said.

PLAIN20.7.4 what is really happening inside#

1. Read the bytes after PE\0\0 in order and every field tells you something useful.
2. 6486 little-endian is 0x8664, the code for the x86-64 processor family. An ARM Windows binary would say 0xAA64.
3. 1300 is 0x0013, decimal 19. The file has nineteen sections.
4. 5e24 7d6a is 0x6A7D245E, a Unix timestamp. Converted, that is 13 August 2026 at 01:56:46 UTC, which is when we compiled it.
5. f000 is 0x00F0, decimal 240: the size of the next header.
6. 0b02 is 0x020B, the marker meaning this is a 64-bit image. A 32-bit one would say 0x010B.
7. The 240-byte header that follows holds the entry point, the preferred load address, the alignment rules, the stack size, and a table of sixteen directories pointing at imports, exports, resources and relocations.
8. After that comes the section table: nineteen fixed-size records, each with a name, a size, a memory address, and a file offset.
9. The loader reads exactly this and nothing more before it starts mapping.

TECHNICAL20.7.5 the engineer’s version#

1. Real dump of the optional header of the file compiled for this chapter, trimmed to fit the page:

$ objdump -x hello.exe
architecture: i386:x86-64
start address 0x0000000140001410
Characteristics 0x26
  executable, line numbers stripped, large address aware
Time/Date        Thu Aug 13 01:56:46 2026
Magic            020b   (PE32+)
SizeOfCode              0000000000006c00
SizeOfInitializedData   0000000000009600
AddressOfEntryPoint     0000000000001410
BaseOfCode              0000000000001000
ImageBase               0000000140000000
SectionAlignment        00001000
FileAlignment           00000200
MajorSubsystemVersion   5
SizeOfImage             0003f000
CheckSum                00041982
Subsystem               00000003   (Windows CUI)
DllCharacteristics      00000160
    HIGH_ENTROPY_VA, DYNAMIC_BASE, NX_COMPAT
SizeOfStackReserve      0000000000200000
SizeOfHeapReserve       0000000000100000
NumberOfRvaAndSizes     00000010

2. Subsystem 3 is IMAGE_SUBSYSTEM_WINDOWS_CUI, a console program. Value 2 is IMAGE_SUBSYSTEM_WINDOWS_GUI. This single 16-bit field is the entire difference between a program that opens a black console window and one that does not.
3. ImageBase 0x140000000 is the default preferred base for a 64-bit executable. A 32-bit executable defaults to 0x400000 and a DLL to 0x10000000.
4. DYNAMIC_BASE means the image opts in to ASLR, so the loader will place it elsewhere and apply base relocations. NX_COMPAT means data pages can be marked non-executable. HIGH_ENTROPY_VA allows 64-bit address randomization.
5. SectionAlignment 0x1000 is the page size in memory. FileAlignment 0x200 is the alignment on disk. Sections are therefore padded differently in the file and in memory, which is why file offsets and virtual addresses differ.
6. The section table from the same binary:

$ objdump -h hello.exe
Idx Name       Size      VMA               File off
  0 .text      00006b68  0000000140001000  00000600
  1 .data      000000c0  0000000140008000  00007200
  2 .rdata     00000da0  0000000140009000  00007400
  3 .pdata     00000468  000000014000a000  00008200
  4 .xdata     0000050c  000000014000b000  00008800
  5 .bss       00000b80  000000014000c000  00000000
  6 .idata     000006d0  000000014000d000  00008e00
  7 .CRT       00000060  000000014000e000  00009600
  8 .tls       00000010  000000014000f000  00009800
  9 .reloc     00000084  0000000140010000  00009a00
 10 .debug_aranges ...

7. What the standard sections hold:

8. Note .bss has file offset 00000000. It occupies no space in the file at all. The loader simply allocates zeroed pages of the stated size. This is why a program with a 100 MB static buffer can be a 50 KB file.
9. The import table, dumped from the same binary:

$ objdump -p hello.exe
  DLL Name: KERNEL32.dll
  d350   281  DeleteCriticalSection
  d368   317  EnterCriticalSection
  d380   628  GetLastError
  d3d8  1034  MultiByteToWideChar
  d422  1489  VirtualProtect
  DLL Name: msvcrt.dll
  d49e    82  __getmainargs
  d4c8    97  __set_app_type
  d51c   219  _fmode
  d526   285  _initterm

10. Each imported name has a slot in the Import Address Table. Before loading, each slot holds a pointer to the name. After loading, the loader overwrites each slot with the real address of the function inside the loaded DLL. Calls go through the slot, so they cost one extra memory read.
11. The IAT location is Data Directory entry 12. In this file it is at RVA 0xd1c8, length 0x188, which is 392 bytes, or 49 pointers.
12. Because the IAT is a table of writable function pointers, hooking it is a classic technique for both debugging tools and malware. Overwrite the slot and every call to that function goes to your code instead.
13. A DLL adds an export table. We built a real DLL with two functions:

$ x86_64-w64-mingw32-gcc -shared -O2 -o mylib.dll mylib.c
$ file mylib.dll
mylib.dll: PE32+ executable (DLL) (console) x86-64,
           for MS Windows, 20 sections
$ objdump -p mylib.dll
Export Address Table -- Ordinal Base 1
  [0] +base[1] 1390 Export RVA
  [1] +base[2] 13a0 Export RVA
[Ordinal/Name Pointer] Table
  [0] add
  [1] mul

14. Every export has both a name and an ordinal number. Importing by ordinal is smaller and faster but breaks if the library is rebuilt with a different order. Importing by name is the normal choice.
15. The addresses in these tables are RVAs, relative virtual addresses: offsets from ImageBase. Add 0x140000000 to 0x1390 to get the real address if the image loads at its preferred base.
16. Tools that read PE files: objdump -x and objdump -p from GNU binutils, dumpbin /headers from Microsoft, llvm-readobj --coff-all, and the Python pefile library.

WORDS20.7.6 remember these#

1. MZ header — the two letters at the start of every Windows program — the DOS IMAGE_DOS_HEADER, whose e_lfanew field at offset 0x3C locates the PE header.
2. DOS stub — the small dead program after the MZ header — 16-bit code invoking int 21h to print a message and terminate.
3. Section — one labelled region of a program image — a record in the section table with a name, RVA, size, file offset and characteristic flags.
4. RVA — an address measured from the start of the image — Relative Virtual Address, added to ImageBase to get a linear address.
5. IAT — the table of addresses of borrowed functions — the Import Address Table, patched by the loader with resolved function pointers.
6. Subsystem — the flag deciding console or window — a 16-bit field; 2 is WINDOWS_GUI, 3 is WINDOWS_CUI, 10 to 13 are UEFI.
7. Base relocation — a fix-up applied when the image loads elsewhere — an entry in .reloc naming an address that must be adjusted by the load delta.

20.8 Anatomy of an ELF file#

PLAIN20.8.1 in simple words#

1. Every Linux program begins with byte 127 followed by the letters E, L, F. That is the whole signature.
2. After it comes a small header of 64 bytes that answers the basic questions: 32-bit or 64-bit, which byte order, which processor, where to start.
3. Then the file has two separate tables of contents, and this surprises people.
4. One table is for the loader: it lists a handful of big chunks to map into memory. These are called segments.
5. The other table is for the linker and for tools: it lists many small named pieces such as .text and .data. These are called sections.
6. Both tables describe the same bytes, grouped differently, for two different readers.
7. When a program runs, only the first table matters. The second can be deleted entirely and the program still runs.
8. Most Linux programs also name a helper program inside themselves: the dynamic linker, which finds the shared libraries at start-up.
9. That helper is usually /lib64/ld-linux-x86-64.so.2, and it is itself an ELF file.

PLAIN20.8.2 a picture in your head#

1. Think of a large printed cookbook that has two indexes at the back.
2. The first index is by chapter: Starters, Mains, Puddings. Three entries, big blocks, useful when you want to open the book roughly in the right place.
3. The second index is by recipe name: four hundred entries, precise, useful when you want one exact thing.
4. Both indexes describe the same pages. Neither is wrong. They serve different readers.
5. A cook in a hurry uses the chapter index. A librarian cataloguing the book uses the recipe index.
6. The loader is the cook in a hurry: it wants three or four big blocks and the permissions for each.
7. The linker and the debugger are the librarian: they want the exact name and boundary of every small piece.

Where this comparison breaks: you cannot tear the recipe index out of a cookbook and still have the chapter index work, because they refer to page numbers that would not change. In ELF you genuinely can delete the whole section table. The strip command does this, the file gets smaller, and the program runs identically. There is no equivalent to that in a book, and it is the clearest proof that the two tables truly serve different masters.

PLAIN20.8.3 a worked example#

1. Here is the start of a real Linux executable compiled for this chapter, sixty-four bytes, which is exactly the ELF header.

$ xxd -l 64 hello
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0300 3e00 0100 0000 8010 0000 0000 0000  ..>.............
00000020: 4000 0000 0000 0000 9836 0000 0000 0000  @........6......
00000030: 0000 0000 4000 3800 0d00 4000 1f00 1e00  ....@.8...@.....

2. 7f 45 4c 46 is the magic: 127, E, L, F.
3. 02 means 64-bit. 01 would mean 32-bit.
4. The next 01 means little-endian byte order. 02 would mean big-endian.
5. 01 again is the ELF version, which has been 1 since 1989 and has never changed.
6. 0300 is 0x0003, the type. Type 3 is a shared object, which is what a modern position-independent executable is.
7. 3e00 is 0x003E, decimal 62, the code for x86-64.
8. 8010 0000 ... is the entry point: address 0x1080.
9. 4000 ... is 64: the program header table starts at byte 64, immediately after this header.
10. 9836 0000 is 0x3698, decimal 13976: where the section header table starts, near the end of the file.
11. And the tool agrees with our hand reading:

$ readelf -h hello
  Class:                ELF64
  Data:                 2's complement, little endian
  Type:                 DYN (Position-Independent Executable)
  Machine:              Advanced Micro Devices X86-64
  Entry point address:  0x1080
  Start of program headers:  64 (bytes into file)
  Start of section headers:  13976 (bytes into file)
  Size of this header:       64 (bytes)
  Size of program headers:   56 (bytes)
  Number of program headers: 13
  Size of section headers:   64 (bytes)
  Number of section headers: 31

PLAIN20.8.4 what is really happening inside#

1. Thirteen program headers, thirty-one section headers. Only the thirteen matter at run time, and of those only four are of type LOAD.
2. Each LOAD entry says: take this many bytes from this file offset, put them at this virtual address, and give them these permissions.
3. One LOAD entry has read and execute permission. That is the code. One has read and write. That is the data.
4. A separate entry of type INTERP holds a plain text string: the path of the dynamic linker. On this machine it is /lib64/ld-linux-x86-64.so.2.
5. If that entry is present, the kernel loads the named program too, and gives control to it, not to your program.
6. The dynamic linker then reads a special region called the dynamic section, which is a list of tagged values.
7. One tag, NEEDED, appears once per required library. Our program has exactly one: libc.so.6.
8. Other tags point at the symbol table, the string table, the relocation lists, and the initialization function.
9. The dynamic linker opens each needed library, maps it, then walks the relocation lists patching addresses until every borrowed function has a real address.
10. Only then does it jump to your entry point.

TECHNICAL20.8.5 the engineer’s version#

1. Program headers of the same binary, as a table. FileSiz and MemSiz differ only for the writable segment, because .bss occupies memory but not file.

2. GNU_STACK with RW and no E is what makes the stack non-executable. If this header were missing or had E set, the kernel would map an executable stack.
3. GNU_RELRO marks a region that the dynamic linker makes read-only after relocation is finished. It covers .init_array, .fini_array, .dynamic and, with full RELRO, the global offset table. This blocks a whole class of pointer-overwrite attacks.
4. The interpreter string is stored literally in the file and reported by readelf -l as [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2].
5. Section headers, abbreviated from the real output of readelf -SW hello:

6. .bss has type NOBITS: no bytes in the file. .symtab has address 0: it is never loaded into memory.
7. The dynamic section, real output, trimmed:

$ readelf -d hello
 (NEEDED)      Shared library: [libc.so.6]
 (INIT)        0x1000
 (FINI)        0x116c
 (INIT_ARRAY)  0x3db8
 (GNU_HASH)    0x3b0
 (STRTAB)      0x480
 (SYMTAB)      0x3d8
 (PLTGOT)      0x3fb8
 (JMPREL)      0x610
 (RELA)        0x550
 (FLAGS)       BIND_NOW
 (FLAGS_1)     Flags: NOW PIE

8. BIND_NOW with full RELRO means every symbol is resolved at start-up rather than lazily on first call. This is the default on Ubuntu and most modern distributions. Lazy binding is faster to start and weaker to attack.
9. Relocations from the same file:

$ readelf -r hello
Relocation section '.rela.dyn':
  R_X86_64_RELATIVE                      1160
  R_X86_64_RELATIVE                      1120
  R_X86_64_GLOB_DAT  __libc_start_main@GLIBC_2.34
  R_X86_64_GLOB_DAT  __cxa_finalize@GLIBC_2.2.5
Relocation section '.rela.plt':
  R_X86_64_JUMP_SLOT puts@GLIBC_2.2.5

10. R_X86_64_RELATIVE entries need no symbol lookup. They just say “add the load base to the value stored here”, and there are three because the image is position independent.
11. Static versus dynamic linking, measured on this machine with the same source:

12. The static binary is 49 times larger because the entire C library is copied in. It has no INTERP header, no NEEDED entries, and ldd reports it as “not a dynamic executable”.
13. Stripping removed .symtab and .strtab, cutting 1,488 bytes and reducing the section count from 31 to 29. nm then reports “no symbols”. The program runs identically; only debugging is harder.
14. PIE, position-independent executable, is Type: DYN with an entry point of 0x1080, a small number, because the real base is decided at load time. Building with -no-pie produced Type: EXEC with entry 0x401070, a fixed address. PIE is the default on Ubuntu since 16.10 and on Debian since Stretch in 2017.
15. Tools that read ELF: readelf, objdump, nm, ldd, eu-readelf from elfutils, llvm-readelf, and LD_DEBUG=libs to watch the dynamic linker work.

WORDS20.8.6 remember these#

1. Segment — one chunk the loader maps — a program header entry of type LOAD with a file offset, virtual address, sizes and permission flags.
2. Section — one named region for tools — a section header entry used by the linker, debugger and stripper, ignored at run time.
3. Dynamic section — the run-time instruction list — an array of tagged values including DT_NEEDED, DT_STRTAB, DT_RELA and DT_INIT.
4. Interpreter — the helper that starts your program — the path in the PT_INTERP header, normally ld.so, loaded by the kernel before your code.
5. PIE — a program with no fixed address — Position-Independent Executable, type ET_DYN, relocated to a random base by the kernel.
6. Stripping — deleting the names from a binary — removing .symtab and .strtab, shrinking the file without changing behaviour.
7. RELRO — making some data read-only after start-up — RELocation Read-Only, applied by ld.so via mprotect after relocations complete.

20.9 What happens when you double-click a program#

PLAIN20.9.1 in simple words#

1. You double-click an icon. A fraction of a second later a window appears. A great deal happened in between.
2. First, something has to work out which file you meant and whether you are allowed to run it.
3. Then a request goes to the core of the operating system: please replace a process with this program.
4. The core opens the file, reads the first bytes, and checks that it recognizes the format and that the instructions are for this processor.
5. It then throws away the old program’s memory entirely and builds a fresh, empty memory space.
6. It maps the pieces of the file into that space at the addresses the header asked for.
7. It writes your command line arguments and the environment settings onto the new program’s stack, so the program can find them.
8. If the program needs shared libraries, the core starts a helper program first and hands control to it.
9. The helper finds each library, maps it in, and fills in every borrowed address.
10. Only then does control reach your program’s real starting point, and even that is not your code yet. A short start-up routine runs first, and it calls your main.

PLAIN20.9.2 a picture in your head#

1. Think of an empty theatre and a script arriving at the stage door.
2. The stage manager checks the envelope: right theatre, right company, do we have permission to perform this.
3. The stage crew clears the stage completely. Everything from the last performance goes.
4. The set pieces are then brought in and placed at marked positions on the floor. The script says exactly where each goes.
5. A list of props is pinned up: the ones we own, and the ones we must borrow from the props house next door.
6. A runner is sent to the props house. He fetches each borrowed item and puts it in its marked place.
7. Only when every marked place is filled does the stage manager call “curtain up”, and the first line is spoken.

Where this comparison breaks: a real stage is fully dressed before the curtain rises. A program is not. The loader mostly writes down promises rather than moving anything. A page of code is fetched from disk only when the processor first tries to execute it, causing a page fault that the kernel quietly services. A large program can start running having read only a few dozen kilobytes of its own file. The set is built while the play is already running.

PLAIN20.9.3 a worked example#

1. We traced the system calls of the tiny program from earlier. This is the real output, lightly trimmed to fit the page.

$ strace ./hello
execve("./hello", ["./hello"], 0x7ffe.. /* 151 vars */) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, ...)  = 0x7ff5dfab6000
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
mmap(NULL, 54027, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff5dfaa8000
close(3)                                     = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\3\0\0..."..., 832)    = 832
mmap(NULL, 2170256, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff5df800000
mmap(0x7ff5df828000, 1605632, PROT_READ|PROT_EXEC, ...)
mmap(0x7ff5df9b0000, 323584, PROT_READ, ...)
mmap(0x7ff5df9ff000, 24576, PROT_READ|PROT_WRITE, ...)
mmap(0x7ff5dfa05000, 52624, PROT_READ|PROT_WRITE|ANONYMOUS)
close(3)                                     = 0
arch_prctl(ARCH_SET_FS, 0x7ff5dfaa5740)      = 0
write(1, "hello, world\n", 13)               = 13
exit_group(0)                                = ?
+++ exited with 0 +++

2. Count the lines. Seventeen system calls to print thirteen characters. Only one of them, the write, is our program doing our work.
3. Look at the four mmap calls for libc.so.6. Each one maps a different segment with different permissions: read only, read plus execute, read only again, then read plus write.
4. The last one has ANONYMOUS and no file. That is .bss: zeroed memory backed by nothing on disk.
5. read(3, "\177ELF...", 832) is the dynamic linker reading the first 832 bytes of the library to parse its ELF header and program headers.
6. Here is the same journey as a picture.

   you double-click
          |
          v
  +-------------------+   name, permissions, association
  |  shell / file mgr |-------------------------+
  +-------------------+                         |
          | fork + execve                       |
          v                                     v
  +-------------------+              +----------------------+
  |  KERNEL           |              | check +x bit, ACLs   |
  |  read first bytes |              +----------------------+
  |  match binfmt     |
  |  new address space|
  |  map LOAD segments|
  |  build stack:     |   argv, envp, auxv
  |  load PT_INTERP   |
  +-------------------+
          | jump to ld.so entry
          v
  +-------------------+
  |  ld.so            |  find libs, mmap them,
  |  dynamic linker   |  apply relocations, run
  +-------------------+  library init functions
          | jump to e_entry
          v
  +-------------------+
  |  _start           |  set up argc/argv
  |  __libc_start_main|  init stdio, TLS, locale
  +-------------------+
          | call
          v
       main(argc, argv)
          | return
          v
   exit -> atexit handlers -> _exit -> exit_group
          |
          v
   kernel frees memory, notifies parent

PLAIN20.9.4 what is really happening inside#

1. Here is the whole sequence in thirty numbered steps. Steps are given for Linux; the Windows path is noted in the technical block below.
2. Step 1. The file manager receives a double-click on an icon and resolves it to a full path, following any symbolic links or shortcuts.
3. Step 2. It decides how to run it. On Linux it checks whether the file is executable; on Windows it consults the registry association for the extension.
4. Step 3. It checks permission. On Linux, the execute bit must be set for you. On Windows, an access control list is evaluated.
5. Step 4. The file manager calls fork, creating a child process that is a copy of itself.
6. Step 5. The child calls execve with the path, the argument list, and the environment list.
7. Step 6. The kernel opens the file and reads the first 256 bytes into a buffer.
8. Step 7. The kernel walks its list of registered binary format handlers, offering the buffer to each in turn.
9. Step 8. The ELF handler checks for 7f 45 4c 46 and accepts.
10. Step 9. It checks the class field, 64-bit here, and the machine field, 62 for x86-64, against the running kernel. A mismatch fails with ENOEXEC.
11. Step 10. The kernel checks the set-user-id and set-group-id bits and decides the new credentials.
12. Step 11. The kernel destroys the old address space: all mappings from the previous program are released.
13. Step 12. A brand-new, empty address space is created for the process.
14. Step 13. The kernel reads the program header table and iterates over the PT_LOAD entries.
15. Step 14. For each, it creates a mapping from the file into memory at the stated address, with the stated permissions. Nothing is copied yet.
16. Step 15. For a PT_LOAD whose memory size exceeds its file size, the extra is mapped as anonymous zeroed pages. That is .bss.
17. Step 16. For a position-independent executable, the kernel picks a random base address and adds it to every mapping address.
18. Step 17. The kernel creates the stack, at a randomized address, sized by the process resource limit, usually 8 MB.
19. Step 18. It writes onto that stack, from the top down: the environment strings, the argument strings, then pointer arrays for each, then a table of auxiliary values called the auxiliary vector.
20. Step 19. The auxiliary vector carries facts the program cannot easily discover otherwise: page size, the address of the program headers, the entry point, the real and effective user ids, and a pointer to sixteen random bytes.
21. Step 20. If the file has a PT_INTERP header, the kernel opens that path, maps its segments too, and remembers its entry point.
22. Step 21. The kernel sets the instruction pointer to the interpreter’s entry point, not the program’s, and returns to user mode.
23. Step 22. The dynamic linker begins. It relocates itself first, since nothing has fixed its own addresses yet.
24. Step 23. It reads the program’s dynamic section and collects every DT_NEEDED library name.
25. Step 24. For each name, it searches: DT_RPATH and DT_RUNPATH in the file, then LD_LIBRARY_PATH, then the cache in /etc/ld.so.cache, then the default directories.
26. Step 25. Each found library is opened and mapped exactly as the main program was, and its own dependencies are added to the queue.
27. Step 26. With every object loaded, the linker resolves symbols. For each relocation entry, it looks up the symbol name in the load order and writes the resolved address into the named slot.
28. Step 27. With full RELRO enabled, it then calls mprotect to make the global offset table and the dynamic section read-only.
29. Step 28. It runs the initialization functions of every library, in dependency order, then those of the program itself, from DT_INIT and DT_INIT_ARRAY.
30. Step 29. It jumps to the program’s real entry point, _start, which is not your code. _start arranges the arguments into registers and calls __libc_start_main.
31. Step 30. __libc_start_main sets up thread-local storage, initializes the standard input, output and error streams, registers the cleanup handler, and finally calls main. When main returns, exit runs the registered handlers, flushes buffers, calls destructors, and issues exit_group, at which point the kernel frees everything and stores the exit status for the parent.

TECHNICAL20.9.5 the engineer’s version#

1. On Linux the kernel entry point is execve(2), implemented in fs/exec.c. It fills a struct linux_binprm, reads BINPRM_BUF_SIZE bytes, currently 256, and calls search_binary_handler.
2. Registered handlers in a typical kernel: binfmt_elf, binfmt_script, binfmt_misc, and binfmt_elf_fdpic on some embedded targets. The old binfmt_aout was removed from mainline Linux in 2022.
3. Mapping is done with vm_mmap using MAP_PRIVATE, so pages are copy-on-write and shared between every process running the same binary until written.
4. The auxiliary vector entries most often used: AT_PHDR, AT_PHENT, AT_PHNUM, AT_ENTRY, AT_PAGESZ, AT_RANDOM, AT_SECURE, AT_HWCAP and AT_HWCAP2. You can print them with LD_SHOW_AUXV=1 ./program.
5. AT_RANDOM points at sixteen kernel-supplied random bytes. glibc uses them to seed the stack canary. This is why the canary differs on every run.
6. The dynamic linker’s work can be observed directly:

$ LD_DEBUG=libs ./hello
  find library=libc.so.6 [0]; searching
   search cache=/etc/ld.so.cache
    trying file=/lib/x86_64-linux-gnu/libc.so.6
  calling init: /lib64/ld-linux-x86-64.so.2
  calling init: /lib/x86_64-linux-gnu/libc.so.6
  initialize program: ./hello

$ LD_DEBUG=bindings ./hello
  binding file ./hello [0] to libc.so.6 [0]:
    normal symbol `__libc_start_main' [GLIBC_2.34]
  binding file ./hello [0] to libc.so.6 [0]:
    normal symbol `puts' [GLIBC_2.2.5]

7. Note that our source called printf but the binding is to puts. The compiler replaced a printf with a constant string and no format specifiers by the cheaper puts. Verified in the disassembly:

$ objdump -d hello
0000000000001060 <main>:
  1060: f3 0f 1e fa    endbr64
  1064: 48 83 ec 08    sub    $0x8,%rsp
  1068: 48 8d 3d 95..  lea    0xf95(%rip),%rdi
  106f: e8 dc ff ff ff call   1050 <puts@plt>
  1074: 31 c0          xor    %eax,%eax
  107a: c3             ret

8. The resulting address space, from /proc/PID/maps of a running process, shows the pattern clearly: five mappings for the executable and five for libc, each with different permissions.

55ffa11de000-55ffa11e0000 r--p /usr/bin/sleep
55ffa11e0000-55ffa11e4000 r-xp /usr/bin/sleep
55ffa11e4000-55ffa11e5000 r--p /usr/bin/sleep
55ffa11e6000-55ffa11e7000 rw-p /usr/bin/sleep
55ffdd305000-55ffdd326000 rw-p [heap]
7f9296600000-7f9296628000 r--p libc.so.6
7f9296628000-7f92967b0000 r-xp libc.so.6
7f92967b0000-7f92967ff000 r--p libc.so.6
7f9296803000-7f9296805000 rw-p libc.so.6

9. On Windows the equivalent path is: CreateProcess in kernel32 calls NtCreateUserProcess in ntdll, the kernel creates a section object for the image, maps it, and starts a thread at RtlUserThreadStart. The user-mode loader in ntdll, often called LdrpInitializeProcess, then walks the import directory, loads each DLL, calls each DLL’s DllMain with DLL_PROCESS_ATTACH, patches the IAT, and finally jumps to AddressOfEntryPoint.
10. On macOS the kernel’s execve recognizes Mach-O, maps segments, and loads /usr/lib/dyld, which reads LC_LOAD_DYLIB commands, consults the shared cache, binds symbols and calls LC_MAIN.
11. Approximate cost on the machine used here: execve plus dynamic linking for a one-library program completes in under two milliseconds. A large application linking against forty shared libraries can spend tens of milliseconds purely in symbol resolution, which is why prelink existed and why static linking is popular again for command-line tools.
12. Tools that observe each stage: strace -f, ltrace, LD_DEBUG, LD_SHOW_AUXV, perf trace, /proc/PID/maps, and on Windows Process Monitor and the !dlls command in WinDbg.

WORDS20.9.6 remember these#

1. exec — replacing a process with a new program — the execve(2) family, which discards the old address space and never returns on success.
2. fork — making a copy of a process — fork(2), creating a child with a copy-on-write duplicate of the parent’s address space.
3. Auxiliary vector — a list of facts the kernel hands the new program — the auxv array of type-value pairs on the initial stack.
4. Page fault — the moment a page is actually fetched — a processor exception the kernel services by reading from the mapped file or allocating a zero page.
5. Copy-on-write — sharing memory until someone writes — mapping pages read-only and duplicating them on the first write fault.
6. Lazy binding — resolving a function’s address on first call — PLT stubs that call into the linker once, then patch the GOT slot.
7. C runtime start-up — the code that runs before main — _start and __libc_start_main, which initialize TLS, stdio and atexit.

20.10 Installers and packaging#

PLAIN20.10.1 in simple words#

1. An installer is just a program whose job is to put other files in the right places.
2. There is nothing sacred about installing. You could do all of it by hand with a file manager, given enough patience.
3. A typical installer does five things.
4. It copies the program’s files into a folder, usually somewhere central rather than in your downloads.
5. It tells the system which file types this program can open, so double-clicking works.
6. It creates shortcuts, in a menu, on a desktop, or in a dock.
7. It writes settings somewhere the program can find them later.
8. And it records enough information for the program to be removed cleanly.
9. That last one is the part most often done badly, which is why uninstalling often leaves rubbish behind.

PLAIN20.10.2 a picture in your head#

1. Think of a new appliance being delivered to a flat.
2. The delivery team carries it to the right room. That is copying the files.
3. They plug it in and register it with the building’s electrics. That is registering file types.
4. They put a label on the fuse box saying which switch controls it. That is the shortcut.
5. They fill in a form for the building manager: what was installed, when, and what has to be undone to remove it. That is the uninstall record.
6. A careless team dumps the box in the hallway and leaves. Later, nobody knows what to remove.

Where this comparison breaks: an appliance stays where it was put. Software changes the building. It can alter shared settings, replace shared libraries that other programs also use, and add background services that start every time the building’s power comes on. Removing it is therefore not the reverse of a delivery. It is closer to undoing a renovation, which is why package managers that track every file exist at all.

PLAIN20.10.3 a worked example#

1. A Debian package is not a mysterious binary. It is an ar archive, the same ancient Unix format used for static libraries.

$ xxd -l 24 zstd_1.5.5+dfsg2-2build1.1_amd64.deb
00000000: 213c 6172 6368 3e0a 6465 6269 616e 2d62  !<arch>.debian-b
00000010: 696e 6172 7920 2020                      inary

$ ar t zstd_1.5.5+dfsg2-2build1.1_amd64.deb
debian-binary
control.tar.zst
data.tar.zst

$ dpkg -I zstd_1.5.5+dfsg2-2build1.1_amd64.deb
 new Debian package, version 2.0.
 size 644020 bytes: control archive=1069 bytes.
     755 bytes,  17 lines   control
    1064 bytes,  17 lines   md5sums
 Package: zstd
 Version: 1.5.5+dfsg2-2build1.1
 Architecture: amd64
 Installed-Size: 1802
 Depends: libc6 (>= 2.34), libgcc-s1 (>= 3.3.1),
          liblz4-1 (>= 1.8.0), liblzma5, libstdc++6,
          zlib1g (>= 1:1.1.4)
 Section: utils
 Priority: optional

2. Three members. debian-binary is a text file containing 2.0, the format version.
3. control.tar.zst holds the metadata: package name, version, dependencies, checksums, and optional scripts to run before and after install.
4. data.tar.zst holds the actual files, with their full destination paths inside.
5. Installing is therefore: read the control data, check the dependencies are satisfied, unpack the data archive at the root of the file system, record the file list, run the post-install script.
6. Uninstalling is exact, because the file list was recorded. This is the whole advantage of a package manager over an installer program.

PLAIN20.10.4 what is really happening inside#

1. Different systems answer the same problem with different shapes.
2. Windows historically used self-contained installer programs: a single .exe that unpacks itself and does the work in code.
3. Microsoft’s own answer, the Windows Installer, replaced code with data. An .msi file is a small relational database of tables: which files, which registry keys, which shortcuts, in which order.
4. Because it is data, the system itself performs the installation, can roll back a failure, and can uninstall precisely.
5. macOS mostly avoids installers. An application is a folder whose name ends in .app, containing the executable, the icons and everything else.
6. The Finder shows that folder as a single item. Dragging it to Applications is the entire installation.
7. Linux distributions use package managers with a central database and hard dependency rules, which is why installing one thing can pull in twenty others.
8. The newer Linux formats invert this. Instead of sharing libraries, each app carries its own, so it works on any distribution at the cost of size.
9. And on every platform there is now a signing layer: the operating system checks a cryptographic signature before running the installer, and warns you loudly if there is not one.

TECHNICAL20.10.5 the engineer’s version#

1. Windows packaging formats:

2. Windows Installer shipped in 1999 alongside Office 2000. An MSI file is actually an OLE compound document holding roughly eighty defined tables such as File, Component, Registry, Shortcut and InstallExecuteSequence.
3. NSIS was written at Nullsoft for distributing Winamp and released in 2000. It produces a single executable containing a compressed payload and a compiled script. It is the most common format for open-source Windows applications.
4. Uninstall information on Windows lives under HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall, one key per product, with DisplayName, UninstallString and InstallLocation.
5. macOS packaging:

6. An .app bundle contains Contents/Info.plist declaring the identifier, version and document types, Contents/MacOS/ with the Mach-O executable, and Contents/Resources/ with icons and localizations. It is an ordinary directory; cd into it from a terminal and everything is visible.
7. Linux packaging: .deb from Debian, described above; .rpm from Red Hat, created in the mid-1990s, consisting of a lead, a signature header, a metadata header, and a compressed cpio payload.
8. The distribution-independent formats:

9. An AppImage is a single executable file: a small runtime concatenated with a SquashFS image. Running it mounts the image with FUSE and executes the application inside. It requires no installation and no root.
10. Flatpak shares common runtimes between applications through OSTree deduplication and sandboxes each app with bubblewrap and portals. Snap uses compressed SquashFS images mounted as loop devices and confined by AppArmor.
11. Code signing by platform:

12. Authenticode embeds a PKCS#7 signature in the PE file, referenced by Data Directory entry 4, the Security Directory. The signed hash deliberately excludes the checksum field and the certificate table itself, so those can change without invalidating the signature.
13. Extended Validation certificates and, since 2023, the requirement that code-signing private keys live on hardware tokens or in a hardware security module, exist because stolen signing keys were used to sign malware. The Stuxnet worm, discovered in 2010, was signed with certificates stolen from two Taiwanese hardware companies.
14. Apple’s Gatekeeper arrived with OS X 10.8 Mountain Lion in 2012. From macOS 10.15 Catalina in 2019, software distributed outside the App Store must also be notarized: uploaded to Apple, scanned, and issued a ticket that the system checks.
15. Windows SmartScreen, part of Windows since version 8 in 2012, warns about executables with little reputation history regardless of signing. A newly signed application still triggers it until enough installs accumulate. This is a reputation system, not a signature check, and the two are often confused.
16. Linux distributions sign the package index rather than each package. apt verifies a detached GPG signature on the Release file, which contains hashes of the package lists, which contain hashes of the packages. Trust flows down that chain.

WORDS20.10.6 remember these#

1. Installer — a program that puts software in place — a bootstrapper performing file placement, registration and uninstall bookkeeping.
2. MSI — the Windows database installer format — an OLE compound file of installation tables executed by the Windows Installer service.
3. Bundle — a macOS application folder shown as one item — a directory with a defined layout and an Info.plist describing it.
4. Package manager — a tool that tracks every installed file — a database-backed system resolving dependencies and enabling exact removal.
5. Sandbox — a fence around a running application — a confinement mechanism such as bubblewrap, AppArmor or the macOS App Sandbox.
6. Code signing — proving who built a program — attaching a signature over a hash of the image, verified against a trusted certificate chain.
7. Notarization — Apple checking your app before users run it — submission to Apple’s service, producing a ticket stapled to the distribution.

20.11 Scripts versus binaries#

PLAIN20.11.1 in simple words#

1. A .sh or .py file contains no machine instructions at all. It is plain text a human wrote.
2. Yet you can mark it executable and run it exactly like a program. Something must be bridging that gap.
3. The bridge is two characters at the very start of the file: a hash and an exclamation mark, written #!.
4. Those two characters are followed by the path of a real program.
5. When you run the file, the system reads that first line, and instead of trying to execute your text, it runs the named program and hands it your file.
6. So ./script.sh really becomes /bin/sh ./script.sh behind your back.
7. The named program is called an interpreter. It reads your text line by line and does what it says.
8. The interpreter itself is an ordinary compiled program, with an ELF or PE header, exactly as described earlier.
9. So nothing was ever magic. One real program is running, and your file is its input.

PLAIN20.11.2 a picture in your head#

1. Think of a sealed envelope handed to a receptionist.
2. On the outside is written: “Please pass to the French translator.”
3. The receptionist does not read the letter. She reads only the instruction on the outside and walks the envelope to the right desk.
4. The translator opens it and works through the contents.
5. The #! line is the writing on the outside of the envelope. The kernel is the receptionist. The interpreter is the translator.
6. If the writing names a desk that does not exist, the receptionist comes back with an error, and the error is confusingly about the envelope, not the desk.

Where this comparison breaks: a receptionist can see the letter is in French even without the label, and would work it out. The kernel cannot and does not try. It reads exactly two characters. If they are not #!, it gives up with an error called “exec format error”, and any success after that is the shell doing extra work of its own accord, not the kernel being clever.

PLAIN20.11.3 a worked example#

1. We wrote two shell scripts, identical except that one has a #! line.

$ cat s1.sh
#!/bin/sh
echo "I am a script, arg1=$1"

$ xxd -l 16 s1.sh
00000000: 2321 2f62 696e 2f73 680a 6563 686f 2022  #!/bin/sh.echo "

$ ./s1.sh hello
I am a script, arg1=hello

$ file s1.sh
s1.sh: POSIX shell script, ASCII text executable

2. The first two bytes are 23 21, which are # and !. That is the whole mechanism.
3. Now the file without a #! line, run two ways.

$ cat s2.sh
echo "no shebang here"

$ ./s2.sh
no shebang here

$ python3 -c "import os; os.execve('/tmp/ch20/s2.sh',
              ['s2.sh'], {})"
execve failed: [Errno 8] Exec format error: 's2.sh'

4. Read that carefully. The shell ran it fine. A direct execve refused.
5. The kernel genuinely could not run the file. Error 8 is ENOEXEC, no recognized format.
6. When the shell got the same error, it did something extra: it decided to run the file itself, as shell commands. That is a rule in the shell, defined by POSIX, not a kernel feature.
7. So the file “worked” for a reason that has nothing to do with the file being executable at the system level.

PLAIN20.11.4 what is really happening inside#

1. Inside the kernel, running any file goes through a list of format handlers, tried in order.
2. One of them, on Linux called binfmt_script, is very simple.
3. It checks whether the first two bytes are # and !.
4. If so, it reads up to a limit of characters, stopping at the first newline.
5. It splits that into the interpreter path and, on Linux, at most one further argument.
6. It then restarts the whole exec operation, but with the interpreter as the program and with the argument list rewritten.
7. The new argument list is: the interpreter path, the optional argument from the #! line, the original file path, and then your original arguments.
8. That restart is limited. A script whose interpreter is itself a script is allowed, but only to a fixed depth, to stop infinite loops.
9. Windows has no #! mechanism in the kernel. It uses file associations instead: .py is registered to the Python launcher, which then reads a #! line itself as a convention borrowed from Unix.

TECHNICAL20.11.5 the engineer’s version#

1. The mechanism is implemented in fs/binfmt_script.c in the Linux kernel. The #! sequence is often called the shebang, or hash-bang.
2. It was added to Unix by Dennis Ritchie around 1980 and first appeared in the Eighth Edition of Research Unix. It is specified in POSIX only as an unspecified behaviour, meaning portable scripts may rely on it in practice but it is not formally guaranteed.
3. Line length limits differ and this matters for portability:

4. Because Linux allows only one argument, #!/usr/bin/env python3 -u passes python3 -u to env as a single string on Linux and fails. Writing #!/usr/bin/env -S python3 -u works on GNU coreutils 8.30 and later, released in 2018.
5. #!/usr/bin/env python3 is the common idiom because it searches PATH instead of hard-coding a location, which differs across distributions and inside virtual environments.
6. A shebang must be an absolute path. The kernel does no PATH search. That is the entire reason env is used as an indirection.
7. If a script has CRLF line endings, the interpreter path becomes /bin/sh\r, which does not exist. The resulting message, bad interpreter: No such file or directory, is one of the most confusing errors in Unix, because the path printed looks correct on screen.
8. binfmt_misc, added to Linux in 1997, generalizes this. You register a magic number or extension with an interpreter path by writing to /proc/sys/fs/binfmt_misc/register.
9. This is how a Linux system transparently runs ARM binaries under QEMU, and how Windows executables can be handed to Wine automatically. It was not mounted in the sandbox used for this chapter, which is common inside containers.
10. The interpreter receives the script by path, not on standard input. It can therefore seek in the file, which is why Python can read a .pyc cache and why a shell script can safely have binary data appended to it after an exit line.
11. Cost comparison, approximate and measured informally: starting a compiled C binary that prints one line takes about 1 ms on the machine used here. Starting the same task through python3 takes roughly 20 to 40 ms, dominated by importing the interpreter’s own startup modules.
12. The honest version: saying “scripts are interpreted and binaries are compiled” is a useful lie. CPython compiles your source to bytecode before running it. Java compiles to bytecode and then to machine code at run time. Modern JavaScript engines do the same. The real distinction is when the translation happens and who ships the result, not whether translation happens.

WORDS20.11.6 remember these#

1. Shebang — the #! at the top of a script — the two-byte magic recognized by binfmt_script, followed by an absolute interpreter path.
2. Interpreter — the program that reads and runs your text — an ordinary executable receiving the script path as an argument.
3. ENOEXEC — the error meaning “I cannot run this” — errno 8, returned by execve when no binary format handler accepts the file.
4. binfmt_misc — the Linux feature for registering new formats — a kernel module mapping magic bytes or extensions to interpreter paths.
5. env — the tool used to find an interpreter on PATH — /usr/bin/env, run from the shebang so the real path need not be hard-coded.
6. Bytecode — a compact instruction set for a virtual machine — the intermediate form produced by CPython, javac and similar compilers.

20.12 Archives and compression in practice#

PLAIN20.12.1 in simple words#

1. Two different jobs get muddled together: putting many files into one, and making things smaller.
2. Putting many files into one is called archiving. Making things smaller is called compression.
3. ZIP does both jobs in one format. It squeezes each file separately and then packs the squeezed pieces together.
4. Unix split the jobs. tar packs files together and does no squeezing at all. gzip squeezes one stream and knows nothing about files.
5. That is why you see .tar.gz: first tar joined everything into one stream, then gzip squeezed that whole stream.
6. The order matters enormously. Squeezing one big stream finds repetition between files. Squeezing each file alone cannot.
7. Squeezing everything as one stream is called solid compression. It gives smaller results and makes extracting a single file slower.
8. Different squeezing methods trade speed against size, and the trade is real and measurable.
9. There is no best one. There is a fastest, a smallest, and several sensible middles.

PLAIN20.12.2 a picture in your head#

1. Think of packing a suitcase for a family of four.
2. Method one: each person rolls their own clothes tightly, and the four bundles go in. Anyone can pull out their own bundle without disturbing the others.
3. Method two: all the clothes are laid out together and vacuum-packed as one slab. It takes far less space, because socks fill the gaps between shirts.
4. But to get one shirt you must open the whole slab and repack it.
5. Method one is ZIP. Method two is tar plus gzip, or a solid 7z archive.
6. If everyone’s clothes are similar, method two wins by a lot, because the machine notices the repetition across people.

Where this comparison breaks: vacuum packing removes air, which is a fixed saving. Compression removes repetition, which is not fixed at all. A slab of already-random data compresses by nothing, and can come out slightly larger because of the bookkeeping added. There is no method that shrinks every possible input. That is not an engineering limit; it is a counting argument, and it is provably true.

PLAIN20.12.3 a worked example#

1. We compressed the same 47,575,040 bytes of C header files, all real text, with six settings on one machine, and timed both directions.

2. Read the first two rows together. zstd -3 produced a smaller file than gzip -6 and did it about seven times faster.
3. Now read the top and bottom rows. xz -6 is 30 percent smaller than zstd -3, and took 85 times longer.
4. Decompression times tell a different story again, and this is the number that matters for software you ship.

5. Note that zstd decompresses at the same speed whether it was compressed at level 3 or level 19. Higher levels cost the packer more, not the unpacker.
6. That single property is why zstd replaced gzip in Debian and Ubuntu packages, in Arch Linux packages, and in the Linux kernel’s own build options.

PLAIN20.12.4 what is really happening inside#

1. Nearly all general compression works by finding repeats and referring back to them instead of writing them again.
2. The reference says “go back this many bytes and copy this many”. That is the LZ77 idea, published by Abraham Lempel and Jacob Ziv in 1977.
3. How far back the tool may look is the window. A bigger window finds more repeats and needs more memory.
4. After that, the remaining symbols are re-coded so that common ones use fewer bits. That is Huffman coding, or the newer arithmetic and range coders.
5. Solid compression matters because of the window. If every file is compressed separately, the window is reset at each file boundary and cross-file repetition is invisible.
6. In a directory of four thousand C header files that all begin with a similar licence comment, that repetition is enormous.
7. This is why the same data, packed two ways, can differ in size by more than a factor of two.

TECHNICAL20.12.5 the engineer’s version#

1. We measured solid versus per-file directly, on the same 4,394 files.

2. Compare rows one and two. Identical Deflate algorithm, identical level. The solid version is 38 percent smaller purely because the window spans file boundaries.
3. Compare rows three and four. Identical LZMA2 algorithm, identical level. Solid mode is 47 percent smaller.
4. The cost of solid mode is random access. Extracting the last file from a solid archive requires decompressing everything before it. ZIP can seek directly.
5. History and specifications:

6. tar means tape archive. Its record size of 512 bytes and its habit of padding to 10,240-byte blocks come directly from nine-track magnetic tape drives. A modern .tar file is still padded to a multiple of 10,240 bytes, which is why a tar of one small file is 10,240 bytes.
7. Window sizes: Deflate uses 32 KiB, fixed by RFC 1951. bzip2 uses blocks of up to 900 KiB. LZMA defaults to 8 to 64 MiB depending on level. zstd’s window grows with level, up to 128 MiB at level 22 with long mode enabled.
8. bzip2 uses a completely different approach, the Burrows-Wheeler transform published in 1994, which reorders data to group similar bytes before coding. It is slow in both directions and has largely been displaced.
9. zstd’s speed comes from finite state entropy coding, an implementation of asymmetric numeral systems published by Jaroslaw Duda in 2009. It achieves arithmetic-coding compression ratios at table-lookup speeds.
10. A practical rule that holds up: choose zstd for anything you compress once and decompress often, gzip when compatibility with everything matters, and xz only when the transfer saving genuinely outweighs a long build.
11. None of these is encryption. ZIP’s original password scheme, from 1990, is broken and can be attacked with known plaintext. ZIP AES-256 and 7z AES-256 are sound, but 7z still leaves file names visible unless header encryption is switched on with -mhe=on.

WORDS20.12.6 remember these#

1. Archive — many files joined into one — a container preserving names, sizes, permissions and directory structure.
2. Compression — making data smaller — removing redundancy so the original can be reconstructed exactly, in lossless schemes.
3. Solid — compressing everything as one stream — allowing back-references to cross file boundaries, at the cost of random access.
4. Window — how far back the packer may look — the maximum LZ77 match distance, 32 KiB in Deflate, up to 128 MiB in zstd.
5. Ratio — how many times smaller it got — original size divided by compressed size, quoted with the exact corpus and level.
6. Lossless — nothing is thrown away — the decompressed output is byte-identical to the input, unlike JPEG or MP3.

20.13 File metadata beyond the basics#

PLAIN20.13.1 in simple words#

1. Beyond size and dates, a file can carry extra labels that most tools never show you.
2. On Linux and macOS these are called extended attributes: small named pieces of data attached to a file but not part of its contents.
3. macOS uses them heavily. When you download a file, the system attaches a label meaning “this came from the internet”.
4. That label is why macOS asks you whether you are sure the first time you open a downloaded application.
5. Windows has a stranger version. A file on NTFS can have several separate contents, each with its own name.
6. The main one is what you see. The extra ones are invisible in Explorer, do not change the reported file size, and were abused for years to hide code.
7. Photographs carry a third kind: a block of camera information inside the image file itself.
8. That block can include the exact time, the camera model, the settings, and often the precise place the photo was taken.
9. Sharing a photo can therefore share your home address without you saying a word.

PLAIN20.13.2 a picture in your head#

1. Think of a book returned to a library.
2. The book has a title and pages. That is the name and contents.
3. A librarian has stuck a small note inside the back cover: “returned late, donated by the Smith family, check for water damage”.
4. The note is not part of the book. Photocopying the book does not copy the note. Lending it might.
5. Those notes are extended attributes. Useful, invisible, and easily lost.
6. Now imagine a book where extra chapters are bound in behind the endpapers, invisible unless you know to look. The spine still says 300 pages.
7. That is a Windows alternate data stream.

Where this comparison breaks: the librarian’s note is written by a human who knows what it means. Extended attributes are written by programs, in binary, with names like com.apple.quarantine, and are silently dropped by many everyday operations: uploading, emailing, extracting from a plain zip, or copying with the wrong flag. The library at least knows the note existed.

PLAIN20.13.3 a worked example#

1. Extended attributes on Linux, all real commands run for this chapter.

$ setfattr -n user.kedbyte -v "chapter20" note.txt
$ getfattr -d note.txt
# file: note.txt
user.kedbyte="chapter20"

$ ls -l note.txt
-rw-r--r-- 1 root root 26 Aug 13 01:57 note.txt

$ cp note.txt note_copy.txt
$ getfattr -d note_copy.txt
(no output)

$ cp --preserve=xattr note.txt note_copy2.txt
$ getfattr -d note_copy2.txt
# file: note_copy2.txt
user.kedbyte="chapter20"

2. The reported size is still 26 bytes. The attribute is stored outside the data.
3. A plain copy lost it silently. No warning, no error. Only the explicit flag preserved it.
4. Now the photo case. We took a real JPEG and added camera and location tags.

$ exiftool -Make="ACME" -Model="Phone X" \
    -DateTimeOriginal="2026:03:14 09:41:00" \
    -GPSLatitude=18.5204 -GPSLatitudeRef=N \
    -GPSLongitude=73.8567 -GPSLongitudeRef=E photo.jpg

$ exiftool -G1 photo.jpg
[IFD0]      Make               : ACME
[IFD0]      Camera Model Name  : Phone X
[ExifIFD]   Date/Time Original : 2026:03:14 09:41:00
[Composite] GPS Position       : 18 deg 31' 13.44" N,
                                 73 deg 51' 24.12" E

5. That is a location accurate to a few metres, a timestamp to the second, and the device, all inside an ordinary picture.
6. Removing it is one command, and it shrank the file by 2,311 bytes:

$ ls -l photo.jpg
-rw------- 1 root root 8520 photo.jpg
$ exiftool -all= photo.jpg
$ ls -l photo.jpg
-rw------- 1 root root 6209 photo.jpg
$ exiftool -Make -Model -GPSLatitude photo.jpg
(nothing listed)

PLAIN20.13.4 what is really happening inside#

1. Extended attributes are stored by the file system next to the inode, or in a separate block if they grow too large.
2. They are name-value pairs. The name has a prefix declaring a namespace: user. for anything, security. for SELinux labels, system. for access control lists, trusted. for privileged data.
3. macOS uses the same mechanism with Apple-defined names. Two matter most.
4. com.apple.quarantine is set by browsers and mail clients on every downloaded file, and holds flags, a timestamp and the downloading application.
5. com.apple.ResourceFork is the modern home of the old resource fork, a second data stream that classic Macintosh files had from 1984.
6. Windows does it differently. On NTFS, every file is a set of named attributes, and the contents are just the unnamed $DATA attribute.
7. Adding a second named $DATA attribute gives the file a second, hidden set of contents. The syntax is a colon: report.txt:hidden.
8. EXIF is not an operating system feature at all. It is a block of data placed inside the image file, in a JPEG marker segment, using the TIFF tag structure.
9. That is why EXIF survives copying, emailing and zipping, whereas extended attributes usually do not. It is part of the file’s own bytes.

TECHNICAL20.13.5 the engineer’s version#

1. Linux extended attribute namespaces and their rules:

2. On ext4 the total size of all extended attributes for one file is limited to one file system block, normally 4096 bytes. XFS allows more. This is why attributes are used for labels, not for storage.
3. macOS quarantine values look like 0083;5f2b1c40;Safari;UUID, giving flags, a hexadecimal timestamp, the agent name and an event identifier. Gatekeeper reads this before first launch and removes it after approval. xattr -d com.apple.quarantine app clears it manually.
4. NTFS alternate data streams have existed since NTFS shipped with Windows NT 3.1 in 1993, originally to support Macintosh resource forks through Services for Macintosh.
5. The reported file size is the length of the unnamed $DATA stream only. dir shows no sign of the others. dir /r, added in Windows Vista, does. Get-Item -Stream * in PowerShell lists them.
6. Windows itself uses one: Zone.Identifier, the mark of the web, written by browsers and by Outlook. Office reads it and opens the document in Protected View. Removing it is what “Unblock” does in the file properties dialog.
7. Malware families including Backdoor.Rustock in 2006 and later commodity loaders stored payloads in alternate data streams to evade scanners that read only the default stream. Every serious scanner now enumerates streams. Streams are lost the moment a file is copied to FAT32, exFAT, a network share without support, or into a plain ZIP.
8. EXIF was first published by JEIDA in 1995 and is maintained today by CIPA and JEITA. It stores TIFF-format image file directories inside a JPEG APP1 marker segment beginning with the bytes Exif\0\0.
9. Commonly present tags, taken from the real file inspected here: Make, Model, DateTimeOriginal, ExposureTime, FNumber, ISO, Orientation, LensModel, plus a full GPS sub-directory with latitude, longitude, altitude and a UTC timestamp.
10. The Orientation tag is the reason a photo appears rotated in one program and upright in another: the pixels are unrotated and the tag says how to display them.
11. Most large social platforms strip EXIF on upload. Most chat applications strip it when sending a photo as an image and keep it when sending as a file. Cloud storage links and direct email attachments keep it. Treat any file you send as carrying its metadata unless you removed it yourself.
12. Tools: getfattr, setfattr, attr on Linux; xattr -l and mdls on macOS; Get-Item -Stream, dir /r and streams.exe on Windows; exiftool and exiv2 for image metadata.

WORDS20.13.6 remember these#

1. Extended attribute — a small named label attached to a file — a namespaced name-value pair stored outside the file’s data.
2. Quarantine flag — the macOS mark on downloaded files — the com.apple.quarantine extended attribute read by Gatekeeper.
3. Resource fork — the classic Macintosh second stream — historically an HFS fork, now the com.apple.ResourceFork extended attribute.
4. Alternate data stream — a hidden second set of contents on NTFS — an additional named $DATA attribute addressed as file:stream.
5. Mark of the web — the Windows record of where a file came from — the Zone.Identifier stream, driving Protected View and SmartScreen.
6. EXIF — camera information stored inside a photo — TIFF image file directories in the JPEG APP1 segment, including an optional GPS directory.

20.14 Reading a file you do not understand#

PLAIN20.14.1 in simple words#

1. Sooner or later you are handed a file with no extension, or a wrong one, and asked what it is.
2. There is a reliable order of steps, and it takes about a minute.
3. Step one: ask the type-guessing tool. It is right most of the time and costs nothing.
4. Step two: look at the first sixteen bytes yourself. Compare them with the magic number table.
5. Step three: pull out the readable text. Format names, version strings, file paths and error messages usually appear in plain sight.
6. Step four: look at the very end. Many formats keep their index there.
7. Step five: if the file looks like nothing, check whether it is compressed or encrypted, because both look like random noise.
8. Step six: scan the whole file for magic numbers, not just the start, because files are often nested inside other files.
9. Then unwrap one layer and start again from step one.

PLAIN20.14.2 a picture in your head#

1. Think of a parcel with no address label.
2. You shake it. Something rattles. That is the type-guessing tool: fast, rough, usually right.
3. You cut off one corner and peek. That is the hex dump of the first bytes.
4. You read the customs sticker for words you recognize. That is strings.
5. You find another wrapped parcel inside, and you start again. Nested formats are the norm, not the exception.

Where this comparison breaks: shaking a parcel gives you a hint about the whole thing at once. A file gives you nothing unless you look in the right place. A 1 GB file whose first bytes are unrecognized may be a perfectly ordinary format that simply begins with a length field, and no amount of staring at the start will tell you. Real identification means checking several places: the start, the end, and a scan across the middle.

PLAIN20.14.3 a worked example#

1. We were handed a file called mystery.dat and asked what it was. Here is the whole session.

$ ls -l mystery.dat
-rw-r--r-- 1 root root 259 mystery.dat

$ file mystery.dat
mystery.dat: gzip compressed data, was "inner.tar",
             last modified Thu Aug 13 02:02:54 2026,
             max compression, from Unix,
             original size modulo 2^32 10240

$ xxd -l 16 mystery.dat
00000000: 1f8b 0808 ce25 7d6a 0203 696e 6e65 722e  .....%}j..inner.

2. 1f 8b is the gzip magic. 08 is the compression method, Deflate. The fourth byte 08 is a flag meaning “an original filename follows”.
3. That is why the readable text inner. appears at offset 10. The gzip header literally stores the original name.
4. So it is one layer of gzip around something called inner.tar. Unwrap it.

$ gzip -dc mystery.dat > mystery.out
$ file mystery.out
mystery.out: POSIX tar archive (GNU)

$ tar tvf mystery.out
-rw-r--r-- root/root  69 2026-08-13 01:55 tiny.png
-rw-r--r-- root/root  26 2026-08-13 01:57 note.txt

$ tar xf mystery.out -C out && file out/*
out/note.txt: ASCII text
out/tiny.png: PNG image data, 1 x 1, 8-bit/color RGB

5. Three layers, three commands, one minute. Nothing was guessed.
6. Note that file told us the original name and the uncompressed size without decompressing anything, because gzip stores both in its header and trailer.

PLAIN20.14.4 what is really happening inside#

1. Scanning for embedded formats is worth doing by hand, because tools like binwalk are not always installed.
2. The method is simple: search the whole file for every known magic byte sequence and report the offsets.
3. We wrote a twenty-line script that does exactly this and ran it on a file made by gluing a Linux executable and a ZIP archive together.

scanning combo.bin (16516 bytes)
  offset       0  ELF object
  offset   15960  ZIP local file header
  offset   16052  ZIP local file header
  offset   16130  ZIP local file header
  offset   16260  ZIP central directory
  offset   16338  ZIP central directory
  offset   16416  ZIP central directory
  offset   16494  ZIP end of central dir

4. The structure is now completely clear without opening the file in an editor: an executable of 15,960 bytes, then a three-entry ZIP archive.
5. That layout is exactly how self-extracting archives and single-file applications are built, and it is what to look for when a program seems suspiciously large.
6. False positives happen. Two-byte signatures like MZ appear by chance roughly once every 65,536 bytes of random data. Always confirm a hit by checking a second field, not just the magic.

TECHNICAL20.14.5 the engineer’s version#

1. The identification toolkit, and what each answers:

2. strings by default prints runs of four or more printable characters. Use -n 8 to cut noise and -e l or -e b for UTF-16 text, which is where Windows binaries hide most of their readable content.
3. Real strings output from the Linux binary we built tells you almost everything about it:

$ strings -n 6 hello
/lib64/ld-linux-x86-64.so.2
__libc_start_main
libc.so.6
GLIBC_2.2.5
GLIBC_2.34
hello, world
GCC: (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
hello.c

4. From eight lines we learned the interpreter, the library, the minimum glibc version, the program’s output, the exact compiler build, and the source file name. This is why release builds are stripped.
5. Practical checks for the “it looks like random noise” case. Compute the byte entropy. Compressed data typically measures 7.9 to 8.0 bits per byte; encrypted data measures the same. Distinguish them by structure: compressed formats have headers, encrypted blobs usually do not.
6. Check the length. AES in a block mode produces output that is a multiple of 16 bytes, often plus a fixed header. That is a strong hint.
7. If the file is a container you half-recognize, list rather than extract. unzip -l, tar tvf, 7z l, ar t, dpkg -I and rpm -qip all read metadata without writing anything to disk.
8. For executables, objdump -x, readelf -a, nm -D, ldd and otool -L on macOS answer what it links against. For unknown binary blobs inside firmware, binwalk -e automates the scan-and-extract loop described above.
9. Never run an unknown executable to find out what it is. Read it statically first, and if you must run it, do so in a disposable virtual machine with no network.
10. When nothing matches, the remaining evidence is structure: look for repeating record lengths, plausible 32-bit little-endian sizes near the start, and offsets that point at other offsets. Most private formats are a header, a count, and an array.

WORDS20.14.6 remember these#

1. Hex dump — the raw bytes shown as pairs of hex digits — an offset-addressed listing produced by xxd, od -t x1 or a hex editor.
2. strings — the readable text pulled out of a binary — runs of printable characters above a minimum length, in a chosen encoding.
3. Carving — pulling embedded files out of a bigger one — locating known signatures at arbitrary offsets and extracting the ranges between them.
4. Entropy — how random the bytes look — Shannon entropy in bits per byte, near 8.0 for compressed or encrypted data.
5. Nested format — a file inside a file inside a file — layered containers, each requiring its own identification pass.
6. False positive — a magic match that is a coincidence — an accidental byte sequence, more likely with short signatures.

20.98 Common wrong ideas#

1. Wrong: the extension tells the computer what a file is. Right: the extension is part of the name. Windows chooses to trust it, Linux tools read the bytes instead, and the file itself is unchanged either way.
2. Wrong: renaming a file converts it. Right: renaming changes a directory entry. Every byte of contents stays exactly the same, as two identical SHA-256 sums in 20.2.3 demonstrated.
3. Wrong: .exe files are a special kind of thing the computer understands natively. Right: an .exe is an ordinary file whose bytes happen to be machine instructions plus a header. The loader does all the work.
4. Wrong: a text file and a binary file are different at the storage level. Right: both are byte sequences. “Text” describes how the bytes happen to render under an encoding, nothing more.
5. Wrong: an empty line ending is just one invisible character everywhere. Right: it is one byte on Linux and macOS, two on Windows, and a different single byte on classic Mac OS, and wc -l reported zero lines for a file full of text because of it.
6. Wrong: a DOCX is a Microsoft binary format. Right: it is a ZIP archive of XML files, openable with any unzip tool, as shown in 20.5.3.
7. Wrong: bigger compression level always means slower decompression. Right: for zstd, decompression took 0.06 seconds at both level 3 and level 19. Level affects the packer’s search effort, not the unpacker’s work.
8. Wrong: a ZIP is read from the front. Right: correct readers start at the end of central directory record and jump backwards. That is why you can prepend a program to an archive and both still work.
9. Wrong: stripping a binary makes it faster. Right: stripping removes the symbol table, which is never loaded at run time. The program is smaller on disk and identically fast, only harder to debug.
10. Wrong: deleting metadata from a photo before sharing is paranoid. Right: a single command revealed a location accurate to metres, a timestamp and the device model from an ordinary JPEG.

20.99 Chapter summary in 20 lines#

1. A file is a named sequence of bytes plus a small metadata record. Nothing in it declares what the bytes mean.
2. Metadata lives in an inode or MFT record: size, permissions, owner, link count and four separate timestamps. The name lives in the directory, not the file.
3. File extensions are a convention inherited from CP/M in 1974, not a rule.
4. Windows decides types from the extension through the registry, macOS from Uniform Type Identifiers introduced in 2005, Linux tools from content sniffing with libmagic.
5. Hiding known extensions by default made invoice.pdf.exe display as invoice.pdf, and the ILOVEYOU worm of May 2000 used exactly that.
6. Most formats begin with a magic number: 89 PNG for PNG, PK for ZIP, 7f ELF for Linux binaries, MZ for Windows binaries.
7. Java class files and macOS universal binaries share the magic CAFEBABE, a real collision resolved by checking the next field.
8. “Text” means the bytes render as readable characters. Line endings are LF on Unix, CRLF on Windows and network protocols, CR on classic Mac OS, all descended from teleprinter mechanics.
9. A UTF-8 byte order mark is three bytes, EF BB BF, permitted but not recommended, and it breaks shebangs, shell scripts and strict JSON parsers.
10. ZIP stores local headers, then the data, then a central directory, then an end record. Readers start at the end, which is why appending and prepending both work.
11. DOCX, XLSX, PPTX, ODT, JAR, APK, EPUB and XPI are all ZIP archives with an agreed internal layout.
12. PDF is a body of numbered objects plus a cross-reference table of byte offsets and a trailer, read from the end, supporting incremental update.
13. An executable is machine code plus a header describing segments, permissions and one entry point. PE on Windows, ELF on Linux, Mach-O on macOS.
14. Every PE still begins with an MZ header and a DOS stub printing “This program cannot be run in DOS mode”, because offset 0x3C inside it holds the pointer to the real header.
15. ELF carries two tables over the same bytes: program headers for the loader and section headers for the linker. strip deletes the second and the program runs identically.
16. Running a program is thirty steps: resolve, permission-check, execve, match the format, build a fresh address space, map segments, build the stack with arguments and the auxiliary vector, load ld.so, resolve libraries and relocations, then _start, then __libc_start_main, then main.
17. A .sh or .py file runs because of two bytes, #!, handled by binfmt_script, which re-execs the named interpreter with your file as an argument. Without them, execve returns ENOEXEC.
18. Installers copy files, register types, create shortcuts, write settings and record uninstall data. Package managers do the same with an exact file list, which is why removal is clean.
19. tar archives without compressing and gzip compresses without archiving; combining them gives solid compression, which measured 38 percent smaller than the same algorithm applied per file.
20. Files carry more than you see: extended attributes, macOS quarantine flags, NTFS alternate data streams, and EXIF blocks that can hold your exact location. file, xxd, strings and a signature scan will identify almost anything in about a minute.