The Engineer's Reference

52.0 How to use this chapter#

1. Every other chapter in this book teaches. This one does not teach. It answers.
2. It is the chapter you open when you are already in the middle of something and you need one number, one flag, or one next step.
3. Nothing here is new. Every fact appeared earlier in the book, in context, with the explanation attached. Here it is stripped down to the lookup.
4. The chapter is arranged so you can find things without reading. The section numbers below are the map.
5. Sections 52.1 to 52.3 are numbers: latency, powers of two, units, bases, character codes. Use these when you are estimating.
6. Section 52.4 is the networking reference: layers, addresses, ports, status codes, record types, states. Use this when you are reading a capture or a log.
7. Section 52.5 is the networking command cheat sheet. Every command has two extra columns that most cheat sheets leave out: what a pass proves, and what a pass does not prove. That distinction is the whole diagnostic method of this book.
8. Section 52.6 is a decision tree for “I cannot reach a site”. It is written as a branching procedure. Follow the numbers.
9. Sections 52.7 and 52.8 are the same two things for git: a full command cheat sheet, then a decision tree for a failed git operation.
10. Sections 52.9 to 52.16 are the rest of the desk reference: shell, file magic numbers, complexity, cloud, security, AI, regular expressions, editors.
11. Section 52.17 is what to do when none of the above worked.
12. Two conventions run through the chapter. First, macOS syntax is given first, because that is what the reader of this book is using, with the Linux difference noted on the next line. Second, anything that is a convention rather than a standard is labelled as such.
13. Anything marked “approximate” is an order of magnitude, not a measurement. Order of magnitude is usually what you need. A number that is right to within a factor of two will stop you making the wrong decision.
14. One warning before you start. A reference table tells you what a thing is. It does not tell you whether the thing is your problem. Keep the habit of asking what an observation rules out, not just what it looks like.

52.1 Numbers every engineer should know#

52.1.1 Why this section exists#

1. Most engineering decisions are decided by a factor of a thousand, not by a factor of two.
2. If you know that memory is roughly one hundred times slower than cache, and a network hop across the planet is roughly a million times slower than memory, you will make the right architectural call without measuring.
3. The list below descends from fastest to slowest. Learn the shape, not the digits.
4. The original list of this kind is credited to Jeff Dean of Google, who presented “Numbers Everyone Should Know” in a 2009 keynote. Peter Norvig published a similar table earlier, in his 2001 essay “Teach Yourself Programming in Ten Years”.
5. The figures below are updated for typical 2025 to 2026 hardware. They are approximate.

52.1.2 The latency ladder, real figures#

1. The single most useful line in that table is the second-from-last group: light in glass fibre travels at about 200,000 kilometres per second, which is roughly two thirds of the speed of light in vacuum.
2. That gives a rule you can do in your head: about 1 millisecond of round trip per 100 kilometres of fibre.
3. Check it. Mumbai to London is about 7,200 km in a straight line and about 10,000 km of real cable route. Ten thousand divided by one hundred is 100 milliseconds. Measured round trips on that path are 110 to 130 ms.
4. This rule is physics. No amount of money buys past it. It is why content delivery networks exist: you cannot make the light faster, so you move the copy closer.

52.1.3 The latency ladder, scaled to human time#

1. Multiply every figure by one billion. One nanosecond becomes one second.
2. Now the ladder becomes something a person can feel.

3. Read the last two lines again. When the reader’s curl waited fifteen seconds and gave up, the processor had, in its own sense of time, waited several human centuries for an answer that never came.
4. That is why a timeout is such strong evidence. Nothing on a working network takes that long. Silence for fifteen seconds is not slowness. It is absence.

52.1.4 Powers of two, 2^0 to 2^64#

1. Every size in computing is a power of two or a multiple of one.
2. Learn 2^10 = 1024, 2^20 = about one million, 2^30 = about one billion. The rest follows.

3. Three lines in that table cause more real bugs than the rest combined.
4. 2^31 minus 1 is 2147483647, the largest signed 32-bit integer. This is the Year 2038 problem: a 32-bit Unix timestamp counting seconds from 1 January 1970 overflows at 03:14:07 UTC on 19 January 2038.
5. 2^53 is 9007199254740992. A double-precision floating point number, the only number type JavaScript had until BigInt was standardized in ES2020, can represent every integer up to 2^53 exactly and not one above it. Git commit hashes and large database IDs sent as JSON numbers are silently corrupted here.
6. 2^32 is 4294967296, the total number of IPv4 addresses. That is fewer than the number of humans alive. This is why NAT exists.

52.1.5 Decimal versus binary prefixes#

1. There are two systems and they disagree by a growing amount.
2. Decimal prefixes are powers of a thousand. Binary prefixes are powers of
   1024. The binary ones were standardized by the International Electrotechnical Commission in IEC 60027-2, amendment 2, in January 1999, and now live in ISO/IEC 80000-13:2008.

3. Who uses which, in practice, is a convention and not a standard.
4. Disk and SSD makers use decimal. A “1 TB” drive holds 10^12 bytes.
5. Operating systems differ. Windows reports that same drive as 931 GB, because it divides by 2^30 but writes the label “GB”. macOS since version 10.6, released in 2009, uses decimal, so it agrees with the box and says 1 TB. Linux tools vary; df -h uses 1024 by default, df -H uses 1000.
6. RAM is always binary. A “16 GB” memory module is 16 GiB, because memory is addressed by binary address lines and cannot be anything else.
7. Network speeds are always decimal, and always in bits. “1 Gbit/s” is 1,000,000,000 bits per second.
8. In November 2022 the 27th General Conference on Weights and Measures added four new SI prefixes: ronna for 10^27, quetta for 10^30, ronto for 10^-27 and quecto for 10^-30. There are no matching binary prefixes.

52.1.6 Common bandwidth figures#

1. Note the unit change halfway down. Network and bus speeds are quoted in bits per second. Memory and PCIe totals are usually quoted in bytes per second. Mixing them up gives you an eight-fold error.

52.1.7 Typical throughput of real systems#

1. These are order-of-magnitude figures for a single well-configured node in 2025 to 2026. Your numbers will differ. The shape will not.

2. One more figure that catches people out. A single TCP connection is limited by its window divided by the round-trip time. With a 64 KiB window and a 100 ms round trip, that is 64 KiB divided by 0.1 seconds, which is about 655 kB/s, which is about 5.2 Mbit/s.
3. That is the bandwidth-delay product limit. It is why a single download from a distant server can crawl on a gigabit line, and why parallel connections or a larger window fix it. Modern systems scale the window automatically, up to 1 GiB, using the window scale option of RFC 7323.

52.2 Units and conversions#

52.2.1 Bit and byte#

1. A bit is one binary digit. It is either 0 or 1.
2. A byte is eight bits. This has been near-universal since the IBM System/360 in 1964. Older machines used 6, 7, 9 and 36-bit units.
3. The correct formal word for exactly eight bits is octet. Networking standards use it, because “byte” once meant other sizes. RFC documents say octet.
4. Lowercase b means bit. Uppercase B means byte. This is a convention, but a very strongly held one.
5. A nibble is four bits, which is exactly one hexadecimal digit.
6. A word has no fixed size. It means the natural unit of the processor: 16 bits on an 8086, 32 on an i386, 64 on modern x86-64 and ARM64.

52.2.2 The full prefix table#

1. Note that kilo is lowercase k in SI. Uppercase K is not an SI prefix; it is the symbol for kelvin. Everyone writes KB anyway. That is a convention that contradicts the standard.
2. The micro symbol is properly the Greek letter mu. In plain text everyone writes u. This book writes us for microseconds.

52.2.3 MB/s versus Mb/s#

1. To convert bits per second to bytes per second, divide by 8.
2. To convert bytes per second to bits per second, multiply by 8.
3. Some people add ten per cent for framing overhead when talking about Ethernet. That is a rough allowance, not a standard.

4. Worked example. Your broadband plan says 300 Mbit/s. Divide by 8 to get 37.5 MB/s. A 4 GB file, at best, takes 4000 divided by 37.5, which is about 107 seconds. If your download manager shows 5 MB/s, you are getting about 40 Mbit/s, which is roughly one seventh of the plan.

52.2.4 Hertz to time period#

1. Frequency and period are reciprocals. Period equals one divided by frequency.

2. Wavelength equals the speed of light divided by frequency. At 2.4 GHz that is 300,000,000 divided by 2,400,000,000, which is 12.5 centimetres. That is why a 2.4 GHz antenna is a few centimetres long.

52.2.5 dBm to milliwatts#

1. dBm is power measured in decibels relative to one milliwatt. It is a logarithmic scale, so it turns multiplication into addition.
2. The formula is: power in dBm equals 10 times the base-10 logarithm of the power in milliwatts.
3. Two shortcuts do almost all the work. Adding 3 dB doubles the power. Adding 10 dB multiplies it by ten.

4. Signal strength in Wi-Fi is always a negative dBm number, because received power is always far below one milliwatt. Closer to zero is stronger. Minus 50 is much better than minus 80.
5. A drop of 3 dBm means you are receiving half the power. A drop of 10 dBm means one tenth.

52.2.6 Temperature#

1. To convert Celsius to Fahrenheit: multiply by 9, divide by 5, add 32.
2. To convert Fahrenheit to Celsius: subtract 32, multiply by 5, divide by 9.
3. To convert Celsius to Kelvin: add 273.15.

4. Silicon does not fail at 90 degrees Celsius. It slows itself down on purpose, which is called thermal throttling, to avoid getting hotter. A laptop that suddenly gets slow after ten minutes of heavy work is almost always throttling.

52.2.7 Storage sizes and what fits#

1. Two facts to keep the scale honest. The English Wikipedia text dump, without images and without history, is roughly 25 GB uncompressed and about 22 GB compressed as of 2025. The complete works of Shakespeare in plain text are about 5.5 MB.

52.3 Number bases and encoding#

52.3.1 Binary, decimal, hexadecimal and octal, 0 to 32#

1. Hexadecimal is base 16. It uses 0 to 9 then A to F.
2. One hex digit is exactly four bits. Two hex digits are exactly one byte. That is the entire reason hex exists in computing.
3. Octal is base 8, three bits per digit. It survives in Unix file permissions and in C escape sequences.

52.3.2 The powers of two in hexadecimal#

1. In hex, every power of two is a 1, 2, 4 or 8 followed by zeros. This makes sizes instantly recognizable in a hex dump.

2. When you see 0xFFFFFFFF in a log, that is 4294967295, which is the largest unsigned 32-bit value, which usually means “all bits set” or “not found”.
3. When you see 0xDEADBEEF or 0xCAFEBABE, that is a human-chosen marker value, put there so it stands out in memory. 0xCAFEBABE is also the real first four bytes of a Java class file, chosen by James Gosling’s team in 1991 for exactly that reason.

52.3.3 The printable ASCII table#

1. ASCII was standardized in 1963 by the American Standards Association and revised in 1967 and 1986. It defines 128 codes, 0 to 127.
2. Codes 0 to 31 and 127 are control codes. Codes 32 to 126 are printable.

Dec  Hx Ch  Dec  Hx Ch  Dec  Hx Ch  Dec  Hx Ch
 32  20 SP   56  38 8    80  50 P   104  68 h
 33  21 !    57  39 9    81  51 Q   105  69 i
 34  22 "    58  3A :    82  52 R   106  6A j
 35  23 #    59  3B ;    83  53 S   107  6B k
 36  24 $    60  3C <    84  54 T   108  6C l
 37  25 %    61  3D =    85  55 U   109  6D m
 38  26 &    62  3E >    86  56 V   110  6E n
 39  27 '    63  3F ?    87  57 W   111  6F o
 40  28 (    64  40 @    88  58 X   112  70 p
 41  29 )    65  41 A    89  59 Y   113  71 q
 42  2A *    66  42 B    90  5A Z   114  72 r
 43  2B +    67  43 C    91  5B [   115  73 s
 44  2C ,    68  44 D    92  5C \   116  74 t
 45  2D -    69  45 E    93  5D ]   117  75 u
 46  2E .    70  46 F    94  5E ^   118  76 v
 47  2F /    71  47 G    95  5F _   119  77 w
 48  30 0    72  48 H    96  60 `   120  78 x
 49  31 1    73  49 I    97  61 a   121  79 y
 50  32 2    74  4A J    98  62 b   122  7A z
 51  33 3    75  4B K    99  63 c   123  7B {
 52  34 4    76  4C L   100  64 d   124  7C |
 53  35 5    77  4D M   101  65 e   125  7D }
 54  36 6    78  4E N   102  66 f   126  7E ~
 55  37 7    79  4F O   103  67 g

3. Four relationships in that table are worth memorizing, because they let you do character arithmetic in your head.
4. The digit 0 is 48, which is hex 0x30. So the numeric value of a digit character is the character minus 48.
5. Uppercase A is 65, hex 0x41. Lowercase a is 97, hex 0x61.
6. The gap between an uppercase letter and its lowercase form is exactly 32, which is a single bit: bit 5, value 0x20. Flipping that one bit changes case. That is not an accident; the 1963 committee arranged it deliberately.
7. Space is 32, hex 0x20. Delete is 127, hex 0x7F.

52.3.4 Control codes and escape sequences#

1. The line ending difference is the single most common cause of a file looking corrupted when moved between systems.
2. Unix and macOS since version 10 end a line with LF alone, which is 0x0A.
3. Windows ends a line with CR then LF, which is 0x0D 0x0A.
4. Classic Mac OS, up to version 9 in 2001, used CR alone, 0x0D.
5. In a hex dump, 0D 0A at the end of each line means the file came from Windows. In git, the setting core.autocrlf controls whether git converts between the two.

52.3.5 UTF-8 byte-length rules#

1. UTF-8 was designed by Ken Thompson and Rob Pike in September 1992, famously sketched on a placemat in a diner in New Jersey. It is defined today by RFC 3629, from 2003.
2. It encodes every Unicode code point in one to four bytes, and it is backward compatible with ASCII: any ASCII file is already valid UTF-8.

3. Every byte after the first in a multi-byte sequence has the pattern 10xxxxxx. That is the self-synchronizing property: from any byte you can find the start of the character by scanning backwards past bytes beginning with 10.
4. A byte beginning with 0 is a lone ASCII character. A byte beginning with 11 starts a multi-byte character. A byte beginning with 10 is a continuation.
5. Practical consequences you will meet.
6. English text costs about 1 byte per character. Latin letters with accents, Greek, Cyrillic, Hebrew and Arabic cost 2. Devanagari, Chinese, Japanese and Korean cost 3. Emoji and rarer historic scripts cost 4.
7. So the same sentence stored in Hindi takes roughly three times the bytes of the English version, before you even count that Hindi may need more characters.
8. len() in Python 3 counts code points, not bytes. strlen() in C counts bytes. .length in JavaScript counts UTF-16 code units, so an emoji outside the basic plane counts as 2.
9. A byte order mark, U+FEFF, encoded as the bytes EF BB BF, sometimes appears at the start of a UTF-8 file written by Windows tools. It is legal but discouraged, and it breaks shell scripts, because the shell sees those three bytes before the #! line.

52.4 The networking reference#

52.4.1 OSI and TCP/IP side by side#

1. The OSI model has seven layers. It was published as ISO/IEC 7498-1, with the first version in 1984.
2. The TCP/IP model has four layers. It was written down in RFC 1122 in October 1989, by which time the protocols had already been running for years.
3. OSI is the vocabulary everyone uses. TCP/IP is the thing that actually runs. Both statements are true at the same time.

4. Three honest notes about that table.
5. TLS does not fit cleanly. It sits above TCP and below HTTP, which is layer 6-ish, but it has its own handshake and session resumption, which is layer 5-ish. People argue about this. Nobody wins.
6. ARP is called layer 2 by most people and layer 2.5 by pedants, because it carries layer 3 addresses inside layer 2 frames.
7. QUIC, standardized in RFC 9000 in May 2021, runs on top of UDP but does transport work: streams, ordering, congestion control, encryption. In OSI terms it is layer 4 built on top of layer 4.
8. Where the reader’s own failure sat: DNS worked, so layer 7 name lookup was fine. The TCP handshake never completed, so the failure was at layer 4 or below, on the path.

52.4.2 The RFC 1918 private ranges#

1. RFC 1918, published February 1996, reserves three blocks that anyone may use inside a private network and that no router on the public internet will forward.

2. The middle block is the one people get wrong. 172.16.0.0/12 covers 172.16.x.x through 172.31.x.x, not 172.16 through 172.16.
3. That is exactly why the reader’s traceroute looked odd. Hops showing 172.31.0.17, 172.26.22.235, 172.26.14.75 and 172.16.18.33 are all private addresses inside the same /12, used by the reader’s ISP for its own core network.
4. Seeing private addresses in a traceroute is normal and proves nothing about a fault. It tells you the ISP numbers its own core privately, which is ordinary practice, especially with carrier-grade NAT.

52.4.3 Reserved IPv4 ranges with their RFCs#

1. The full list lives in the IANA IPv4 Special-Purpose Address Registry. The rows above are the ones you meet.
2. 169.254.x.x on your own machine means DHCP failed. The machine gave itself a link-local address because no server answered. If you see it, fix DHCP before you look at anything else.
3. 100.64.0.0/10 is the range your ISP uses when it puts many customers behind one public address. If your router’s outside address is in this range, you are behind carrier-grade NAT and you cannot accept inbound connections.
4. Use 192.0.2.0/24, 198.51.100.0/24 and 203.0.113.0/24 in documentation and examples. They exist for that. Never invent an example address that belongs to somebody.

52.4.4 CIDR from /8 to /32#

1. The prefix length is how many leading bits are the network part. The rest is the host part.
2. Total addresses is 2 to the power of (32 minus the prefix). Usable hosts is that minus two, because the all-zeros address is the network itself and the all-ones address is the broadcast.

3. Two special cases. A /31 has no network or broadcast address; RFC 3021, from December 2000, allows both addresses to be hosts, which is how router-to-router links are numbered. A /32 is one single address, used for loopbacks and for host routes.
4. Classless Inter-Domain Routing replaced the old class A, B and C scheme. It was defined in RFC 1518 and RFC 1519 in September 1993, and was a direct response to the routing table growing faster than router memory.

52.4.5 The ports worth memorizing#

1. Port numbers are 16 bits, so 0 to 65535. RFC 6335 splits them into three ranges: 0 to 1023 system, 1024 to 49151 user or registered, and 49152 to 65535 dynamic or ephemeral.
2. On Linux the outgoing ephemeral range is usually 32768 to 60999. On macOS it is 49152 to 65535. Both are settings, not standards.

3. Note the two rows that mattered in the reader’s session. Git over SSH uses port 22. Git over HTTPS uses port 443. In the reader’s session one transport worked while the other did not, which is only possible because they are different ports to different endpoints.

52.4.6 ICMP types worth knowing#

1. Type 3 carries a code that says exactly what was unreachable.

2. Code 4 is the one that causes mysterious hangs. It is how path MTU discovery works. If a firewall blocks it, large packets vanish and small ones get through, and a page loads its HTML but never its images.
3. Code 13 is a firewall openly telling you it refused. That is a polite refusal, and it is evidence. Silence is not.
4. Traceroute works entirely on type 11 code 0. Each probe goes out with a time-to-live one higher than the last, and each router that decrements it to zero sends back type 11.
5. Under IPv6, ICMPv6 is defined by RFC 4443 and the numbers change. Echo Request is 128, Echo Reply is 129, Packet Too Big is 2. ICMPv6 is not optional: blocking it breaks IPv6 entirely, because Neighbor Discovery rides on types 133 to 137.
6. The reader’s trace showed * * * from hop 13 to hop 20. That means no type 11 replies came back from those hops. It does not mean the packets died there. Many routers rate-limit or refuse to generate ICMP.

52.4.7 HTTP methods#

1. Safe means the method is not supposed to change anything on the server.
2. Idempotent means doing it twice has the same effect as doing it once.
3. Both properties are promises made by the specification, currently RFC 9110, published June 2022. A server can break them. A well-behaved one does not.

4. The practical consequence: a client may retry a GET, HEAD, PUT or DELETE automatically after a network failure. It must not retry a POST automatically, because it cannot know whether the first one landed.
5. That is exactly the reasoning behind the git failure in section 52.8. A disconnect while reading the response tells you nothing about whether the write was applied.

52.4.8 HTTP status codes#

1. The first digit gives the family. One hundred series is informational, two hundred success, three hundred redirection, four hundred client error, five hundred server error.
2. Codes marked with a star are ones you should know without looking up.

3. Four distinctions that people get wrong every day.
4. 401 versus 403. A 401 means “I do not know who you are; send credentials”. A 403 means “I know who you are and you may not”. The reader’s missing workflow scope produced a 403-class refusal, not a 401.
5. 502 versus 503 versus 504. A 502 means the gateway got a broken answer from the thing behind it. A 503 means the service is deliberately unavailable, often overloaded or in maintenance. A 504 means the thing behind it never answered in time.
6. 301 versus 308, and 302 versus 307. The older pair, 301 and 302, historically allowed clients to change a POST into a GET when following. The newer pair, 307 and 308, forbid changing the method. Use 308 and 307 when you care.
7. 418 is real and is a joke. It comes from RFC 2324, the Hyper Text Coffee Pot Control Protocol, published on 1 April 1998. RFC 7168 extended it to teapots in 2014. It is not part of HTTP.
8. 451 is named for the novel Fahrenheit 451 and was standardized in RFC 7725 in February 2016. It means content is blocked for legal reasons.

52.4.9 Common DNS record types#

1. A CNAME cannot exist at the zone apex, that is at example.com itself, because the apex must also hold SOA and NS records and a CNAME forbids any other record at the same name. Providers work around this with a non-standard record they call ALIAS or ANAME. That is a vendor convention, not a standard.
2. SVCB and HTTPS records were standardized in RFC 9460 in November 2023. The HTTPS record can tell a browser to use HTTP/3, and can carry the encrypted client hello parameters, before any connection is made.
3. TTL is a number of seconds attached to every record. It is a request, not a guarantee. Resolvers may cache longer or shorter.

52.4.10 TCP flags and states#

1. The TCP header has a flags field. The current specification is RFC 9293, from August 2022, which replaced RFC 793 from 1981.

2. The three-way handshake is SYN, then SYN plus ACK, then ACK. The four-way close is FIN, ACK, FIN, ACK.
3. An RST in reply to a SYN means “nothing is listening on that port”. That is a refusal, and it is information. It arrives in milliseconds.
4. No reply at all to a SYN means the packet was dropped somewhere and nobody told you. That is what the reader saw. It costs the full timeout.

5. Two states are diagnostic on their own.
6. Many sockets stuck in CLOSE_WAIT means your own application is not calling close. That is an application bug, not a network problem.
7. Many sockets in TIME_WAIT is normal on a busy client. The specification says wait twice the maximum segment lifetime, which by the letter of the standard is four minutes. Linux uses a fixed sixty seconds and does not let you change it without recompiling.
8. A connection that sits in SYN_SENT and never leaves it is the exact signature of the reader’s fault: the SYN went out, nothing came back.

52.5 Networking command cheat sheet#

52.5.1 How to read this section#

1. Every entry below gives the macOS form first, because that is the machine in the running example, then the Linux difference on the next line.
2. Then two lines that most cheat sheets omit.
3. Proves is the set of things that must be true for the command to have succeeded. If it passed, all of these are true.
4. Does not prove is the trap. These are things people assume a pass means and that it does not mean.
5. Getting this second line right is the difference between diagnosing a problem and guessing at it.

52.5.2 The commands at a glance#

52.5.3 ping#

1. macOS: ping -c 5 github.com
2. Linux: same, plus -w 5 sets a total deadline in seconds.
3. Useful flags, macOS: -c count, -i interval, -s size, -t timeout in seconds for the whole run, -D set the do-not-fragment bit.
4. Careful: on Linux -t means time-to-live, not timeout. Same letter, different meaning. Linux uses -M do for do-not-fragment.
5. Proves: your interface is up, your route exists, the target address is reachable by ICMP, and something at that address answers ICMP.
6. Does not prove: that any TCP port is open, that the web server is running, that DNS is correct, or that the service you want works. Ping tests ICMP and only ICMP.
7. A failed ping proves even less. Very many hosts and networks drop ICMP on purpose. Cloudflare, AWS load balancers and corporate firewalls commonly do. A ping that fails while curl succeeds is completely normal.
8. Path MTU test, macOS: ping -D -s 1472 github.com. If 1472 fails and 1400 works, something on the path has a smaller MTU. Add 28 bytes of headers to get the real figure: 1472 plus 28 is 1500.

52.5.4 traceroute#

1. macOS: traceroute -n github.com
2. Linux: traceroute -n github.com, and tracepath is a simpler alternative that needs no privileges.
3. Useful flags: -n no reverse lookups so it runs fast, -m 30 maximum hops, -q 1 one probe per hop, -w 2 wait two seconds, -I use ICMP probes instead of UDP.
4. macOS default probes are UDP to high port numbers. Linux default is the same. Windows tracert uses ICMP by default. This matters, because firewalls filter the three differently.
5. TCP mode is the most useful on a modern network. Linux: traceroute -T -p 443 github.com. macOS: traceroute -P TCP -p 443 github.com.
6. Proves: which routers along the path are willing to answer, in order, up to the last one that replied.
7. Does not prove: that the path is symmetric, that the reply came back the same way it went, that a hop with no reply is broken, or that the last replying hop is where the trouble is.
8. The reader’s trace stopped replying after hop 12, ae106-0.rwa04.pnq20.ntwk.msn.net (104.44.20.42). That does not mean hop 12 dropped the traffic. It means hops beyond it did not send back ICMP time-exceeded messages, which is extremely common.
9. Hops showing more than one address, as hops 5, 6, 8, 9 and 11 did in the reader’s trace, mean the traffic is being spread over parallel links. That is load balancing working correctly, not a fault.

52.5.5 mtr#

1. mtr combines ping and traceroute and keeps running, so it shows loss per hop over time.
2. macOS: not installed by default. Install with brew install mtr, then run with sudo mtr github.com because it needs raw sockets.
3. Linux: mtr github.com, often already installed.
4. Report mode, for pasting into a ticket: mtr -rwc 100 github.com. That means report, wide output, one hundred cycles.
5. TCP mode: mtr -T -P 443 github.com.
6. Proves: sustained loss or latency at and beyond a specific hop.
7. Does not prove: that loss shown at a middle hop is real. Routers deliberately rate-limit the ICMP replies they generate for themselves while forwarding everything else perfectly. Only loss that continues to the final hop and stays there is real loss.
8. That single rule saves more wasted hours than any other line in this chapter. Read the last row, not the middle rows.

52.5.6 dig, host and nslookup#

1. dig is the precise one. Use it.
2. macOS: dig github.com. On recent macOS versions Apple has removed the BIND lookup tools from the base install. If dig is missing, get it with brew install bind.
3. Linux: dig github.com, from the dnsutils or bind-utils package.

dig +short github.com                 # just the answer
dig +short A github.com               # only IPv4
dig +short AAAA github.com            # only IPv6
dig @1.1.1.1 github.com               # ask a specific resolver
dig @8.8.8.8 github.com               # ask a different one
dig +trace github.com                 # walk from the root down
dig -x 20.207.73.82                   # reverse lookup
dig +noall +answer github.com         # answer section only
dig NS github.com                     # nameservers
dig MX github.com                     # mail servers
dig TXT github.com                    # text records
dig +tcp github.com                   # force TCP not UDP
dig +dnssec github.com                # ask for signatures

4. Proves: that a resolver answered, what it answered, and how long the answer may be cached.
5. Does not prove: that the address is reachable, that it is the right address for you, or that your operating system will use it. dig talks to the resolver directly and bypasses the system cache.
6. That last point matters on macOS, which has its own resolver daemon. To see what the system will actually do, use dscacheutil -q host -a name github.com, and to see which resolvers are configured use scutil --dns.
7. host github.com gives a one-line answer. nslookup github.com 1.1.1.1 works everywhere including Windows. Both are fine for a quick check and worse than dig for anything else.
8. In the reader’s session, name resolution worked: github.com resolved to 20.207.73.82 through the Cloudflare resolver at 1.1.1.1. That single fact eliminated DNS as the cause.

52.5.7 curl#

1. The single most useful network command on any machine.
2. macOS and Linux syntax are the same. macOS has shipped curl since 10.4.

curl -v https://github.com            # show the whole conversation
curl -I https://github.com            # headers only, HEAD request
curl -sS -o /dev/null -w '%{http_code}\n' https://github.com
curl -L https://example.com           # follow redirects
curl --max-time 10 https://github.com # give up after 10 seconds
curl --connect-timeout 3 https://x.io # give up connecting after 3
curl -4 https://github.com            # force IPv4
curl -6 https://github.com            # force IPv6
curl --resolve github.com:443:20.207.73.82 https://github.com
curl -H 'Accept: application/json' https://api.github.com
curl -X POST -d 'a=1' https://example.com/f
curl -k https://self-signed.example   # skip cert check
curl --http1.1 https://github.com     # force HTTP/1.1
curl -x http://proxy:3128 https://x.io # go via a proxy

3. The timing breakdown is the hidden gem. It tells you which stage was slow.

curl -sS -o /dev/null -w \
'dns %{time_namelookup} conn %{time_connect} tls
%{time_appconnect} first %{time_starttransfer} total
%{time_total}\n' https://github.com

4. Proves, when it returns a status code: DNS resolved, TCP connected, TLS completed if the URL was HTTPS, the request was sent, and a server answered.
5. Does not prove: that the answer is correct, that the site works in a browser, or that the certificate is trusted if you passed -k.
6. A hang with no output after Trying <address>:443... proves the TCP handshake never completed. That is a layer 4 or below problem and nothing above it can be at fault yet.
7. That is exactly what the reader saw. curl -v https://github.com printed Trying 20.207.73.82:443... and then produced nothing for fifteen seconds. No RST, no ICMP unreachable, no TLS error. Silence.
8. Silence is a specific finding. A server that is down sends RST. A firewall that refuses politely sends ICMP type 3 code 13. Nothing at all means a silent drop somewhere on the path.

52.5.8 wget#

1. macOS: not installed by default. brew install wget.
2. Linux: usually present.

wget https://example.com/file.tar.gz   # save with its own name
wget -O out.bin https://example.com/f  # save under a chosen name
wget -O - https://example.com          # print to standard output
wget --spider https://example.com    # exists? do not save
wget -c https://example.com/big.iso    # continue a partial download
wget -T 10 -t 3 https://example.com    # 10s timeout, 3 tries

3. Proves and does not prove: the same as curl.
4. Use wget for downloading files and mirroring. Use curl for inspecting a request. That split is a convention, not a rule.

52.5.9 nc, the netcat tool#

1. The cleanest way to test one TCP port and nothing else.
2. macOS: nc -vz github.com 443. Apple ships its own build of the OpenBSD version.
3. Linux: nc -vz -w 3 github.com 443 with the OpenBSD build.

nc -vz github.com 443            # is port 443 open
nc -vz -w 3 github.com 22        # with a 3 second timeout
nc -vzu 1.1.1.1 53               # UDP, far less reliable
nc -l 8080                       # listen on 8080, a fake server
nc -vz github.com 20-25          # a small range

4. Proves, on success: the full TCP three-way handshake completed to that address on that port.
5. Does not prove: that the service behind the port speaks the protocol you expect, that TLS will succeed, or that your credentials are valid.
6. This is the correct tool for separating “the network is blocked” from “the application is refusing me”. If nc -vz succeeds and curl fails, the problem is above layer 4.
7. UDP testing with -u is close to useless on its own, because a silent UDP port and an open UDP port look identical. Only the application reply proves anything.

52.5.10 telnet#

1. Removed from macOS since High Sierra, version 10.13, in 2017. Install with brew install telnet or, better, use nc instead.
2. telnet github.com 443 then type text. Historically used to speak HTTP by hand.
3. Proves: a TCP connection opened, and you can type raw bytes into it.
4. Does not prove: anything about encryption. Telnet is plaintext. Never use the telnet protocol itself, on port 23, for real work. It sends passwords in the clear. SSH replaced it in 1995.

52.5.11 nmap#

1. Not installed by default anywhere. brew install nmap on macOS, apt install nmap on Debian and Ubuntu.

nmap -Pn -p 443 github.com         # one port, skip host discovery
nmap -p 22,80,443 192.168.0.1      # a few ports
nmap -p- 192.168.0.1               # all 65535 ports, slow
nmap -sV -p 443 example.com        # try to identify the service
nmap -sn 192.168.0.0/24            # who is on my network
nmap --top-ports 100 192.168.0.1   # the common ones only
sudo nmap -sS -p 443 example.com   # half-open scan, needs root

2. Proves: which of the scanned ports answered, and how.
3. Does not prove: that a filtered result means blocked by the target. It may be blocked anywhere along the path, including by your own network.
4. One legal and social rule, and it is not optional. Scan only machines you own or have written permission to scan. Port scanning other people’s networks is, in many countries including India under the Information Technology Act 2000, potentially an offence, and at minimum it will get your connection blocked.

52.5.12 netstat, ss and lsof#

1. These look at your own machine, not the network. None of them need connectivity.
2. macOS: netstat -an -p tcp shows all TCP sockets. netstat -rn shows the routing table. netstat -i shows interface counters. netstat -s shows protocol statistics.
3. Linux: netstat is deprecated in favour of ss, from the iproute2 package.

ss -tulpn                     # Linux: listening TCP+UDP, pids
ss -tn state established      # Linux: established TCP only
ss -s                         # Linux: summary counts
netstat -an -p tcp            # macOS: all TCP sockets
netstat -rn                   # both: routing table
lsof -i -P -n                 # both: sockets with process names
lsof -nP -iTCP:443            # both: who is using port 443
lsof -nP -iTCP -sTCP:LISTEN   # both: what is listening

4. Proves: exactly which local process holds which port, and what state each connection is in.
5. Does not prove: anything about the remote end. A socket shown as ESTABLISHED locally can be dead, if the peer vanished without sending FIN or RST. Only traffic proves liveness.
6. On macOS, lsof needs sudo to see other users’ processes.

52.5.13 arp#

1. macOS and Linux: arp -a lists the address resolution cache, which maps IPv4 addresses to hardware addresses on your local network segment.
2. Linux modern form: ip neigh show.
3. sudo arp -d 192.168.0.1 deletes one entry, which forces a fresh lookup.
4. Proves: which machines your computer has recently talked to on the same local segment, and their MAC addresses.
5. Does not prove: anything beyond your own network. ARP does not cross a router. If the gateway is in the table with a MAC address, your link to the router is working.
6. arp -a showing 192.168.0.1 with a valid MAC address would have been the first thing to check in the reader’s case, and it would have passed. The local network was fine.

52.5.14 ifconfig, ip and route#

1. macOS: ifconfig and ifconfig en0. Apple has not adopted the ip command.
2. Linux: ip addr show, or the short form ip -br a.

ifconfig -a                    # macOS: every interface
ifconfig en0                   # macOS: just Wi-Fi
ipconfig getifaddr en0         # macOS: just the IPv4 address
ipconfig getpacket en0         # macOS: the whole DHCP lease
netstat -rn                    # macOS: routing table
route -n get default           # macOS: which route wins
ip -br a                       # Linux: brief addresses
ip link show                   # Linux: layer 2 state
ip route show                  # Linux: routing table
ip route get 20.207.73.82      # Linux: which route wins

3. On macOS, en0 is usually Wi-Fi on a laptop and Ethernet on a desktop. lo0 is loopback. utun0 and upwards are tunnel interfaces, created by VPNs, by iCloud Private Relay, and by some corporate agents.
4. The reader’s machine showed several utun interfaces. That is worth noticing but does not prove a VPN was routing the traffic. Check with route -n get default and see which interface the default route uses.
5. Proves: your own addresses, your own routes, your own link state.
6. Does not prove: that the route leads anywhere.

52.5.15 tcpdump#

1. The ground truth. Everything else is an interpretation; this is the packets.
2. macOS: sudo tcpdump -i en0 -n host 20.207.73.82
3. Linux: same, with the interface usually named eth0, ens3 or wlan0.

sudo tcpdump -i en0 -n host 20.207.73.82
sudo tcpdump -i en0 -nn port 443
sudo tcpdump -i any -n icmp
sudo tcpdump -i en0 -n -c 20 tcp port 443
sudo tcpdump -i en0 -w capture.pcap        # save for Wireshark
sudo tcpdump -i en0 -n 'tcp[tcpflags] & tcp-syn != 0'
sudo tcpdump -i en0 -n -A port 80          # show payload as text

4. Flags that matter: -n no name lookups, -nn also no port names, -i any all interfaces on Linux, -c N stop after N packets, -s 0 full packet, -w file write a capture file, -A print payload as ASCII, -X print hex and ASCII.
5. Proves: exactly what left your machine and exactly what came back.
6. Does not prove: what happened in the middle. You see your end of the wire and nothing else.
7. In the reader’s case, tcpdump would have shown repeated SYN packets going out to 20.207.73.82 on port 443, with retransmissions at roughly one, three and seven seconds, and nothing at all coming back. That is the signature of a silent drop, and it is different from every other failure.

52.5.16 openssl s_client#

1. Tests the TLS handshake by itself, with nothing above it.

openssl s_client -connect github.com:443 -servername github.com
openssl s_client -connect github.com:443 -showcerts
openssl s_client -connect github.com:443 -tls1_2
openssl s_client -connect github.com:443 -tls1_3
echo | openssl s_client -connect github.com:443 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

2. -servername sets Server Name Indication, which almost every modern host requires. Without it you may get the wrong certificate or a failure.
3. Proves: TCP connected, a TLS handshake completed, which version and cipher were agreed, and which certificate chain was presented.
4. Does not prove: that the certificate is trusted by your operating system. The openssl command uses its own trust store, which on macOS is usually not the system keychain. A certificate that openssl rejects may work fine in Safari, and the reverse.
5. Check expiry dates with the last command above. Certificate expiry is one of the most common causes of a site that worked yesterday and not today.
6. Since 2024 the industry has been shortening maximum certificate lifetimes. The CA/Browser Forum voted in April 2025 to reduce the maximum from 398 days down in stages, reaching 47 days by March 2029. Expiry checks are getting more important, not less.

52.5.17 whois#

1. whois github.com looks up who registered a domain name.
2. whois 20.207.73.82 looks up who owns an address block.
3. Proves: what a registry says. Registry data is often stale, and since the European General Data Protection Regulation took effect in May 2018, most personal contact details in domain records are hidden.
4. Does not prove: who is operating the machine today.
5. In the reader’s session, whois on 20.207.73.82 would show a Microsoft range. That is consistent with GitHub being owned by Microsoft, which acquired it in 2018 for 7.5 billion dollars in stock, and fronting traffic through Microsoft’s network edge.
6. The traceroute agreed: from hop 7 onward the names all ended in ntwk.msn.net, which is Microsoft’s backbone naming.

52.5.18 macOS-only tools#

1. scutil --dns shows every resolver the system will use, in order, with the search domains and the interface each is bound to. This is the truth about macOS DNS; /etc/resolv.conf on macOS is a compatibility file and can be misleading.
2. scutil --nwi shows network information: which interfaces are usable, and whether IPv4 and IPv6 are available on each.
3. scutil --proxy shows the proxy configuration.
4. networksetup -listallnetworkservices lists the named services such as Wi-Fi and Ethernet.
5. networksetup -getinfo "Wi-Fi" shows address, mask, router and MAC.
6. networksetup -getdnsservers "Wi-Fi" shows configured resolvers.
7. sudo networksetup -setdnsservers "Wi-Fi" 1.1.1.1 8.8.8.8 sets them.
8. sudo networksetup -setdnsservers "Wi-Fi" Empty puts it back to DHCP.
9. sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder clears the DNS cache. Both halves are needed on modern macOS.
10. networkQuality runs Apple’s built-in throughput and responsiveness test. It has shipped since macOS 12 Monterey, released in 2021.
11. ipconfig getpacket en0 prints the entire DHCP lease including the offered DNS servers, which is how you tell what the router wanted you to use.
12. The reader was using 1.1.1.1, Cloudflare’s public resolver, rather than the router. scutil --dns is what proves that.

52.5.19 Worked example: the reader’s own five commands#

1. Here is the exact sequence that settled the reader’s case, with what each step eliminated.

Step 1  ping 192.168.0.1
        -> replies.  Local network and router: fine.

Step 2  dig +short github.com
        -> 20.207.73.82.  DNS: fine.

Step 3  curl -v --max-time 15 https://github.com
        -> "Trying 20.207.73.82:443..." then nothing.
        -> TCP handshake never completed.

Step 4  traceroute -n github.com
        -> reaches Microsoft's Pune edge, then * * *.
        -> path exists at least that far.

Step 5  same URL on mobile data, same phone
        -> loads instantly.
        -> the server is up; the destination is fine.

2. What that combination proves: the failure is specific to the path between this home connection and that destination, at the transport layer, and it is a silent drop rather than a refusal.
3. What it does not prove: who is dropping the packets, or why. The evidence does not name a party, and you should not either.

52.6 Decision tree: I cannot reach a site#

52.6.1 The shape of the procedure#

1. Work upward through the layers. Never start in the middle.
2. Each step either passes, which eliminates everything below it, or fails, which tells you exactly where to stop looking.
3. Do not skip a step because you are sure it is fine. The reader’s case looked like a DNS problem and was not.

      START: "I cannot reach a site"
                  |
       1. Is any other site reachable?
          no -> your machine or link. Go to A.
          yes v
       2. Does the name resolve?
          no -> DNS. Go to B.
          yes v
       3. Does TCP connect on the port?
          no -> path or firewall. Go to C.
          yes v
       4. Does TLS complete?
          no -> certificate or version. Go to D.
          yes v
       5. Does HTTP return a status code?
          no -> the app is hung. Go to E.
          yes v
       6. Is the status what you expected?
          no -> application layer. Go to F.
          yes -> it works. The problem is elsewhere.
                 Go to G.

52.6.2 Step 1: is anything reachable at all#

1. Run ping -c 3 192.168.0.1, using your own gateway address.
2. Find your gateway with route -n get default on macOS or ip route show on Linux.
3. If the gateway does not answer, stop. The problem is your machine, your cable, or your Wi-Fi. Check ifconfig en0 for an address.
4. If your address begins 169.254., DHCP failed. Nothing else matters until that is fixed. Renew the lease or restart the interface.
5. If the gateway answers, run ping -c 3 1.1.1.1.
6. If a public address answers but names do not resolve, jump straight to step 2. That combination is the classic DNS-only failure.
7. If the gateway answers and 1.1.1.1 does not, the problem is between your router and the internet. Check the router’s own status page and the physical line.
8. Branch A conclusion: local. Fix the interface, the lease, the cable or the router before going further.

52.6.3 Step 2: does the name resolve#

1. Run dig +short github.com.
2. If you get an address, DNS works. Note the address and move on.
3. If you get nothing, ask a different resolver: dig @1.1.1.1 +short github.com and dig @8.8.8.8 +short github.com.
4. If a public resolver answers and your configured one does not, your resolver is the problem. On macOS, check with scutil --dns and change it with networksetup -setdnsservers.
5. If no resolver answers for that one name but others resolve, the domain itself may be broken. Check with dig +trace github.com, which walks from the root servers down and shows where the chain stops.
6. If the answer looks wrong, for example a private address for a public site, suspect a captive portal, a filtering resolver, or a modified hosts file. Check /etc/hosts on macOS and Linux.
7. Clear the cache and try again: on macOS, sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder.
8. Branch B conclusion: DNS. Nothing above this layer can work until a name turns into an address.

52.6.4 Step 3: does TCP connect#

1. Run nc -vz github.com 443.
2. Success means the three-way handshake completed. Move to step 4.
3. Now read the failure carefully, because the three failure modes mean completely different things.
4. Connection refused, immediately. You received a TCP RST. Something is there and it said no. The host is up; the port is closed. Suspect a service that is not running, or the wrong port.
5. No route to host or host unreachable, quickly. You received an ICMP type 3. A router along the way openly told you it could not deliver. Suspect routing or a firewall that refuses politely.
6. Timeout, after many seconds, with no message at all. Nothing came back. This is a silent drop. It is the hardest case and it is the reader’s case.
7. Confirm which of the three you have with curl -v --max-time 15 and, if you can, with sudo tcpdump -i en0 -n host <address> and port 443 in another window.
8. If it is a timeout, test whether it is that one destination or all destinations: try nc -vz 1.1.1.1 443 and nc -vz google.com 443.
9. Test whether it is that one port: try nc -vz github.com 22 and nc -vz github.com 80.
10. Test whether it is that one network path: use a different network. Mobile data on a phone, with tethering or with the same URL typed into the phone, is the cleanest test available to a home user.
11. If it works on a different network and not on yours, the destination is fine and the path is not.
12. Branch C conclusion: transport or path. See section 52.6.8 for what to do next, because this is where the reader ended up.

52.6.5 Step 4: does TLS complete#

1. Run openssl s_client -connect github.com:443 -servername github.com.
2. Success prints a certificate chain and a line saying which protocol version and cipher were negotiated.
3. certificate verify failed means the chain did not validate. Look at the reason code printed on the same line.
4. certificate has expired means exactly that. Check the dates with openssl x509 -noout -dates.
5. unable to get local issuer certificate usually means an intermediate certificate is missing from the server’s chain, or your trust store is old.
6. wrong version number or no protocols available means a version mismatch. Old servers that only speak TLS 1.0 are refused by modern clients. Test with -tls1_2 and -tls1_3 explicitly.
7. A hostname mismatch means the certificate does not cover the name you asked for. Check the subject alternative names with openssl x509 -noout -text and look for the SAN extension.
8. If TLS fails only in your browser but succeeds with openssl, suspect a corporate inspection proxy that is re-signing traffic, or a VPN.
9. Branch D conclusion: TLS. The network is fine; the trust is not.

52.6.6 Step 5 and 6: does HTTP answer, and correctly#

1. Run curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' against the URL.
2. No status code and a long time means the connection opened and the server never replied. That is an application hang, not a network fault. Branch E.
3. A 502, 503 or 504 means you reached a gateway and the thing behind it is broken or overloaded. That is the operator’s problem, not yours. Branch F.
4. A 401 or 403 means you reached the right service and it refused you. Check your credentials, token, or scopes. Branch F.
5. A 404 means you reached the right service and asked for the wrong path.
6. A 429 means you are being rate limited. Read the Retry-After header.
7. A 200 with wrong content means caching, a stale CDN copy, or the wrong virtual host. Try curl -H 'Cache-Control: no-cache' and try the origin directly with --resolve.
8. Branch F conclusion: application layer. The whole network stack worked.

52.6.7 Everything passes and it still fails#

1. This happens, and it has a small number of causes. Work through them in this order.
2. It works from the command line but not in the browser. Suspect a browser extension, a proxy setting inside the browser, or a service worker holding a cached broken response. Test in a private window with extensions off.
3. It works over IPv4 and not IPv6, or the reverse. Force each with curl -4 and curl -6. Broken IPv6 with working IPv4 is common and produces long pauses because the browser tries IPv6 first.
4. The reader’s machine reported IPv6: (none), which removes this branch from the reader’s own case entirely.
5. It works for small requests and fails for large ones. That is an MTU problem. Test with ping -D -s 1472 and step the size down. If 1400 works and 1472 does not, something on the path has a smaller MTU and is not sending the ICMP message that would let path MTU discovery work.
6. It works sometimes. Run mtr -rwc 200 and look at the loss on the final row only. Intermittent means loss, congestion, or a load-balanced path where one of several parallel links is broken.
7. That last case is worth naming: if a hop shows several addresses, as hops 5, 6, 8, 9 and 11 did in the reader’s trace, then some flows take one link and some take another. A single bad link out of four gives you a fault that appears for one in four connections and looks random.
8. It works for other people on the same network. Compare scutil --dns, route -n get default and the proxy settings between the two machines.
9. It stopped working at a specific time. Ask what changed then: a certificate expiry, a DNS TTL running out, a scheduled firewall rule, a router reboot.
10. It works with a VPN on. That proves the destination and the server are fine and that the direct path is the problem, because the VPN carries your traffic through a different route. It does not prove who is blocking.

52.6.8 Exactly what to report to an ISP#

1. Support staff cannot act on “the internet is not working”. They can act on a reproducible measurement. Give them these seven things, in this order.
2. Your own public address and the time, including the time zone. Get the address with curl -4 ifconfig.me when you have any working path, or from the router’s status page.
3. The destination name and the exact address it resolves to on your machine. For the reader: github.com resolving to 20.207.73.82.
4. The exact symptom, in protocol terms. Not “it is slow”. Say: “TCP SYN to 20.207.73.82 port 443 receives no response of any kind, no RST and no ICMP unreachable, and times out after 15 seconds.”
5. A full traceroute, pasted verbatim, with -n so it is fast and unambiguous. Include the hop where replies stop.
6. The control test that proves it is not the destination. For the reader: “The same URL loads instantly over mobile data on the same phone, at the same moment.”
7. The control test that proves it is not everything. For example: “Other HTTPS sites work normally from the same machine at the same time.”
8. An mtr report over at least 100 cycles, in report mode, so they can see loss over time rather than one snapshot.
9. What you have already ruled out, listed plainly: local network, DNS, the destination server, IPv6.
10. One sentence of what you want: “Please check whether traffic from my connection to 20.207.73.82 on port 443 is being dropped in your network.”
11. Do not accuse anyone. The evidence supports a silent drop somewhere on that path. It does not identify who is doing it, and claiming otherwise makes the ticket easier to dismiss.
12. Keep a copy of every command output with timestamps. If the fault is intermittent, that record is the only thing that will get it taken seriously.

52.7 Git command cheat sheet#

52.7.1 How to read the two extra columns#

1. Net says whether the command needs the network. If it says no, the command works with the cable unplugged. That is most of git.
2. Risk says whether the command can destroy work that is not recoverable from the reflog.
3. Low means safe. Medium means it rewrites history but the old commits stay reachable through git reflog for 90 days by default. High means it can delete uncommitted changes or untracked files, which nothing can recover.

52.7.2 Setup and configuration#

1. The three settings worth doing on a new machine, once:

git config --global user.name "Your Name"
git config --global user.email "you@example.com"
git config --global init.defaultBranch main
git config --global pull.rebase false
git config --global core.editor "vim"

52.7.3 Local work#

1. git restore file and git clean -fd are the two most dangerous commands in daily use. They delete things git has never seen. The reflog cannot help you. Always run git clean -n first.

52.7.4 History and inspection#

1. git reflog is the undo button for almost everything. If a rebase or reset went wrong, find the old SHA in the reflog and git reset --hard to it. Entries expire after 90 days for reachable commits and 30 days for unreachable ones, by default.

52.7.5 Branching#

1. The one fact about rebase that matters. Rebase does not move commits. It creates new commit objects with the same changes and new parents, so every rebased commit gets a new SHA.
2. That is exactly what the reader saw: a local branch rebased onto a moved main came out with a different SHA, because it was a different object.
3. Merge preserves history and adds a commit. Rebase rewrites history and adds none. Neither is correct in general; teams pick one by convention.

52.7.6 Remotes and the network#

1. Only the rows marked yes need the network. Everything in 52.7.3, 52.7.4, 52.7.5 and 52.7.7 works offline, because the whole history is already in .git.
2. That is why the reader could keep committing to branches during a total outage and push them later. Commits are local writes to local files.
3. git ls-remote origin is the only way to learn the true current state of a remote. git rev-parse origin/main reads a cached value from your last fetch.
4. In the reader’s session, origin/main showed 0a95cc8 during the outage. That was the remote-tracking ref, which is a note about the past, not a live fact.
5. Never use --force on a shared branch. Use --force-with-lease, which checks that the remote is still where you last saw it and refuses if somebody else pushed in the meantime.

52.7.7 Undoing#

1. The rule that keeps you safe: revert is for history that other people have. reset is for history only you have.
2. --soft keeps everything staged. --mixed, the default, keeps changes in the working tree but unstaged. --hard deletes them.

52.7.8 Plumbing, for looking inside#

1. The four object types, seen with git cat-file -t: blob is file contents, tree is a directory listing, commit is a snapshot plus metadata, tag is an annotated tag.
2. Git stores content addressed by hash. Git 2.29, released October 2020, added experimental SHA-256 repositories; SHA-1 remains the default in 2026.

52.8 Decision tree: my git operation failed#

52.8.1 The five families#

1. Every git failure belongs to one of five families. Naming the family is 90 per cent of the fix.

   git command failed. Read the FIRST error line.
                 |
   Does it mention a host, port, TLS,
   timeout, disconnect, or reset?
      yes -> TRANSPORT.  See 52.8.3
      no  v
   Does it say Authentication failed,
   Permission denied (publickey),
   or ask for a password repeatedly?
      yes -> AUTHENTICATION.  See 52.8.4
      no  v
   Does it name a scope, a permission,
   or say 403 / not authorized?
      yes -> SCOPE.  See 52.8.5
      no  v
   Does it say rejected, non-fast-forward,
   protected branch, hook declined, or
   pre-receive?
      yes -> SERVER REJECTION.  See 52.8.6
      no  -> LOCAL.  See 52.8.7

52.8.2 First, always: did it actually happen#

1. Before diagnosing anything, find out the current true state.
2. Run git ls-remote origin. That is a live query. It is the only reliable answer.
3. Compare with git rev-parse HEAD locally.
4. Do not trust git rev-parse origin/main or git status after a failure. Those read remote-tracking refs, which are a cached memory of your last successful fetch.
5. This step is not optional. Half of all git panic is caused by acting on a stale local belief about the remote.

52.8.3 Transport failures#

1. Symptoms: Could not resolve host, Failed to connect, Connection timed out, Connection reset by peer, SSL_ERROR_SYSCALL, RPC failed, early EOF, unexpected disconnect while reading sideband packet.
2. Diagnostics, in order:

git ls-remote origin              # does anything work at all
ssh -T git@github.com             # SSH transport, port 22
nc -vz github.com 22              # is port 22 even reachable
nc -vz github.com 443             # is port 443 reachable
curl -sS -o /dev/null -w '%{http_code}\n' \
  https://github.com
GIT_CURL_VERBOSE=1 git fetch      # verbose HTTPS
GIT_TRACE=1 GIT_TRACE_PACKET=1 git fetch

3. Retry: usually safe for fetch, clone and pull, because they only read.
4. Retry for push: safe only after you have re-queried with git ls-remote and know whether the previous attempt landed.
5. Common fixes. If port 22 is blocked but 443 is not, use SSH over 443 by putting this in ~/.ssh/config:

Host github.com
  Hostname ssh.github.com
  Port 443
  User git

6. If large pushes fail but small ones work, raise the buffer: git config --global http.postBuffer 524288000. This is a workaround, not a fix, and it helps only with HTTP transport.
7. If HTTPS fails and SSH works, or the reverse, that is not a git problem at all. It is two different ports to two different endpoints, and one path is broken. The reader saw exactly this in the same session.

52.8.4 Worked branch: sideband packet disconnect#

1. The exact error, seen twice in the reader’s session: send-pack: unexpected disconnect while reading sideband packet.
2. What it means, literally. Git pushed the pack of objects, then waited on the sideband channel for the server’s progress and status messages, and the connection died while it was reading them.
3. Now the important part. The disconnect happened while reading the response. It says nothing about whether the write was applied.
4. Three outcomes are all consistent with that one message. The server never received the pack. The server received it and rejected it. The server received it, applied it, and the acknowledgement was lost on the way back.
5. This is the same problem as a POST over HTTP that times out. The request is not idempotent, so the client cannot safely assume anything.
6. The only correct next step is to ask the server what it now believes:

git ls-remote origin
git ls-remote origin refs/heads/main
git rev-parse HEAD

7. If the remote SHA equals your local SHA, the push landed. Do nothing.
8. If it does not, push again. A repeated push of the same commits is harmless, because git will only send objects the server lacks.
9. In the reader’s case, both times the operation had not been applied. That was established by re-querying, not by reading the error.
10. Retry safety: safe to retry, but only after checking. Never force-push as a response to this error. Force is how a lost acknowledgement turns into lost commits.

52.8.5 Authentication and scope failures#

1. Authentication symptoms: Permission denied (publickey), Authentication failed, repeated password prompts, Invalid username or password.

ssh -vT git@github.com            # verbose SSH handshake
ssh-add -l                        # keys the agent holds
ssh-add ~/.ssh/id_ed25519         # load a key
git config --get remote.origin.url
git credential-osxkeychain erase  # macOS, then type host lines

2. GitHub stopped accepting account passwords for git over HTTPS on 13 August
   2021. If something is prompting you for a password, it wants a personal access token, not your password.
3. Scope symptoms are different and more specific. The push authenticates fine and is then refused with a named reason.
4. Worked branch, from the reader’s session. A push was refused because the token lacked the workflow scope, and the push modified a file under .github/workflows/.
5. That is not a bug. It is a deliberate rule: a token that can push code should not be able to change what runs automatically in CI, because CI has its own secrets and its own privileges.
6. The diagnosis is the error text plus one question: does this push touch a workflow file? Check with git diff --stat origin/main..HEAD and look for .github/workflows/.
7. The fix is to add the workflow scope to the token, or to move the workflow change into a separate push made with a token that has it.
8. Retry safety: retrying with the same token will fail identically, every time. Nothing was applied, because the server refused before applying.
9. This is a clean example of the difference between 401 and 403 thinking. You were identified successfully. You were then found to lack a permission.
10. The general rule for tokens: check three things, in this order. Is it expired. Does it cover this repository. Does it have the scope this specific action needs.

52.8.6 Server-side rejections#

1. Symptoms: ! [rejected], non-fast-forward, Updates were rejected because the remote contains work that you do not have locally, protected branch, pre-receive hook declined, GH006.
2. A non-fast-forward rejection means somebody pushed since your last fetch. The server is protecting their work.

git fetch origin
git log --oneline HEAD..origin/main   # what they added
git log --oneline origin/main..HEAD   # what you added
git rebase origin/main                # or: git merge origin/main
git push

3. A protected-branch rejection means the repository’s rules forbid a direct push. Open a pull request instead. This is a setting, not an error.
4. A pre-receive hook declined means the server ran a script that said no. The reason is usually in the lines just after. Common causes are commit message format rules, file size limits, and secret scanning.
5. GitHub blocks any single file over 100 MiB, and warns above 50 MiB. That limit has been in place since 2015 and is a service policy, not a git limit. Use Git Large File Storage for big files.
6. Retry safety: retrying unchanged always fails again. Nothing was applied. Fix the cause first.

52.8.7 Local failures#

1. Symptoms: not a git repository, pathspec did not match any file, your local changes would be overwritten, You have unmerged paths, index.lock exists, detached HEAD.

git rev-parse --git-dir            # am I in a repo at all
git status                         # the honest current state
git diff --name-only --diff-filter=U   # conflicted files
rm -f .git/index.lock              # only if no git is running
git fsck --full                    # object integrity
git switch -                       # leave detached HEAD safely

2. index.lock exists almost always means a previous git process died or an editor’s git integration is running. Check with ps aux | grep git before deleting the lock file.
3. detached HEAD is not an error. It means HEAD points at a commit rather than a branch. Commits made there are reachable only from the reflog, so create a branch before you go further: git switch -c rescue.
4. Retry safety: always safe. Nothing left your machine.
5. Finally, one distinction that matters and that the reader’s session showed clearly. Git the tool and GitHub the service are different things. During the outage plain git kept working locally, and the GitHub web API did not. An error from gh or from the API is not a git error.

52.8.8 Verifying CI properly#

1. A green tick next to a branch name means “some run for something on this branch passed”. It does not say which commit.
2. The strong check is by run identifier against the exact head SHA:

git rev-parse HEAD
gh run list --branch main --limit 5
gh run view <run-id> --json headSha,status,conclusion

3. Compare the headSha field with your local SHA character by character. If they differ, the tick is about a different commit.
4. Merge state, seen through the API, uses values such as CLEAN, BLOCKED and UNKNOWN. UNKNOWN usually means the server has not finished computing the merge yet; re-query rather than assuming a failure.

52.9 Shell and system cheat sheet#

52.9.1 Files and directories#

52.9.2 Looking inside files#

52.9.3 Finding things#

find . -name '*.log'              # by name
find . -type f -size +100M        # files over 100 MB
find . -mtime -1                  # changed in the last day
find . -name '*.tmp' -delete      # find and delete
find . -type f -exec grep -l x {} +   # search inside
which python3                     # which one runs
type -a python3                   # every match, and aliases
command -v python3                # portable form
locate filename                   # from an index, if built

52.9.4 Processes and resources#

1. Signal 15, SIGTERM, asks a program to shut down cleanly. Signal 9, SIGKILL, cannot be caught or ignored and gives the program no chance to save. Try 15 first, always.

52.9.5 Archives, permissions and transfer#

tar -czf out.tar.gz dir/     # create, gzip
tar -xzf in.tar.gz           # extract
tar -tzf in.tar.gz           # list without extracting
zip -r out.zip dir/          # create a zip
unzip in.zip                 # extract a zip
gzip file                    # replaces file with file.gz
gunzip file.gz               # and back
chmod 644 file               # rw for owner, r for others
chmod 755 script             # add execute for everyone
chmod +x script              # just add execute
chown user:group file        # change ownership
umask                        # default permission mask
rsync -avz src/ host:dst/    # copy, resume, compress
scp file host:/path          # simple copy over SSH
ssh -i key user@host         # connect with a key

1. Permission digits are octal. Read is 4, write is 2, execute is 1. Add them. The three digits are owner, group, others. So 755 is read-write-execute for the owner and read-execute for everyone else.

52.9.6 Keyboard shortcuts in the shell#

1. On macOS Terminal, Alt is the Option key, and you may need to enable “Use Option as Meta key” in the settings before Alt-B and Alt-F work.

52.9.7 The shell expansion order#

1. The shell rewrites your command line in a fixed order before running anything. Knowing the order explains almost every strange quoting bug.

1. brace expansion        {a,b}  ->  a b
2. tilde expansion        ~      ->  /Users/you
3. parameter expansion    $VAR   ->  its value
4. arithmetic expansion   $((2+2)) -> 4
5. command substitution   $(cmd) ->  its output
6. word splitting         on spaces, tabs, newlines
7. pathname expansion     *.txt  ->  matching files
8. quote removal          "x"    ->  x

2. The two consequences that bite people.
3. Word splitting happens after variable expansion. So an unquoted $FILE holding my report.txt becomes two arguments. Quoting it stops that.
4. Pathname expansion happens after variable expansion too, so a variable containing * will expand to filenames unless you quote it.

52.9.8 The safety rules#

1. Quote every variable. Write "$FILE", not $FILE. Write "$@", not $@. This one habit prevents most shell bugs.
2. Start scripts with strict mode.

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

3. -e exits on the first failing command. -u treats an unset variable as an error. -o pipefail makes a pipeline fail if any stage fails, not just the last. Setting IFS stops word splitting on plain spaces.
4. Never pipe an unknown URL into a shell. curl ... | sh hands a remote server the ability to run anything as you. Download first, read it, then run it.
5. There is a further reason beyond trust: a server can serve different bytes to curl piped into a shell than to curl writing a file, because it can see the download stall as the shell executes. Reading the file you downloaded is the only real protection.
6. Test destructive commands first. Use git clean -n, rm -i, rsync --dry-run, and find ... -print before find ... -delete.
7. Never run rm -rf with a variable that might be empty. If $DIR is unset, rm -rf "$DIR"/ becomes rm -rf /. set -u prevents this.
8. Use sudo for one command, not for a session. A root shell left open is a mistake waiting for a typing error.
9. Prefer absolute paths in scripts and cron jobs. The working directory and the PATH in an automated context are not the ones you have interactively.

52.10 File format identification by magic number#

1. A magic number is a fixed byte sequence at a known offset that identifies a file’s format. The extension is a hint; the magic number is the fact.
2. Check with file thing for a guess, or xxd file | head -2 for the bytes.

3. Two entries in that table are the same bytes, and this is a real trap. CA FE BA BE is both a Java class file and an old Mach-O universal binary header. The disambiguation is what follows.
4. 50 4B 03 04 covers a very large family. Modern Office files, Java jar files, EPUB books, Android APK files and OpenDocument files are all zip archives with different contents inside.
5. Two other useful facts. MZ at the start of a Windows executable stands for Mark Zbikowski, one of the original MS-DOS developers, and has been there since 1983. 7F 45 4C 46 is a non-printable byte followed by the letters ELF, deliberately chosen so text tools do not mangle it.

52.11 Complexity and performance reference#

52.11.1 Big-O and what fits in one second#

1. Big-O describes how the work grows as the input grows. It hides constant factors on purpose.
2. The table assumes a rough budget of 100 million simple operations per second, which is a fair figure for interpreted-to-compiled code across the range. Compiled C can do ten times more; Python often ten times less.

3. The practical reading. If n is a million, you need O(n) or O(n log n). If n is ten thousand, O(n^2) is acceptable. If n is twenty, anything works.
4. Constants matter in real code. A cache-friendly O(n^2) can beat a pointer-chasing O(n log n) for small n. Measure before optimizing.

52.11.2 Data structure operation costs#

1. A hash table is O(1) on average and O(n) in the worst case, when every key lands in the same bucket. Modern languages randomize the hash seed at startup to stop attackers forcing that deliberately.
2. A B-tree is the shape used by databases and filesystems because its nodes are sized to a disk block, so one comparison level costs one read.
3. A Bloom filter can say “definitely not present” or “probably present”. It never gives a false negative and cannot delete.

52.11.3 Sorting algorithms#

1. Stable means two records with equal keys keep their original order. That matters when you sort by one column and then another.
2. Timsort was written by Tim Peters in 2002 for Python. Java adopted it for objects in Java 7, released 2011. It finds runs that are already sorted and merges them, which is why it is O(n) on nearly sorted data.
3. Introsort, by David Musser in 1997, is what most C++ standard libraries use for std::sort. It runs quicksort and switches to heapsort if the recursion gets too deep, which removes the O(n^2) worst case.
4. No comparison sort can beat O(n log n) in the worst case. That is a proved lower bound, not an engineering limit. Counting and radix sort beat it only because they do not compare.

52.11.4 The cost of common operations#

1. System calls got more expensive in 2018, when the Meltdown and Spectre mitigations forced page table isolation. Depending on the machine and the mitigation set, a simple system call can cost several times what it did before.
2. The gap between a Python operation and a C operation is roughly fifty to a hundred times. That is why numerical Python work is done inside NumPy, where the loop lives in C.

52.12 Cloud and infrastructure reference#

52.12.1 Service equivalence#

1. Names change. Microsoft renamed Azure Active Directory to Microsoft Entra ID in 2023, and has been folding Synapse into Microsoft Fabric since 2023. Google renamed Cloud Functions to Cloud Run functions in 2024.
2. The equivalences are approximate. Spanner and Aurora are not the same thing; both solve “a relational database bigger than one machine” and solve it differently.

52.12.2 The service model responsibility split#

1. One rule survives every model: you are always responsible for your own data and your own access control. No provider takes that on.
2. The shared responsibility model is the standard phrase. AWS states it as “security of the cloud” being theirs and “security in the cloud” being yours.
3. The most common real breach in cloud is not a provider failure. It is a storage bucket left open to the public, which is entirely on the customer side of the line.

52.12.3 Instance families and what they are for#

1. The letters above are the AWS convention. Azure uses series letters such as D for general purpose, F for compute, E and M for memory, and N for GPU. Google uses names such as N2, C3, M3 and A3.
2. A modern name such as m7g.2xlarge reads as: family m, generation 7, Graviton Arm processor, size 2xlarge. The suffix g for Arm is an AWS convention.
3. Size doubling is a convention too. Each step up in size usually doubles vCPU and memory and doubles the price, so cost per unit of work is roughly flat. That is why splitting across smaller instances rarely saves money by itself.

52.12.4 Cost dimensions to check on any bill#

1. Compute time, per hour or per second, per instance.
2. Storage, per gigabyte-month, separately for each class.
3. Data transfer out to the internet, per gigabyte. This is the item that surprises people. Ingress is usually free; egress is not.
4. Data transfer between availability zones and between regions, per gigabyte, in both directions.
5. Request counts. Object storage charges per PUT and per GET, and a workload with many tiny objects can cost more in requests than in storage.
6. Provisioned throughput and IOPS on disks and databases, whether or not you use them.
7. Load balancer hours plus a per-capacity-unit charge.
8. NAT gateway hours plus a per-gigabyte processing charge. This is a common source of unexplained cost in private networks.
9. Public IPv4 addresses. AWS began charging for every public IPv4 address on 1 February 2024, at 0.005 US dollars per hour, which is about 3.60 dollars a month each.
10. Snapshots and backups, which keep growing and which nobody deletes.
11. Log ingestion and log retention, charged separately.
12. Idle waste: unattached volumes, stopped instances still holding disks, old snapshots, forgotten test environments.
13. Support plan, usually a percentage of the total spend.
14. Commitment discounts. Reserved instances and savings plans trade flexibility for a lower rate. Spot instances are much cheaper and can be taken away with two minutes of notice.

52.13 Security reference#

52.13.1 Hash functions, as of 2026#

1. MD5 was broken by Xiaoyun Wang and colleagues in 2004. SHA-1 was broken in practice on 23 February 2017, when Google and CWI Amsterdam published two different PDF files with the same SHA-1 digest, an attack they named SHAttered. A chosen-prefix collision followed in January 2020.
2. Neither is safe for signatures or integrity against an attacker. Both are still acceptable as non-security checksums, and git still uses SHA-1 with an added collision-detection check.

52.13.2 Ciphers and key exchange, as of 2026#

1. NIST disallowed three-key Triple DES for new encryption after 31 December
   2023. RC4 was prohibited in TLS by RFC 7465 in February 2015.
2. Post-quantum standards arrived on 13 August 2024, when NIST published FIPS 203 for ML-KEM, derived from Kyber, FIPS 204 for ML-DSA, derived from Dilithium, and FIPS 205 for SLH-DSA, derived from SPHINCS+.
3. Deployment is hybrid: browsers and servers have been negotiating X25519 combined with ML-KEM-768 since 2024, so a future quantum computer cannot decrypt traffic captured today. This is established practice now, not a research claim.

52.13.3 Password storage parameters#

1. Never store a password. Store a slow, salted hash of it. These are the OWASP recommended minimums.

2. m is memory in kibibytes, t is the number of passes, p is parallelism. Memory cost is the point: it is what makes custom hardware attacks expensive.
3. bcrypt silently ignores everything past 72 bytes of input. Enforce a maximum length or pre-hash. The work factor is a power of two, so 12 is four times the work of 10.
4. Always use a random salt per password, at least 16 bytes. A pepper, a secret stored outside the database, is optional and adds real value.
5. Plain SHA-256 of a password is not acceptable, with or without a salt. It is far too fast: modern hardware computes billions per second.

52.13.4 TLS configuration essentials#

1. Enable TLS 1.3 and TLS 1.2. Disable everything older. TLS 1.0 and 1.1 were formally deprecated by RFC 8996 in March 2021.
2. TLS 1.3, standardized in RFC 8446 in August 2018, has only five cipher suites and all of them are safe. The three you will see are TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384 and TLS_CHACHA20_POLY1305_SHA256.
3. For TLS 1.2, allow only suites beginning ECDHE_ with GCM or CHACHA20. That gives forward secrecy and authenticated encryption.
4. Turn on OCSP stapling so clients do not have to contact the certificate authority themselves.
5. Send the HTTP Strict Transport Security header with a long lifetime: Strict-Transport-Security: max-age=31536000; includeSubDomains.
6. Disable TLS compression and client-initiated renegotiation. The CRIME attack of 2012 used compression to recover session cookies.
7. Publish a CAA DNS record naming which certificate authorities may issue for your domain.
8. Watch expiry. The CA/Browser Forum agreed in April 2025 to reduce the maximum certificate lifetime in stages, reaching 47 days by March 2029. Automated renewal stops being optional.

52.13.5 The OWASP Top 10, 2025 release#

1. This list replaced the 2021 edition. Two changes are worth knowing. Supply chain failures moved up to third place, reflecting attacks on dependencies and build systems rather than on the application itself. Mishandling of exceptional conditions is new at tenth: error handling that leaks information or fails open.
2. The list is an awareness document, ordered by a mix of measured incidence and survey. It is not a checklist you can pass.

52.13.6 A personal security checklist#

1. Use a password manager and a different password for every site. This one change removes the largest category of real-world account loss.
2. Turn on two-factor authentication everywhere it is offered. Prefer a passkey or a hardware security key, then a time-based code app, and use SMS only if nothing else is available. SMS codes can be stolen by taking over the phone number.
3. Turn on full disk encryption. On macOS that is FileVault, in System Settings under Privacy and Security.
4. Let the operating system and browser update themselves.
5. Use a non-administrator account for daily work where you can.
6. Back up on the 3-2-1 rule: three copies, on two kinds of media, one of them off-site. Test a restore at least once; an untested backup is a guess.
7. Lock the screen when you walk away.
8. Review which applications have access to your accounts. On GitHub, check authorized OAuth applications and personal access tokens, and delete the ones you do not recognize.
9. Give every token the smallest scope that works, and an expiry date. The reader’s workflow scope refusal is the system working correctly.
10. Check your email addresses against public breach databases once a year.
11. Do not install browser extensions you do not need. An extension can read everything you see on every page.
12. Be suspicious of urgency. Almost every successful phishing message manufactures a deadline.

52.14 AI and machine learning reference#

52.14.1 The memory formula#

1. For inference, the base memory is simple: bytes = number of parameters times bytes per parameter.

2. Add roughly 20 per cent on top for the runtime, activations and fragmentation.
3. Then add the key-value cache, which grows with the conversation: KV bytes = 2 times layers times kv-heads times head-dim times sequence length times batch size times bytes per element.
4. The leading 2 is because there is one key tensor and one value tensor.
5. For training with the Adam optimizer in mixed precision, budget roughly 16 to 20 bytes per parameter: 4 for the master weights, 4 for momentum, 4 for variance, 2 for the working copy, 2 for gradients, plus activations.
6. So a 7 billion parameter model needs about 14 GB to run at 16-bit and about 112 GB or more to train the same way. That gap is why fine-tuning methods such as LoRA exist.

52.14.2 Model size against hardware#

1. Practical readings for 2026 hardware. A laptop with 16 GB of unified memory runs a 7B or 8B model comfortably at 4-bit. A 24 GB consumer GPU runs 13B at 8-bit or 32B at 4-bit. A 70B model at 16-bit needs two 80 GB accelerators. A 405B model needs a multi-node cluster.
2. Quantization costs quality, and the cost is not linear. Going from 16-bit to 8-bit is usually almost free. Going to 4-bit is usually acceptable. Going below 4-bit degrades noticeably, though the frontier moves.

52.14.3 Token cost rules of thumb#

1. Non-Latin scripts cost more because the tokenizer was trained mostly on English text, so it has fewer merged pieces for them. This is a measured property of specific tokenizers, not a law.
2. Cost scales with tokens in plus tokens out, and output tokens are usually priced several times higher than input tokens.
3. Attention cost grows with the square of the sequence length in the classical transformer, so doubling the context roughly quadruples that part of the compute.

52.14.4 Common hyperparameters#

1. Temperature 0 means always take the most likely token, which makes output nearly deterministic. Temperature 1 samples from the model’s own distribution. Above about 1.2 the output usually becomes incoherent.
2. Learning rate is the hyperparameter that matters most. Too high and training diverges; too low and it never gets anywhere.

52.14.5 How to read a model announcement#

1. Separate the three kinds of claim before you read anything else.
2. Established fact: parameter count, context length, licence, price per million tokens, whether weights can be downloaded, release date.
3. Active research: benchmark scores, reasoning ability, agentic behaviour, long-context quality. These are measured, but the measurements are contested and move quickly.
4. Marketing claim: “state of the art”, “PhD-level”, “reasoning”, “understands”. These words have no agreed definition.
5. Then ask these ten questions.
6. Which exact benchmark, which version, and which prompting setup. A score with chain-of-thought and 64 samples is not comparable to a single-shot score.
7. Was the benchmark possibly in the training data. Contamination is common and usually undetectable from outside.
8. Is this a base model or an instruction-tuned model. They are not interchangeable.
9. What is the licence. “Open weights” means you can download the numbers. “Open source” properly means you also get the training data and code, which is rare.
10. What is the real context window, and is there an independent measurement of quality at the far end of it. Advertised context and usable context often differ.
11. What is the throughput in tokens per second, and at what batch size.
12. What is the price per million input tokens and per million output tokens.
13. Who ran the evaluation. First-party numbers should be treated as a claim until someone independent repeats them.
14. What is the cut-off date of the training data.
15. What is not mentioned. Announcements omit the benchmarks where the model did badly, and that omission is information.

52.15 Regular expressions#

52.15.1 The syntax table#

1. Flags are set outside the pattern in most languages: i ignore case, m multi-line so ^ and $ match at each line, s so . also matches a newline, g find all matches.
2. Two warnings. Nested quantifiers such as (a+)+ can take exponential time on a failing input, which is a real denial-of-service class called catastrophic backtracking. And you cannot parse HTML with a regular expression; use a parser.

52.15.2 Five patterns worth keeping#

1. Match an IPv4 address, roughly:

\b(?:\d{1,3}\.){3}\d{1,3}\b

2. It finds four groups of one to three digits separated by dots. It will also match 999.999.999.999, so validate the ranges afterwards in code. A regular expression that fully validates IPv4 is long and unreadable, and checking each number is under 256 in code is clearer.
3. Find the start of every log line with a timestamp:

^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}

4. This matches both 2026-08-13 14:05:09 and the ISO 8601 form with a T. Use it with grep -E to pick out real log lines and drop stack-trace continuation lines.
5. Match a quoted string, without running past the closing quote:

"(?:[^"\\]|\\.)*"

6. The trick is [^"\\], which means any character that is not a quote and not a backslash, alternated with \\., which means a backslash followed by anything. That handles escaped quotes inside the string.
7. Find a possible secret in a diff before you commit it:

(?i)(api[_-]?key|secret|token|password)\s*[:=]\s*\S+

8. Run it with git diff --staged | grep -E before every commit. It produces false positives, which is the correct trade-off for this job.
9. Capture the status code and byte count from a common log format line:

"\s(\d{3})\s(\d+|-)$

10. Group 1 is the HTTP status, group 2 is the response size or a dash. Pipe it through awk or sed -E to build a count by status code.

52.16 Keyboard shortcuts and editor basics#

52.16.1 Terminal#

1. The tmux prefix is Ctrl-B by default; Ctrl-A is a very common reconfiguration and a convention rather than a default.

52.16.2 Shell line editing#

1. The shortcuts in section 52.9.6 are Emacs-style bindings, which is the default in bash and zsh.
2. If you prefer vi bindings, run set -o vi. Then Escape puts you in command mode and the vim movement keys work on the command line.
3. Ctrl-R then typing searches history backwards. Press Ctrl-R again to go further back. Ctrl-G cancels.

52.16.3 Enough vim to edit a file and get out#

1. vim has modes. That is the whole thing people find confusing. You start in normal mode, where letters are commands, not text.

2. The five you actually need, in order: press Escape to be sure you are in normal mode. Press i to type. Press Escape again. Type :wq and press Enter to save and leave. Type :q! and press Enter to leave without saving.
3. If a git command drops you into an editor you did not expect, that is vim, because it is the default core.editor on many systems. Change it with git config --global core.editor "nano" if you would rather.
4. nano is the friendly alternative. Ctrl-O writes, Ctrl-X exits, and it prints the shortcuts along the bottom of the screen.

52.17 What to do when you are stuck#

52.17.1 The debugging method#

1. Reproduce it. A bug you cannot trigger on purpose cannot be fixed, only guessed at. Write down the exact steps.
2. Read the actual error. Not the summary. The first line and the deepest line of the stack.
3. Establish what is true. Write down three things you have proved and how you proved them. Write down three things you are assuming.
4. Check the assumptions first. Most bugs live there. The version you think you are running, the file you think you are editing, the environment variable you think is set.
5. Bisect. Cut the problem in half. Does the failure happen with half the input, half the config, half the code. git bisect does this over history automatically.
6. Change one thing at a time. Two changes at once means you learn nothing from the result.
7. Look at the boundary. Most faults sit where two systems meet: your code and a library, your machine and the network, your process and the filesystem.
8. Ask what changed. If it worked yesterday, something moved. A deployment, a certificate, a DNS record, a dependency update, a disk filling up.
9. Explain it out loud to a person or an object. Saying it in order forces the gap into view. This is called rubber-duck debugging, and it works because it makes you state your assumptions.
10. Take a break. After ninety minutes with no progress, stop. Returning with fresh attention beats another hour of the same.
11. Write down the fix and why it worked. The same problem will come back to you or to somebody else.

52.17.2 How to read an error message#

1. Read the first line and the last line before anything in between. The first says what went wrong. The last is often where it went wrong.
2. Find the first line that mentions your own code. Everything above it is usually library internals.
3. Separate the error type from the message from the location. FileNotFoundError is the type. The path is the message. The line number is the location.
4. Look for a number: an exit code, an errno, a status code, a signal number. Those are searchable and precise.
5. Common exit codes: 0 success, 1 general error, 2 misuse of a shell builtin, 126 found but not executable, 127 command not found, 130 interrupted by Ctrl-C, 137 killed by signal 9, often the out-of-memory killer.
6. Common errno values: 2 ENOENT no such file, 13 EACCES permission denied, 32 EPIPE broken pipe, 98 EADDRINUSE address already in use, 110 ETIMEDOUT connection timed out, 111 ECONNREFUSED connection refused.
7. Note what the error does not say. An error that says “connection refused” tells you something answered. An error that says “timed out” tells you nothing answered. Those are different problems.

52.17.3 The minimal reproduction#

1. Copy the failing thing into a new, empty directory or file.
2. Delete half of it. Does it still fail. If yes, delete half of what is left. If no, put it back and delete the other half.
3. Repeat until removing anything makes the failure go away.
4. What remains is the reproduction. It is usually five to twenty lines.
5. Remove your credentials, your company’s names and your real data. Replace them with example.com and the documentation addresses from section 52.4.3.
6. Half the time the bug becomes obvious during this process, and you never need to ask anyone. That is not wasted effort; that is the method working.

52.17.4 Searching effectively#

1. Search the exact error text, in quotes, with your own paths, names and numbers removed.
2. Add the technology and the version: postgres 16, not postgres.
3. Add the year if the technology moves fast, or restrict the results to the last year. Answers about a 2019 version of a fast-moving tool are often wrong now.
4. Search the project’s own issue tracker before the general web. The people who wrote the code discuss it there.
5. Read the official documentation for the exact version you are running. Not the latest, the one you have.
6. When an answer says “just disable the check”, read why the check exists before doing it.
7. Prefer answers that explain the mechanism. An answer that gives you a command with no reason will not help you next time, and may not be safe.

52.17.5 Asking a question that gets answered#

1. Give these seven things, in this order, and you will get an answer.
2. What you are trying to do, in one sentence, at the level of the goal and not the current attempt.
3. What you did, as exact commands or code that someone else could run.
4. What you expected to happen.
5. What actually happened, pasted verbatim, including the full error, as text and not as a picture.
6. Your environment: operating system and version, tool versions, and how it was installed.
7. What you have already tried and what each attempt ruled out. This is the single line that most changes the quality of the answers, because it stops people suggesting things you have done.
8. A minimal reproduction, from section 52.17.3.
9. Two things not to do. Do not ask “does anyone know about X”; just ask the question. And do not describe the problem you invented while trying to solve the real one, without saying what the real one is; that is the most common reason a thread goes nowhere.
10. If you solve it yourself before anyone replies, post the answer. Somebody with the same problem will find it.

52.98 Common wrong ideas#

1. Wrong: a failed ping means the host is down. Right: it means ICMP did not come back. Very many hosts drop ICMP deliberately and serve HTTPS perfectly.
2. Wrong: * * * at the end of a traceroute proves the fault is at that hop. Right: it proves those routers did not send ICMP time-exceeded messages, which is common and often deliberate.
3. Wrong: a timeout and a connection refused are both just failures. Right: refused means something answered and said no; a timeout means nothing answered at all. They point at different causes.
4. Wrong: private addresses in a traceroute mean something is misconfigured. Right: ISPs number their own cores privately as normal practice, which is why the reader saw 172.31, 172.26 and 172.16 addresses.
5. Wrong: git rev-parse origin/main tells you where the remote is. Right: it tells you where the remote was at your last fetch. Only git ls-remote asks the server.
6. Wrong: a disconnect during a push means the push did not happen. Right: the disconnect happened while reading the reply, so it says nothing about whether the write landed. Re-query and find out.
7. Wrong: 1 GB and 1 GiB are the same. Right: they differ by 7.4 per cent, and the gap grows with every step up the scale.
8. Wrong: MB/s and Mb/s are the same. Right: they differ by a factor of eight. A 100 Mbit/s line delivers about 12 MB/s at best.
9. Wrong: a green tick on a branch means your commit passed CI. Right: it means some run passed for something on that branch. Check the run’s head SHA against your own.
10. Wrong: hashing a password with SHA-256 is secure storage. Right: SHA-256 is far too fast. Use Argon2id, scrypt or bcrypt with the parameters in section 52.13.3.

52.99 Chapter summary in 20 lines#

1. This chapter is for looking things up, not for reading through.
2. Learn the latency ladder as shape, not as digits: cache in nanoseconds, memory in hundreds of nanoseconds, storage in microseconds, disks and networks in milliseconds.
3. Scaled to human time at one nanosecond per second, a cross-continent round trip is several years and a fifteen-second timeout is several centuries.
4. About one millisecond of round trip per hundred kilometres of fibre is a rule you can do in your head, and it is physics.
5. Learn 2^10, 2^20 and 2^30, and the three that cause bugs: 2^31, 2^32 and 2^53.
6. Decimal and binary prefixes differ by a percentage that grows with scale: 2.4 per cent at kilo, 7.4 at giga, 20.9 at yotta.
7. One hex digit is four bits and two are one byte. That is the only reason hexadecimal exists in computing.
8. UTF-8 costs one byte for English, two for most European scripts, three for Devanagari and East Asian scripts, four for emoji.
9. The seven OSI layers are the vocabulary; the four TCP/IP layers are what actually runs. Both statements are true together.
10. The private ranges are 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16, from RFC 1918, and the middle one reaches to 172.31.
11. For every command, ask two questions: what does a pass prove, and what does it not prove. That second question is the whole method.
12. For “I cannot reach a site”, work upward: link, DNS, TCP, TLS, HTTP, content. Each pass eliminates everything below it.
13. Refused, unreachable and silent are three different failures with three different causes, and only the third costs you a full timeout.
14. Report a fault to an ISP in protocol terms with a control test, and never accuse anyone; the evidence supports a silent drop and does not name a party.
15. Almost every git command works offline, because the whole history is in the .git directory on your machine.
16. Git failures fall into five families: transport, authentication, scope, server-side rejection and local. Naming the family is most of the fix.
17. After any failed push, ask the server what it now believes with git ls-remote before doing anything else.
18. Quote your variables, start scripts with set -euo pipefail, and never pipe an unknown URL into a shell.
19. A magic number identifies a file; the extension only suggests one.
20. When stuck: reproduce, read the real error, separate what is proved from what is assumed, change one thing at a time, and write down the fix.