Why Mobile Data Worked, and the Diagnostic Method

29.0 What this chapter gives you#

1. You will be able to state the difference between “the site is down” and “I cannot reach the site”, and say who is affected by each.
2. You will be able to give a one-line test that separates those two claims, and explain why one success anywhere beats a hundred failures here.
3. You will be able to list every single thing that changed when the reader’s phone switched from home broadband to mobile data, and there are more than a dozen.
4. You will be able to explain why that test proved the destination was alive and yet proved nothing at all about which difference mattered.
5. You will be able to run a controlled experiment on a network: one variable at a time, with a stated hypothesis and a stated eliminating power.
6. You will be able to climb a seven-rung diagnostic ladder from “is my cable plugged in” to “where exactly does the path stop”, and say for each rung what a pass proves, what a pass does not prove, and what a failure means.
7. You will be able to map the reader’s own real fault onto that ladder and name the small set of causes that survive the evidence.
8. You will be able to use ping, traceroute, dig, nc, curl, mtr and a dozen other macOS commands with the flags that matter, and know what each one can and cannot prove.
9. You will be able to write up a fault so that an internet provider acts on it instead of asking you to reboot your router.
10. You will be able to catch an intermittent fault with a logging script, and tell a regional outage from a personal one.

29.1 “The site is down” and “I cannot reach the site”#

PLAIN29.1.1 in simple words#

1. These two sentences sound like the same complaint. They are not.
2. “The site is down” is a claim about the whole world. It says nobody can use it. Not you, not your friend, not a person in another country.
3. “I cannot reach the site” is a claim about you. It says that from where you are sitting, right now, with this machine and this connection, it fails.
4. The second claim is small and almost always true when you say it. You did try. It did fail. There is nothing to argue about.
5. The first claim is enormous. To be true, it has to hold for every person on the planet at the same moment.
6. You almost never have the evidence for the first claim. You only ever have evidence for the second.
7. So when you say “GitHub is down” after one failed page load, you have quietly upgraded a small true statement into a huge unproven one.
8. That upgrade is what wastes the hours. Once you believe the site is down, you stop testing. Waiting feels like the correct action.
9. And if the real fault is on your own path, waiting fixes nothing, because nobody at the other end knows anything is wrong.
10. The whole method in this chapter is built to stop you making that jump.

PLAIN29.1.2 a picture in your head#

1. Imagine you post a letter to a friend in another city, and no reply comes.
2. You could say “my friend has moved away”. That is a claim about your friend.
3. Or you could say “my letter did not get a reply”. That is a claim about one letter, on one day, through one postal route.
4. The second is what you actually know. The first is a guess dressed up as knowledge.
5. There are many ways to get silence. The friend moved. The friend is ill. The letter was lost. The postbox at the end of your street is broken. A sorting office is on strike. The address on the envelope was fine but a customs check quietly binned it.
6. Now here is the trick that settles it in one move. You ask a person in another city to post the same letter to the same address.
7. If they get a reply, your friend has not moved. That is now certain. The problem is somewhere on your route, not at the far end.
8. You still do not know which part of your route. But you have cut the space of possible causes roughly in half, with one action.

Where this comparison breaks: letters are slow and rare, so you only get to test once or twice. Network tests are cheap and instant, so you can run twenty of them in a minute and narrow the cause much further than a letter ever could. Also, a letter has one route. A network packet may take a different route on every attempt, which is why an intermittent fault is so much harder than a missing friend.

PLAIN29.1.3 a worked example#

1. Here is the reader’s real situation, in the two forms.
2. Weak form, the one that wastes time: “GitHub is down.”
3. Strong form, the one that leads somewhere: “From my Mac, on my home broadband in India, curl to github.com on port 443 gets no response of any kind and gives up after 15 seconds.”
4. The strong form names the machine, the network, the destination, the port, the tool, the symptom and the time. Every one of those is a variable you can change to learn something.
5. Now the one-line test that separates the two claims.

# Run this from a network that shares nothing with yours.
# Mobile data on a phone, or a friend's connection.
curl -sS -o /dev/null -w '%{http_code}\n' \
     --max-time 10 https://github.com

6. If that prints 200, the site is not down. That is settled, permanently, by a single observation. No amount of failing from your desk can undo it.
7. If it also hangs and prints nothing, then the site being down has become a live possibility, and now a status page is worth reading.
8. The reader did exactly this test, in the simplest possible way. The same phone, switched from home Wi-Fi to mobile data, loaded github.com instantly.
9. So the claim “GitHub is down” was dead within about ten seconds. Everything after that was about finding the reader’s own path problem.

PLAIN29.1.4 what is really happening inside#

1. There is a piece of plain logic underneath this, and it is worth learning once, because it applies far beyond networks.
2. “The site is down” means “for all people, the site fails”. That is called a universal claim. It talks about everybody.
3. “I cannot reach the site” means “there exists at least one person, me, for whom it fails”. That is called an existential claim. It talks about one case.
4. To prove a universal claim you would have to check everybody. You cannot.
5. To destroy a universal claim you need one counterexample. One person who can reach it. That is cheap, and you can do it in ten seconds.
6. To prove an existential claim you need one example. Your own failure is that example. So your own claim is already proved.
7. This gives an unfair-looking rule that is completely correct: a single success anywhere beats any number of failures in one place.
8. It also gives the reverse warning: no number of failures from your desk ever proves the site is down. Not ten, not a thousand. They are all the same observation repeated.
9. So the cheapest useful action in any connectivity fault is to buy one counterexample from a different vantage point.
10. A vantage point is just a place to test from: another network, another machine, another country, another provider.

TECHNICAL29.1.5 the engineer’s version#

1. The distinction maps onto a standard split in incident work: a service-side fault versus a client-side or path-side fault.
2. A service-side fault has a blast radius covering all or a defined subset of users, is visible to the operator’s own monitoring, and appears on a status page. Only the operator can fix it.
3. A path-side fault has a blast radius bounded by a network prefix, an autonomous system, a region or a single subscriber line. The operator’s monitoring usually shows nothing at all.
4. This asymmetry is why large operators publish per-region status, and why “all systems operational” on a status page is compatible with your total inability to reach the service.
5. The logic in block 4 is Karl Popper’s asymmetry between verification and falsification, set out in Logik der Forschung in 1934 and in English as The Logic of Scientific Discovery in 1959. Universal statements cannot be verified by observation but can be falsified by one counterexample.
6. The operational form of that rule is the single-vantage-point problem. One observer cannot distinguish global outage from local unreachability.
7. Distributed measurement platforms exist precisely to fix this. RIPE Atlas, launched by the RIPE Network Coordination Centre in 2010, runs a global fleet of small hardware and software probes, on the order of ten thousand active at any time, that anyone can task with ping, traceroute, DNS and TLS measurements from many countries at once.
8. Commercial equivalents include ThousandEyes, Catchpoint and Pingdom. Free web-based checkers do a weaker version of the same thing.
9. Terms to use precisely when reporting:

10. The verb matters. “Down” is a statement about the server. “Unreachable” is a statement about a pair: a source and a destination. Reachability is always a property of a pair, never of a single host.
11. That single sentence, that reachability is a property of a pair, is the most useful idea in this chapter.

WORDS29.1.6 remember these#

1. Down — nobody can use it — a service-side failure affecting all or a defined subset of users, visible in the operator’s own monitoring.
2. Unreachable — I cannot get to it from here — a property of a source and destination pair, not of the destination alone.
3. Vantage point — a place you test from — an independent measurement source with its own address, provider and path.
4. Universal claim — a statement about everybody — falsifiable by a single counterexample, not verifiable by observation.
5. Existential claim — a statement about at least one case — proved by a single example, which your own failed attempt already supplies.
6. Blast radius — how many people a fault hits — the set of users, prefixes or regions affected by a given failure.

29.2 Everything that changed when the phone used mobile data#

PLAIN29.2.1 in simple words#

1. The reader’s test was simple. Same phone. Turn off Wi-Fi. Turn on mobile data. Load github.com. It loaded instantly.
2. It is tempting to read that as “so the problem is my Wi-Fi” or “so the problem is my router”. Both may be wrong.
3. Switching to mobile data does not change one thing. It changes almost everything between you and the destination.
4. Here is the honest list of what changed, and there are more items than most people expect.
5. The company carrying your traffic changed. Home broadband company out, mobile company in.
6. Your address on the internet changed. Every reply now comes back to a different number.
7. The network that owns that address changed, which means the route the reply takes back to you is completely different too.
8. The physical path changed. Different cables, different cities, different handover point into the destination’s network.
9. The name lookup probably changed. The phone on mobile data usually asks the mobile company’s name server, not the one your Mac was using.
10. Because the lookup changed, the address you got back may have changed, so you may have talked to a different machine at the far end entirely.
11. The boxes in the middle changed. Home router, home firewall and your provider’s inspection equipment are all out of the picture. The mobile company’s own equipment is in it.
12. The kind of address may have changed. Mobile networks in India commonly give you a modern address type that home broadband often does not.
13. And a small one that matters more than people think: time passed. The second test happened minutes after the first.
14. Only one of those needs to be the cause. The test cannot tell you which.

PLAIN29.2.2 a picture in your head#

1. Imagine your parcel never arrives, so you try a different courier.
2. The second courier delivers it in an hour. You now know the address is real and somebody is there to sign for it.
3. That is genuinely valuable. It ends the argument about whether the recipient exists.
4. But look at what else changed when you switched courier. A different van, a different driver, a different depot, a different route, a different customs agent, a different set of rules about what they will carry, and a different day.
5. If someone now says “so the first courier’s van was broken”, they have invented an explanation. The van is one of seven things that changed.
6. Maybe the first courier refuses parcels over a certain size. Maybe their depot in one city is on strike. Maybe their scanner rejects your postcode.
7. You cannot tell. You changed everything at once, so the experiment has no resolving power about the cause.
8. It has enormous power about one specific question, and that question is “does the recipient exist”. That question is now closed.

Where this comparison breaks: couriers deliver or fail visibly, and you get a tracking page. A network path can fail silently, with no message to anybody, and neither end learns that a packet was dropped. The far end in the reader’s case never knew a connection had been attempted at all.

PLAIN29.2.3 a worked example#

1. Let us do the enumeration properly for the reader’s case, item by item.
2. On home broadband, the Mac had a private address behind the router at 192.168.0.1, and used the public name server 1.1.1.1.
3. github.com resolved to 20.207.73.82, which is an address owned by Microsoft, the company that has owned GitHub since 2018.
4. The connection attempt to that address on port 443 got no reply of any kind.
5. On mobile data, at minimum, the following changed.

6. That is eleven differences, and a careful person could find more.
7. Now the key sentence. Only one of those eleven needs to be the cause.
8. The test therefore proves exactly one thing: at that moment, some client somewhere could complete a request to github.com, so the service was up.
9. It does not prove your Wi-Fi is bad. It does not prove your router is bad. It does not prove your provider is blocking anything. It does not prove the address was different.
10. Those are all still open. The test closed one question and left eleven.

PLAIN29.2.4 what is really happening inside#

1. Let us go through why each change could, on its own, produce the observed result.
2. Different provider. Providers buy connectivity from different upstream networks. If one of the reader’s provider’s upstream links to the destination network is broken, packets die there and nowhere else.
3. Different public address. Some destinations block or rate-limit ranges of addresses. If the reader’s home range was on such a list, packets would be dropped at the far end with no reply. That behaviour is normal for abuse controls.
4. Different owning network. The reply has to find its way back to you. If the return route from the destination network to your provider is broken, your request may arrive perfectly and the answer never comes home. You cannot tell the difference from your side.
5. Different name server. A different name server may hand back a different address for the same name, because large services steer users to a nearby machine based on where the question came from.
6. Different address, therefore a different machine at the far end. If the original machine was in trouble and the new one was not, the fault is at the destination but only for one of its many front doors.
7. Different address type. If mobile data used the newer, longer kind of address and home broadband had none, then the successful request travelled over a completely separate network layer, sharing almost nothing with the failed one.
8. Different middle boxes. Equipment that inspects traffic can drop a connection silently based on the name being requested, the destination address, the port, or a pattern inside the first packet.
9. Different time. A brief routing problem lasting a few minutes looks exactly like a permanent block if you only test twice.
10. So the correct conclusion from the mobile-data test is one sentence: the destination service is alive and serving at least some clients, and the fault lies in something that differs between the two attempts.
11. Anything more specific than that is a guess. Guesses are allowed, but they must be labelled as guesses and then tested one at a time.

TECHNICAL29.2.5 the engineer’s version#

1. Enumerated precisely, the variables changed by a Wi-Fi to cellular switch are as follows.
2. Access technology: IEEE 802.11 wireless local area network to 4G LTE or 5G New Radio, which means a different link layer, a different maximum frame size and a different scheduler.
3. Source address: a new public IPv4 address, almost always behind carrier grade network address translation on the mobile side.
4. Autonomous system: the autonomous system number, the identifier for a network under one routing policy, changes. The reader’s fixed-line provider and mobile operator hold different numbers.
5. Border Gateway Protocol path: both the forward path and, critically, the return path change. The return path is chosen by the destination network based on the new source prefix and is invisible from the client.
6. Peering relationship: the point where traffic hands off into Microsoft’s network changes. The reader’s fixed-line trace entered Microsoft at ae66-0.del01-96cbe-1b.ntwk.msn.net, a Delhi edge router. A mobile operator may hand off in Mumbai, Chennai or through a different transit provider altogether.
7. Recursive resolver: from 1.1.1.1, Cloudflare’s public resolver launched on 1 April 2018, to the mobile operator’s own resolver, reached through the cellular data profile.
8. Resolved address: GitHub fronts through Microsoft edge, and the answer depends on the querying resolver’s location and on any EDNS Client Subnet information, defined in RFC 7871. A different resolver can and often does yield a different anycast front end.
9. Address family: the session reported IPv6: (none) on the fixed line. Indian mobile networks commonly run IPv6-only bearers with 464XLAT, defined in RFC 6877 from April 2013, so the phone may have used a AAAA record and an entirely separate IPv6 path.
10. Middlebox set: home NAT, provider transparent proxies, deep packet inspection appliances and any Server Name Indication filtering differ completely between a fixed-line and a mobile bearer.
11. Path maximum transmission unit: typical fixed-line paths carry 1500 or 1492 bytes; cellular bearers frequently present 1400 to 1428 bytes, which changes fragmentation and TCP maximum segment size behaviour.
12. TCP and TLS fingerprint: a different operating system produces a different initial window, a different option order and a different TLS ClientHello. Filtering devices that key on ClientHello fingerprints will treat them differently.
13. Temporal variable: the two observations are not simultaneous, so any transient routing event separates them.
14. In experimental terms this is a confounded comparison. Thirteen variables moved together, so the experiment has a very high power to answer one binary question and near-zero power to localize a cause.
15. The formally correct conclusion is exactly this: the destination service was reachable and functional from at least one vantage point at time T, therefore the hypothesis “service down” is falsified for time T.

WORDS29.2.6 remember these#

1. Autonomous system — one company’s network under one policy — a routing domain identified by an autonomous system number and announced in BGP.
2. Return path — how the answer comes back — the route chosen by the remote network towards your source prefix, invisible from the client side.
3. Peering point — where two networks hand traffic over — a physical or virtual interconnection governed by a peering or transit agreement.
4. Anycast front end — many machines sharing one address — the same address announced from multiple locations, with routing picking the nearest.
5. Confounded comparison — too many things changed at once — an experiment in which multiple independent variables move together, so no single cause can be attributed.
6. Middlebox — a box in the path that does more than forward — any device that inspects or alters traffic, such as a NAT, proxy, firewall or DPI appliance.
7. 464XLAT — a way to run IPv4 apps on an IPv6-only phone network — RFC 6877, combining a customer-side translator with a provider-side NAT64.

29.3 The logic of a controlled experiment on a network#

PLAIN29.3.1 in simple words#

1. A controlled experiment is an old and simple idea. Change one thing. Keep everything else the same. See what happens.
2. If the result changes, the thing you changed is involved. If it does not, the thing you changed is probably innocent.
3. That is it. There is no more to the method than that sentence.
4. Networking breaks the rule constantly, because the easy tests change many things at once. Switching to mobile data changes thirteen things.
5. So the discipline is to design tests that move one thing and hold the rest still.
6. Before you run a test, say out loud what each possible result would mean.
7. If you cannot say what a pass would prove and what a fail would prove, the test is not worth running yet. Think first, type second.
8. This book gives that idea a name: the eliminating power of a test. It means how many possible causes a test can strike off your list.
9. A good test has high eliminating power. It strikes off half the list whichever way it goes.
10. A weak test is one where a pass tells you nothing and only a fail helps, or the other way round.
11. And there is one rule you must never break. Never conclude more than the evidence supports. Write down what you proved, not what you suspect.

PLAIN29.3.2 a picture in your head#

1. Think of the game where someone thinks of a number between 1 and 100 and you have to find it, and they only answer yes or no.
2. A bad question is “is it 37”. Usually the answer is no, and you have removed one number out of a hundred.
3. A good question is “is it more than 50”. Whatever the answer, half the numbers vanish.
4. Seven good questions find any number in 1 to 100. Ninety-nine bad questions might still miss.
5. Fault finding is that game. The list of possible causes is the numbers. Each command you type is a question.
6. Your job is to ask the question that halves the list, not the question that confirms your favourite theory.
7. And notice something about the game. You are only allowed to trust the answers you actually got. If they said “more than 50”, you may not write down “so it is 73”.

Where this comparison breaks: in the number game each answer is truthful and final. Networks lie. A test can pass by luck on an intermittent fault, and a test can fail for a reason unrelated to what you were testing, such as a tool that blocks certain traffic. So you sometimes need to ask the same question twice, in two different ways.

PLAIN29.3.3 a worked example#

1. Suppose the list of possible causes for the reader’s fault starts like this.

A  The Mac's network settings are wrong
B  The Wi-Fi link to the router is bad
C  The home router is broken or blocking
D  The ISP's access network is broken
E  The ISP blocks this destination
F  The path into Microsoft is broken
G  Microsoft drops traffic from this address range
H  github.com is down
I  DNS gave a wrong or stale address

2. Nine causes. Now consider two candidate tests.
3. Test one: reload the page in a different browser. If it fails, you have removed nothing at all, because every cause above would still produce a failure. If it succeeds, you have learned that the first browser was at fault, which was not even on the list. Very low eliminating power.
4. Test two: from the same Mac, run a name lookup and then try to open a connection to the numeric address directly.
5. If the name lookup works and the connection still fails, you strike off I, because naming worked, and you learn the failure is later than naming.
6. Test three: load a different website, say example.com, from the same Mac on the same Wi-Fi. If it loads, you strike off A, B, C and D in one move, because a working page proves the Mac, the Wi-Fi, the router and the access network all carry traffic to the wider internet.
7. That single test removed four of nine causes. That is high eliminating power, and it takes two seconds.
8. Test four: mobile data. It strikes off H, because the site answered someone.
9. After those tests the list is E, F and G. Three causes instead of nine, in under a minute, and every step was justified.
10. Notice that no test so far can separate E, F and G, because all three produce identical silence from where you sit. That is an honest and useful place to stop and say so.

PLAIN29.3.4 what is really happening inside#

1. There is a hidden structure to good fault finding, and it is worth naming.
2. Every fault has a chain of things that must all work. Cable, address, router, route, naming, connection, encryption, application.
3. A chain fails at exactly one link first. Your job is to find the first broken link, not to admire the whole chain.
4. Searching a chain has a best method: cut it in the middle. Test something halfway along. Then you know which half holds the break, and you repeat.
5. That is called bisection, and it is the same method used to find which change in a code history broke a program.
6. The ladder in the next section is a chain laid out in order, from the nearest link to the furthest, so that you can walk it or bisect it.
7. Walking from the bottom is slower but never confuses you. Bisecting is faster but needs you to know the chain well.
8. One more rule. When a test surprises you, believe the test and change your model. Do not explain the result away.
9. Most long debugging sessions are long because somebody decided early what the answer was and then read every result as support for it.
10. Write down, in words, what you have proved after each step. A three-line note stops you from re-testing the same thing four times.

TECHNICAL29.3.5 the engineer’s version#

1. Formally, define the hypothesis space H as the set of candidate root causes consistent with all evidence so far.
2. A test T partitions H into the subset consistent with a pass and the subset consistent with a fail. Define the eliminating power of T as the size of the smaller of those two subsets, expressed as a fraction of H.
3. A test with eliminating power near 0.5 is a bisecting test. A test with power near 0 is decorative.
4. This is the information-theoretic view: the ideal test maximizes expected information gain, which for a binary test is maximized when the two outcomes are equally likely.
5. Medicine names the same two properties. A highly sensitive test with a negative result rules a condition out. A highly specific test with a positive result rules it in. Network tests have exactly these two flavours.
6. Worked eliminating power for common tests, taking the nine-cause list above:

7. Design rules that follow from this.
8. Hold the destination fixed and vary the source, or hold the source fixed and vary the destination. Never vary both.
9. Vary one layer at a time. Name resolution, then network reachability, then transport, then application.
10. Prefer tests whose failure mode is distinguishable. A silent timeout is a weak signal because many causes produce it. A TCP reset, an ICMP administratively prohibited message, or an HTTP 403 are strong signals because few causes produce them.
11. Record the exact command, the exact output and the timestamp. Use date -u +%FT%TZ on macOS to stamp a log line in coordinated universal time, so that evidence from two machines can be lined up.
12. Beware of the tool as a variable. curl, a browser, and git do not use identical name resolution, proxy settings or TLS libraries. A browser may use its own encrypted DNS while curl uses the system resolver.
13. Beware of caching as a hidden variable. Negative DNS answers, TCP connection reuse and HTTP caches can make a second test not independent of the first. Start a fresh process for each test where it matters.
14. State conclusions in the form: evidence, therefore eliminated set. Anything else is a hypothesis and must be labelled as one.

WORDS29.3.6 remember these#

1. Controlled experiment — change one thing only — a test in which a single independent variable is varied while all others are held constant.
2. Eliminating power — how much of your list a test removes — this book’s name for the fraction of the hypothesis space excluded by either outcome.
3. Bisection — cut the chain in the middle — binary search over an ordered sequence of dependencies to locate the first failing link.
4. Confounding variable — something else that moved too — an uncontrolled variable that could equally explain the observed change.
5. Silent failure — nothing comes back at all — a drop with no ICMP error and no TCP reset, which is the least informative failure mode there is.
6. Strong signal — a failure that names its own cause — an explicit error such as TCP RST, ICMP type 3 code 13, or an HTTP status code.

29.4 The diagnostic ladder, rung by rung#

PLAIN29.4.1 in simple words#

1. Getting a web page involves a chain of things that must all work.
2. The ladder puts that chain in order, from the closest thing to you up to the furthest thing away.
3. Rung 1: does my machine have a working network connection with an address.
4. Rung 2: can I talk to my own router, the box in my home.
5. Rung 3: can I talk to a machine out on the internet using its number.
6. Rung 4: can I turn a name into a number.
7. Rung 5: can I open a connection to the exact service I want, on its port.
8. Rung 6: does the full request, with encryption, actually succeed.
9. Rung 7: if it fails, how far along the path do my packets get.
10. You climb from 1 upwards. The first rung that fails is where to look.
11. Every rung passing means the rungs below it are fine. That is the whole value of putting them in order.
12. And every rung has a limit. Passing a rung proves something specific and nothing more, and it is worth knowing exactly what.

PLAIN29.4.2 a picture in your head#

1. Think of posting a parcel, and the checks a careful person makes.
2. Rung 1 is “do I have a pen and paper and a return address”. Without an address of your own, nothing can come back.
3. Rung 2 is “is the postbox at the end of my street working”. If your own postbox has been sealed, nothing leaves.
4. Rung 3 is “can I send anything at all to another city”. Not to a specific person. Just anything, anywhere far away.
5. Rung 4 is “can I look up my friend’s street address from their name”.
6. Rung 5 is “is there a door at that address, and does it open”.
7. Rung 6 is “did the person inside take the parcel, check my identity papers and give me a receipt”.
8. Rung 7 is “if it never arrived, which sorting office was the last to scan it”.
9. Notice you cannot sensibly ask question 6 before question 5. There is no point asking whether the person signed for it if the door does not exist.
10. That ordering is what stops you wasting time. Most people start at rung 6, fail, and then guess.

Where this comparison breaks: with a parcel, every stage leaves a record and somebody can tell you where it stopped. On a network the middle stages usually leave no record at all, and a packet can be dropped by a machine that never tells anyone, including its own operator.

PLAIN29.4.3 a worked example#

1. Here is the ladder as a table. macOS commands, since that is what the reader uses. Linux differences come later in this section.

2. And here is what each rung settles, and what it leaves open.

3. Third table: what a failure at each rung means.

4. Two of those entries deserve a warning right away.
5. Rung 3 failing may only mean that ping replies are blocked, not that routing is broken. Always confirm with a second test that does not use ping.
6. Rung 7 failing to reach the end is the weakest evidence on the whole ladder. Silence at the end of a trace is completely normal.

PLAIN29.4.4 what is really happening inside#

1. Rung 1. Your machine needs three things: a working link, an address of its own, and a default route, which is the instruction “send anything not local to the router”.
2. A pass proves the network card is on and the machine has an address, usually handed out by the router. A pass does not prove that any packet has ever left the building.
3. A failure means the cable is out, the Wi-Fi did not join, or the address handout failed. If you see an address starting 169.254, the handout failed and the machine gave itself a placeholder.
4. Rung 2. Ping your own router. On the reader’s network that is 192.168.0.1.
5. A pass proves the wireless or cable link works, your address is on the right network, and the router is alive and answering.
6. A pass does not prove the router can reach anything else. A router with a dead internet line still answers you cheerfully.
7. A failure means the local link is broken, or your address is on a different network from the router, or the router blocks ping to itself, which is rare for home routers and common for company ones.
8. Rung 3. Ping a numbered address out on the internet, such as 1.1.1.1 or 8.8.8.8. Use the number, never a name.
9. A pass proves your packets leave your network, cross your provider, reach a distant machine, and come back. That is a lot, and it is proved in one line.
10. It also separates two questions that beginners mix up: reaching things, and naming things. Using a number takes naming out of the experiment.
11. A pass does not prove that naming works, and does not prove any particular site is reachable. Different destinations take different paths.
12. A failure is ambiguous, because many networks drop ping on purpose. Retest with something that is not ping before concluding anything.
13. Rung 4. Turn a name into a number. This is the naming system doing its job.
14. A pass proves your machine can reach a name server and get an answer, so both naming and the path to the name server are healthy.
15. A pass does not prove the returned address is reachable, or correct, or fresh. A cached wrong answer looks exactly like a right one.
16. A failure means your name server is unreachable or refusing. Test again naming a different server directly to separate the two.
17. Rung 5. Open a connection to the actual port. Web pages over encryption use port 443. This is the first rung that tests the service you actually want.
18. A pass proves that packets reach the service, that the service is listening and that a full handshake completed in both directions. This is the strongest single proof of reachability there is.
19. A pass does not prove the encryption will succeed, that the certificate is valid, or that the site will answer with a page.
20. A failure with a refusal message means something answered and said no. A failure with silence means nothing answered at all, which is the harder case and the one the reader hit.
21. Rung 6. Make the real request. Now encryption and the web protocol are involved as well.
22. A pass proves everything below, plus certificate checking, plus a server that produced a real answer.
23. A pass does not prove it works for other people, other names on the same machine, or the rest of the site.
24. A failure here, with rung 5 passing, is a very specific and useful state: the network is fine and the problem is encryption or the application.
25. Rung 7. Find out how far your packets get by sending probes with a short life and collecting the complaints from each machine along the way.
26. A pass, meaning a complete trace, proves the forward path exists and each machine on it was willing to identify itself.
27. A pass does not prove your actual traffic follows that same path, because the probe may use a different protocol and port from your real connection.
28. A failure, meaning stars at the end, mostly means the last machines choose not to reply. It is a hint, not a verdict.

TECHNICAL29.4.5 the engineer’s version#

1. Rung 1, interface and address. Three commands, each answering a different part of the question.

ifconfig en0
ipconfig getifaddr en0
networksetup -getinfo "Wi-Fi"
netstat -rn -f inet | head -5

2. ifconfig en0 prints link state and addresses. Look for status: active and an inet line. macOS prints the mask in hexadecimal, for example netmask 0xffffff00, which is a /24.
3. ipconfig getifaddr en0 prints only the IPv4 address and exits with status 1 and no output if there is none, which makes it ideal inside scripts.
4. networksetup -getinfo "Wi-Fi" prints the service-level view: address, mask, router and hardware address. Use networksetup -listallhardwareports first to map en0 to a service name, because on many Macs en0 is Wi-Fi and on others it is a wired or adapter port.
5. netstat -rn -f inet shows the routing table. The default line names the gateway and interface. Without a default route, nothing off-link works.
6. On macOS, utun interfaces are point-to-point tunnels. The reader had several. They are created by VPN clients, by Apple’s own services such as Back to My Mac and iCloud Private Relay, and by content filters. A utun with a default route can silently capture traffic. Check with route -n get default which interface actually wins.
7. Rung 1 pass criteria: status: active, a non-169.254 address, a default route. 169.254.0.0/16 is link-local, defined in RFC 3927 from May 2005, and self-assigned when DHCP fails.
8. Rung 2, the default gateway. The reader’s is 192.168.0.1, inside the 192.168.0.0/16 private range from RFC 1918, published February 1996.

ping -c 3 -W 1000 192.168.0.1
arp -a -n | grep 192.168.0.1

9. -c 3 sends three packets. -W 1000 waits at most 1000 milliseconds per reply. Note that on macOS -W is milliseconds, while on Linux -W is seconds. This is a real and frequent source of confusion.
10. arp -a -n shows the Address Resolution Protocol cache. If the gateway appears with a hardware address, layer 2 resolution succeeded even if ICMP is filtered. An (incomplete) entry means the router did not answer ARP at all, which is a much more serious finding.
11. Rung 3, off-network reachability by address.

ping -c 5 1.1.1.1
ping -c 5 8.8.8.8
nc -vz -G 3 1.1.1.1 443

12. 1.1.1.1 is Cloudflare’s resolver, launched 1 April 2018. 8.8.8.8 is Google Public DNS, launched 3 December 2009. Both answer ICMP echo and both listen on TCP 443, so you can test the same target two ways.
13. If ping fails and nc succeeds, ICMP is filtered and routing is fine. That pair of results is worth more than either alone.
14. Rung 4, name resolution.

dig +short github.com
dig @1.1.1.1 +short github.com
dig @8.8.8.8 +short github.com
scutil --dns | head -20
dscacheutil -q host -a name github.com

15. Plain dig uses the system resolver list. dig @server bypasses it and asks a named server directly. Running both separates “resolution is broken” from “my configured resolver is broken”.
16. scutil --dns is the macOS-specific view of resolver configuration, including per-domain resolvers installed by VPN clients. This is where a utun interface reveals that it has hijacked name resolution for some domains.
17. dscacheutil -q host -a name github.com asks the macOS directory service, which is what applications actually use. dig talks to DNS directly and bypasses the system cache, so the two can disagree. That disagreement is itself a diagnosis.
18. Apple ships dig, host and nslookup as an old bundled copy of the BIND utilities from the Internet Systems Consortium. The bundled version has lagged upstream by years. If your Mac lacks dig, install a current one with brew install bind and use dscacheutil or dns-sd -G v4 name in the meantime. Treat the exact presence of these tools as version-dependent rather than guaranteed.
19. Rung 5, transport reachability.

nc -vz -G 5 github.com 443
nc -vz -G 5 20.207.73.82 443
curl -v --connect-timeout 5 -o /dev/null https://github.com

20. -v makes nc report the outcome, -z means connect and send no data, -G 5 sets a five second TCP connection timeout. Without -G, nc waits for the operating system default.
21. On macOS the operating system default for a connection attempt is governed by sysctl net.inet.tcp.keepinit, whose default is 75000, in milliseconds. So an unanswered connection takes about 75 seconds to fail if no shorter timeout is set.
22. On Linux the equivalent is net.ipv4.tcp_syn_retries, default 6, giving roughly 127 seconds with exponential backoff from an initial retransmission timeout of 1 second as specified in RFC 6298 from June 2011.
23. Distinguish the three transport outcomes precisely:

24. Rung 6, application level.

curl -sSI https://github.com
curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' \
     https://github.com

25. -I sends a HEAD request, which asks for headers only. -s silences the progress meter, -S keeps error messages visible. Together -sS is the standard pairing for scripts.
26. A pass here exercises TCP, TLS 1.2 or 1.3, certificate chain validation against the system trust store, ALPN negotiation and HTTP.
27. Rung 7, path discovery.

traceroute -n -q 1 -w 2 -m 20 github.com
sudo traceroute -I -n github.com
sudo traceroute -n -P TCP -p 443 github.com

28. -n skips reverse name lookups and makes the trace much faster. -q 1 sends one probe per hop instead of three. -w 2 waits two seconds instead of five. -m 20 stops at twenty hops instead of the macOS default, which is the system TTL, normally 64.
29. macOS traceroute defaults to UDP probes to ports from 33434 upwards, the base port chosen by Van Jacobson when he wrote the tool in 1987 at Lawrence Berkeley Laboratory.
30. Path asymmetry rule: traceroute shows the forward path only, one hop at a time, and each displayed round-trip time includes the return path from that intermediate router, which may be entirely different.

WORDS29.4.6 remember these#

1. Default route — where to send anything not local — the 0.0.0.0/0 entry in the routing table, pointing at the gateway.
2. Link-local address — the placeholder a machine gives itself — 169.254/16 under RFC 3927, a sign that DHCP failed.
3. ARP cache — the table of neighbours’ hardware addresses — the result of Address Resolution Protocol, viewable with arp -a.
4. Three-way handshake — the opening of a connection — SYN, SYN-ACK, ACK, after which TCP considers the connection established.
5. RST — an explicit refusal — a TCP reset segment, meaning a host exists but nothing is listening or a device rejected the attempt.
6. HEAD request — ask for headers, not the page — an HTTP method returning the same headers as GET with no message body.
7. Bisection point — the rung that halves the problem — in practice rung 5, because it separates all network causes from all application causes.

29.5 The reader’s own fault, mapped onto the ladder#

PLAIN29.5.1 in simple words#

1. Now we take the real fault from the reader’s session and put every piece of evidence on a rung.
2. The Mac had an address and a working local network. Rung 1 passed.
3. The router at 192.168.0.1 answered. Rung 2 passed.
4. Packets reached machines out on the internet and came back. Rung 3 passed.
5. The name github.com turned into the number 20.207.73.82 quickly and correctly. Rung 4 passed.
6. The attempt to open a connection to 20.207.73.82 on port 443 got nothing at all. Rung 5 failed, and failed by silence.
7. Because rung 5 failed, rung 6 never happened. There was no encryption step and no web request, because there was no connection to run them over.
8. The trace showed packets travelling through the reader’s provider, into Microsoft’s network at Delhi, through Mumbai, into Pune, and then silence from hop 13 onwards. Rung 7 gave a partial answer.
9. And the same site loaded instantly on mobile data, which killed the theory that GitHub was down.
10. Put those together and something very specific appears.
11. Naming worked but connecting did not. That combination is one of the most informative results you can get, and the next block explains why.

PLAIN29.5.2 a picture in your head#

1. Go back to the parcel. You looked up your friend’s street address from a directory, and the directory answered instantly with a correct address.
2. So the directory service is fine, the roads to the directory office are fine, and you hold a real address in your hand.
3. You then walk to that address, knock, and nothing happens. No answer. No “go away”. No locked-gate sign. Just silence.
4. And you know from a friend in another city that somebody was home the whole time and answered their knock.
5. What is left? Something between your street and that door swallowed you.
6. The important part of the picture is the silence. A locked gate is information. A sign saying “no entry” is information. Nothing at all is the deliberate absence of information.
7. Anything that wants to hide the fact that it is stopping you produces exactly this: nothing.

Where this comparison breaks: a person walking can see the buildings they pass. Your packet cannot. It has no memory of where it has been, and if it is dropped it does not scream. Neither end is told. That is not a flaw in your test, it is how the network was designed.

PLAIN29.5.3 a worked example#

1. Here is the reader’s evidence, rung by rung.

2. And the cross-check, which sits outside the ladder.

3. A note on rung 3, told honestly. The session record does not include a ping 1.1.1.1 result. What it does include is that name lookups against 1.1.1.1 worked.
4. That is stronger evidence than ping, not weaker. It means packets left the Mac, crossed the provider, reached Cloudflare’s resolver, were processed, and the answers came all the way back.
5. So rung 3 is proved, but by a different observation than the one in the table. Say which observation proved a thing, not just that it is proved.
6. Here is the same picture as a diagram.

 RUNG                        READER'S RESULT
  7  traceroute            |  hop 12 Pune, then * * *
  6  curl -sSI https://    |  never ran
  5  nc / curl to :443     |  FAIL - silence, 15 s
  4  dig github.com        |  PASS -> 20.207.73.82
  3  reach a public IP     |  PASS via DNS to 1.1.1.1
  2  ping 192.168.0.1      |  PASS - hop 1 answered
  1  ifconfig / address    |  PASS - address held
       -------------------------------------------
       break is between rung 4 and rung 5

PLAIN29.5.4 what is really happening inside#

1. Why is “naming works, connecting fails” such a strong pair of results?
2. Because the two use different machines, different protocols, different ports, and, most importantly, different destinations.
3. The name lookup went to 1.1.1.1, a Cloudflare machine, over the simple message protocol on port 53.
4. The connection attempt went to 20.207.73.82, a Microsoft machine, over the connection protocol on port 443.
5. Naming working proves the general machinery is fine: the Mac, the router, the provider’s access network, the provider’s core, and the ability to send and receive from a distant network.
6. So the fault cannot be anything general. It must be specific to something about the second attempt.
7. Only three kinds of thing differ between the two attempts: the destination address, the port and protocol, and the path taken to that destination.
8. That is the entire surviving list. Everything else has been eliminated by the naming test.
9. Now add the silence. The failure produced no refusal message and no error report from any router.
10. If the far machine were up but not listening, you would get a refusal. You did not.
11. If a router had no route, it would normally send back a “cannot get there” complaint. You did not get one.
12. If a firewall were configured to reject politely, it would say so. It did not.
13. Silence is what a device does when it is configured to drop, or when a packet enters a hole in the routing and dies without anyone taking responsibility.
14. Now add the mobile data result. The destination answered somebody else at the same time.
15. The honest conclusion: the destination service was up, the reader’s own machine and local network were fine, naming was fine, and packets to that specific address on that specific port were being dropped somewhere between the reader’s provider and that service, with no error returned.
16. What we cannot say: which device dropped them, whether it was intentional, or which organization owns it. The evidence does not reach that far.

TECHNICAL29.5.5 the engineer’s version#

1. The evidence set, stated formally.
2. Layer 1 and 2: link active, ARP resolution to 192.168.0.1 succeeded, hop 1 in the trace answered. Eliminated as a cause.
3. Layer 3 local: address held, default route present, gateway responsive. Eliminated.
4. Layer 3 transit: UDP port 53 to 1.1.1.1 completed round trips with normal latency. Forward and return paths to Cloudflare’s anycast prefix are intact. Provider access network and core eliminated as general failures.
5. Layer 4 naming path: github.com resolved to 20.207.73.82, inside 20.192.0.0/10, registered to Microsoft and announced by AS8075. Correct and geographically plausible for India. DNS eliminated.
6. Layer 4 transport to target: curl -v https://github.com printed Trying 20.207.73.82:443... and produced no further output for 15 seconds. No TCP RST, no ICMP type 3 of any code.
7. That means the SYN was sent and retransmitted according to the operating system’s schedule, and nothing came back. On macOS a bare connect would run for about 75 seconds under net.inet.tcp.keepinit, so the 15 second cutoff indicates a client-imposed limit rather than the kernel giving up. Many tools set one.
8. Layer 7: never reached. No TLS ClientHello was ever sent, because TCP never established. Therefore no conclusion about certificates, SNI filtering at the TLS layer, or HTTP is available from this evidence.
9. Path evidence. The recorded trace, exactly as captured:

 1  192.168.0.1
 2  172.31.0.17
 3  137.97.29.249
 4  172.26.22.235
 5  172.16.18.33 / 172.26.14.75 / 172.26.22.234
 6  172.26.14.75 / 172.16.18.33
 7  ae66-0.del01-96cbe-1b.ntwk.msn.net (104.44.196.187)
 8  be23.rwa02.bom01.ntwk.msn.net (104.44.55.163)
 9  be1010.owr02.bom01.ntwk.msn.net (104.44.53.119)
10  be5.ibr02.pnq21.ntwk.msn.net (104.44.31.62)
11  ae104-0.rwa03.pnq21.ntwk.msn.net (104.44.20.52)
12  ae106-0.rwa04.pnq20.ntwk.msn.net (104.44.20.42)
13-20  * * *   (no reply)

10. Hops 2 to 6 are RFC 1918 private addresses inside the provider’s core. 172.16.0.0/12 covers 172.16.x through 172.31.x, so 172.31.0.17, 172.26.x and 172.16.18.33 are all private. Hop 3, 137.97.29.249, is public and belongs to the provider.
11. Hops with two or three addresses listed, namely 5, 6, 8, 9 and 11, are per-flow or per-packet load balancing across parallel links. Normal, not a fault.
12. From hop 7 the path is inside Microsoft’s backbone, ntwk.msn.net. Site codes follow airport-style city codes: del01 Delhi, bom01 Mumbai, pnq20 and pnq21 Pune.
13. Router role codes ibr, rwa and owr mean inter-region backbone router, regional wide-area router and outer wide-area router. These are Microsoft naming conventions, not a standard. Interface prefixes ae, be and po all mean a bundle of physical links treated as one logical link, and are vendor-specific words for the same idea.
14. Hops 13 to 20 returning * * * proves nothing on its own. Control-plane ICMP time-exceeded generation is routinely rate-limited or disabled inside large backbones. Cisco IOS, for example, rate-limits ICMP unreachables to one every 500 milliseconds by default.
15. The decisive evidence is therefore not the trace. It is the pairing of a silent TCP timeout with a successful request from a different vantage point at the same time.
16. Surviving hypothesis set, in the reader’s case:

17. Four hypotheses survive and no test performed so far separates them, because all four produce identical silence at the client.
18. Separating them requires evidence from outside the client: a TCP traceroute to port 443, a test to a second address in the same Microsoft prefix, a test from another host inside the same provider, or a looking-glass query from the destination side.

WORDS29.5.6 remember these#

1. Silent drop — nothing comes back — a packet discarded without generating ICMP or TCP error, by policy or by a routing black hole.
2. Black hole — a place packets vanish — a route that accepts traffic and discards it, often a null route or a stale advertisement.
3. Return path failure — your request arrives, the answer cannot come home — asymmetry where the destination network lacks a working route to your prefix.
4. Load-balanced hop — several addresses on one trace line — parallel links between the same two routers, chosen per packet or per flow.
5. Control plane — the router’s own brain — the processor that generates ICMP errors and runs routing protocols, deliberately rate-limited.
6. Surviving hypothesis set — what is still possible — the causes not yet eliminated by any observation, which is the only honest output of a diagnosis.

29.6 Cross-checks that multiply your certainty#

PLAIN29.6.1 in simple words#

1. A cross-check is a test that deliberately changes one big thing and keeps the rest as similar as possible.
2. Five of them cover almost every case, and each one eliminates a different family of causes.
3. Same destination, different network. Use mobile data or a hotspot.
4. Same network, different device. Use a phone on the same Wi-Fi, or a second computer.
5. Same destination, tested from far away. Use a shell on a server elsewhere, or a website that checks a site for you.
6. Same network, different destination. Load a different site on the same Wi-Fi.
7. Same destination, but tunnelled. Turn on a virtual private network, which wraps your traffic and sends it out from somewhere else.
8. Each of these answers a different question, and it is worth learning which.
9. The order matters too. Do the cheap ones first. Loading a different site takes two seconds and removes four causes.

PLAIN29.6.2 a picture in your head#

1. Think of a doctor with five simple tests and no scanner.
2. Press here. Does it hurt? Press there. Does it hurt? Each answer removes a set of possibilities.
3. No single press gives a diagnosis. The pattern of yes and no does.
4. The doctor is not looking for the answer in one test. The doctor is narrowing.
5. And a good doctor knows which press is worth doing first: the one that will change what they do next, whichever way it goes.
6. A test whose result would not change your next action is not worth doing.

Where this comparison breaks: a body is one system and stays put. Your network path can change between two presses, and a fault that lasts four minutes will give you a yes and then a no with no change in your hands.

PLAIN29.6.3 a worked example#

1. Here are the five cross-checks with exactly what each result eliminates.

2. Walk through them one at a time with the reader’s case.
3. Other network. The reader ran this. It passed. The claim “GitHub is down” is dead. What remains is everything on the reader’s own path.
4. Other device, same network. If a phone on the same Wi-Fi also fails, the Mac is not the problem, and every Mac-specific idea can be struck off: the utun tunnel interfaces, a local firewall, a broken resolver setting.
5. If the phone succeeds on the same Wi-Fi, that is enormous. It means the router and the provider carry the traffic, and something on the Mac is the cause.
6. Public shell elsewhere. Any server you can log into, in any country, confirms the site independently and with a proper command rather than a phone browser.
7. Other site, same network. Loading example.com from the same Mac on the same Wi-Fi proves the Mac, the Wi-Fi, the router, the provider’s access network and the provider’s core all work.
8. That is the highest-value two-second test on the list.
9. Same site over a virtual private network. Your traffic is wrapped up and sent out from a machine elsewhere, so it enters the destination from a different address and a different path.
10. If it works over the tunnel and not without it, that is a strong pointer towards path or policy rather than anything at the destination.
11. But notice the honest entry in the table. A failure over the tunnel eliminates very little, because tunnels themselves can be blocked, slow, or misconfigured, and you have added a whole new set of possible faults.

PLAIN29.6.4 what is really happening inside#

1. Each cross-check works by holding a set of components constant and swapping one whole layer.
2. Different network swaps the entire path and your address, and keeps the destination.
3. Different device swaps the operating system, the software and the local settings, and keeps the path.
4. Different destination swaps the far end and most of the path, and keeps everything local.
5. Tunnel swaps the middle of the path and your apparent address, and keeps both endpoints.
6. Elsewhere-shell swaps everything except the destination.
7. Put them in a grid and you can see that between them they cover every component: your machine, your local network, your provider, the path, and the destination.
8. This is why five tests can produce a confident answer where one test cannot.
9. There is a trap here. Tests are only independent if they really do differ.
10. A phone on the same Wi-Fi that has fallen back to mobile data without you noticing is not a same-network test. Turn mobile data off first.
11. A second computer that shares your virtual private network is not an independent path.
12. And an online checker that tests from a data centre may reach a completely different front-door machine than a home connection would, so a pass there does not prove your address family or your region is served.

TECHNICAL29.6.5 the engineer’s version#

1. Precise commands for each cross-check on macOS.

# 1. Same network, different destination
curl -sSI --max-time 8 https://example.com
curl -sSI --max-time 8 https://www.wikipedia.org

# 2. Same destination, force the address, skip DNS
curl -v --max-time 8 --resolve github.com:443:20.207.73.82 \
     https://github.com

# 3. Same destination, different port
nc -vz -G 5 github.com 22
nc -vz -G 5 github.com 443

# 4. Same destination, different address family
curl -4 -sSI --max-time 8 https://github.com
curl -6 -sSI --max-time 8 https://github.com

2. Test 2 is the one people forget. --resolve host:port:address tells curl to use that address for that name without a lookup, so you can test a specific front end while keeping the correct Server Name Indication and certificate name.
3. Test 3 separates destination reachability from port policy. In the reader’s session, the git transports gave exactly this kind of split evidence: one transport worked while the other did not, and those two use different ports.
4. Test 4 matters because the session reported IPv6: (none). If a destination is reachable over IPv6 and not IPv4 on your network, or the reverse, forcing the family names the fault immediately.
5. Independent vantage points, ranked by strength:

6. A looking glass is a public web or telnet interface on a provider’s router that lets you run ping and traceroute from that provider’s network towards you. Many transit providers and internet exchanges publish one. This is the only easy way to see the return path towards your own address.
7. RIPE Atlas, run by the RIPE Network Coordination Centre since 2010, will run a traceroute or TLS check towards your target from probes in chosen countries or chosen autonomous systems, which turns “is it just me” into a measurable question.
8. Record for each cross-check: source address family, source autonomous system, resolver used, resolved address, port, exact command, exact result, and a coordinated universal time stamp. Without the resolved address, two tests may not have been aimed at the same machine at all.

WORDS29.6.6 remember these#

1. Cross-check — a test that swaps one whole layer — an independent measurement designed to eliminate a family of causes at once.
2. Vantage point — where you measure from — a source with its own address, provider and path to the target.
3. Looking glass — someone else’s router you may query — a public interface for running ping, traceroute and BGP queries from a remote network.
4. Server Name Indication — telling the server which site you want — the SNI field in the TLS ClientHello, sent before encryption of the certificate exchange, and therefore visible to middleboxes.
5. Independent test — one that shares nothing with the last — a measurement whose failure modes do not overlap with the previous one.
6. Address family — the kind of address in use — IPv4 or IPv6, selectable in most tools with -4 and -6.

29.7 Ping, honestly#

PLAIN29.7.1 in simple words#

1. Ping sends a tiny message that means “are you there”, and waits for a message back that means “yes”.
2. That is the whole of it. It does not fetch a page. It does not check a service. It does not know what a website is.
3. It is not part of the web, and it is not part of the connection machinery that carries your data. It is a separate, simpler kind of message.
4. Because it is separate, a machine can answer ping while its website is broken, and can refuse ping while its website works perfectly.
5. Both of those situations are completely normal. Neither is a bug.
6. Ping is the most over-trusted tool in networking. People treat “it pings” as proof that a service works, and “it does not ping” as proof of a fault.
7. Neither reading is safe. Ping tells you about one particular kind of message reaching one particular machine and coming back.
8. It is still useful. It is fast, it needs no name lookup if you use a number, and it measures how long the round trip takes.
9. Use it to measure, not to conclude.

PLAIN29.7.2 a picture in your head#

1. Ping is shouting “hello” at a building and listening for “hello” back.
2. If you hear it, someone is inside and the air between you carries sound.
3. If you hear nothing, there are three possibilities and you cannot tell them apart: nobody is in, somebody is in but has been told not to shout back, or the wind ate your shout.
4. And even a cheerful “hello” back tells you nothing about whether the shop inside is open, whether it has your item, or whether the till works.
5. The shout and the shopping are different activities.

Where this comparison breaks: your shout travels in a straight line, but a ping travels through a dozen machines that each decide whether to pass it on, and several of them treat it as the least important thing they handle that second.

PLAIN29.7.3 a worked example#

1. Here is real output from a Mac, with every field named.

$ ping -c 4 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=57 time=8.412 ms
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=9.021 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=57 time=8.703 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=57 time=9.633 ms

--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 8.412/8.942/9.633/0.441 ms

2. 56 data bytes is what you sent. Add the 8-byte header and you get the 64 bytes that came back.
3. icmp_seq is a counter so you can tell which reply belongs to which request, and see if any went missing. On macOS it starts at 0. On Linux it starts at 1.
4. ttl=57 is the remaining life of the reply packet. Every packet starts with a number and every machine that forwards it subtracts one. When it hits zero the packet is thrown away, which is what stops packets circling forever.
5. You can read hop count from it. Most systems start replies at 64. Seeing 57 means about 7 machines forwarded it on the way back.
6. time=8.412 ms is the round trip: out and back, in thousandths of a second. It is not the one-way time and you cannot halve it reliably, because the two directions may take different routes.
7. 0.0% packet loss is the fraction that never came back.
8. The last line is the spread. A small difference between minimum and maximum means a steady path. A large one means the path is congested or wireless is struggling.

PLAIN29.7.4 what is really happening inside#

1. Ping works but the site does not. Why this is normal.
2. The machine answering ping may not be the machine running the site. Large services put a shared address at the front, and the thing that answers a simple “are you there” is not the thing that serves pages.
3. The site’s software may have crashed while the machine itself is fine.
4. The port you need may be closed or filtered while the simple message type is allowed.
5. Ping fails but the site works. Why this is also normal.
6. Very many networks are configured to ignore this message type, on purpose, to reduce noise and to make scanning harder.
7. Routers treat it as low-priority work handled by their slow general-purpose processor, while real traffic goes through fast dedicated hardware. Under load, the simple message is dropped first.
8. Some large cloud platforms block it by default. On Amazon Web Services, a new security group does not permit it unless you add a rule. Traffic through several kinds of cloud load balancer does not answer it at all.
9. The practical rule: use ping to measure a path you already know works, and never use its failure alone as evidence.
10. And how to read loss. One or two percent on a wireless link is common and usually harmless for web browsing. Ten percent will make everything feel broken. A hundred percent means either a real break or a policy of silence, and you cannot tell which from ping.
11. The most misleading case: loss shown at a middle hop of a trace but not at the end. That is almost always the middle router deprioritizing replies to itself, not loss on the path. If the final destination shows no loss, the path has no loss.

TECHNICAL29.7.5 the engineer’s version#

1. Ping uses the Internet Control Message Protocol, specified by Jon Postel in RFC 792 in September 1981. Echo Request is type 8, code 0. Echo Reply is type 0, code 0.
2. RFC 1122, the host requirements document from October 1989, states that every host must implement an ICMP Echo server. Blocking it is therefore a deliberate deviation from the standard, and a very common one.
3. Mike Muuss wrote the original ping in December 1983 at the United States Army Ballistic Research Laboratory, naming it after sonar.
4. The default payload is 56 bytes, making a 64-byte ICMP message and an 84-byte IPv4 packet including the 20-byte IP header.
5. The first 8 bytes of payload on BSD systems carry a timestamp, which is how the round-trip time is computed without keeping state.
6. Initial TTL values by system, useful for inferring hop counts:

7. ICMP messages you should recognize when they do appear:

8. Type 3 code 13 is a filter announcing itself. Type 3 code 4 is a path maximum transmission unit problem and causes connections that establish and then stall on the first large transfer.
9. Type 11 code 0 is the message traceroute depends on entirely.
10. Rate limiting is standard practice. Cisco IOS by default limits generation of ICMP unreachables to one every 500 milliseconds. Linux uses net.ipv4.icmp_ratelimit, default 1000 milliseconds, with a bitmask controlling which types are limited.
11. For IPv6 the message set is ICMPv6, RFC 4443 from March 2006, and filtering it wholesale breaks path MTU discovery and neighbour discovery. RFC 4890, May 2007, gives explicit guidance on which ICMPv6 types must be permitted.
12. Ping proves ICMP reachability of an address. It does not prove TCP or UDP reachability, service health, or that the responder is the service host, because anycast and load balancers separate the two.

WORDS29.7.6 remember these#

1. ICMP — the network’s own message system — Internet Control Message Protocol, RFC 792, carrying errors and diagnostics rather than user data.
2. Echo request and reply — “are you there” and “yes” — ICMP types 8 and 0, what ping sends and receives.
3. TTL — how many machines a packet may pass — an 8-bit hop counter decremented by each router, discarded at zero, called hop limit in IPv6.
4. Round-trip time — out and back, in milliseconds — measured from a timestamp embedded in the echo payload.
5. Jitter — how much the timing wobbles — variation in round-trip time, shown as the standard deviation in ping’s summary line.
6. Control plane policing — routers protecting their own brain — rate limits on traffic addressed to the router itself, the reason middle hops show loss.

29.8 The commands in full, for macOS#

PLAIN29.8.1 in simple words#

1. A command is a word you type into the Terminal application, sometimes followed by short options that begin with a dash.
2. Those options are called flags. Each flag changes one small thing about how the command behaves.
3. You do not need to memorize them. You need to know which command answers which question, and about three flags for each.
4. The rest of this section is a reference. Read it once, then come back to it.
5. Everything here is for macOS. Where Linux differs, it says so, because the differences are exactly the kind that waste an afternoon.

PLAIN29.8.2 a picture in your head#

1. Think of a mechanic’s toolbox with a dozen tools, not one big machine.
2. A torch shows you whether a thing is there. A meter measures. A stethoscope listens without touching anything.
3. Nobody uses a torch to measure voltage. Each tool answers one question.
4. ping is the torch. curl is the meter. tcpdump is the stethoscope.
5. Picking up the right one first is most of the skill.

Where this comparison breaks: mechanics’ tools do not change what they are measuring. Some network tools do. A packet capture can slow a machine, and a port scan can trip a security system into blocking you.

PLAIN29.8.3 a worked example#

1. One question, five tools, five different answers.

ping -c 3 20.207.73.82        # does it answer hello
traceroute -n 20.207.73.82    # how far do probes get
nc -vz -G 5 20.207.73.82 443  # does the port open
curl -sSI https://github.com  # does the site answer
sudo tcpdump -n host 20.207.73.82   # what is on the wire

2. In the reader’s case, the fourth and third both fail with silence, the second stops at hop 12, and the fifth would show outgoing SYN packets with nothing coming back at all.
3. That last observation, from the packet capture, is the one that converts “it does not work” into “we send, they never reply”.

PLAIN29.8.4 what is really happening inside#

1. These tools split into four groups, and knowing the group tells you what a result is worth.
2. Group one asks the operating system what it thinks: ifconfig, networksetup, scutil, netstat, route, arp, lsof. They never send a packet. They report configuration and state.
3. Group two sends probes and interprets replies: ping, traceroute, mtr, nmap.
4. Group three does real work with a real protocol: dig, nc, curl.
5. Group four watches without taking part: tcpdump.
6. Group one can be wrong about reality. It shows intention, not outcome.
7. Group four cannot be wrong about what left your machine, which is why it settles arguments.
8. When groups three and four disagree with group one, believe four, then three, then two, then one.

TECHNICAL29.8.5 the engineer’s version#

1. The command table. One line each on what the command proves.

2. ping flags that matter on macOS.

3. Linux difference, and it bites: on Linux -t sets the TTL, not a timeout; the deadline flag is -w in seconds, and -W is a per-reply timeout in seconds rather than milliseconds. Intervals below 0.002 seconds require superuser rights on macOS.
4. traceroute flags on macOS: -I uses ICMP echo instead of UDP; -P proto selects UDP, TCP, GRE or ICMP; -p port sets the base port, default 33434; -n prints numbers only; -q n sets probes per hop, default 3; -w s sets the wait, default 5 seconds; -m n sets maximum hops, defaulting to the system TTL of 64; -f n sets the starting hop.
5. Linux difference: the common Linux traceroute takes -T for TCP and -U for UDP, defaults to 30 hops, and ships alongside tracepath.
6. dig options: +short prints only the answer; +trace walks the hierarchy from the root itself; @server queries a named server directly; -x addr does a reverse lookup; +tcp forces TCP instead of UDP, which is how you test large answers and firewalls that only allow UDP port 53.
7. nc flags: -v reports the outcome, -z connects and sends no data, -w s sets an idle timeout, -G s sets the TCP connection timeout on macOS, -u uses UDP, -n skips lookups, -4 and -6 force a family.
8. UDP with nc -uvz is close to meaningless as a test, because a silent UDP port and an open one look identical. Only an ICMP port-unreachable reply distinguishes them, and that is often filtered.
9. curl flags: -v shows the connection steps; -I sends HEAD; -s silences the meter; -S keeps errors visible; --resolve host:port:addr pins an address without DNS; --connect-timeout s bounds the TCP phase; --max-time s bounds everything; -w fmt prints timing fields; -4 and -6 force an address family.
10. curl exit codes worth recognizing:

11. Code 6 is a rung 4 fault. Code 7 and 28 are rung 5. Codes 35 and 60 are rung 6. The exit code alone places the fault on the ladder.
12. Timing breakdown in one command:

curl -o /dev/null -sS -w \
'dns %{time_namelookup}\ntcp %{time_connect}\n\
tls %{time_appconnect}\nttfb %{time_starttransfer}\n\
total %{time_total}\nip %{remote_ip}\n' \
https://github.com

13. Each field is cumulative from the start, so subtract to get phase durations. A large gap between tcp and tls is a TLS problem; a large gap between tls and ttfb is a server-side problem.
14. macOS-specific state commands: scutil --dns shows the ordered resolver list including per-domain entries installed by VPN software; ipconfig getpacket en0 prints the full DHCP reply with lease time, router and domain_name_server; route -n get default names the winning interface; networksetup -listallhardwareports maps en0 to Wi-Fi or Ethernet, which differs between Mac models.
15. tcpdump needs sudo. Useful invocations:

sudo tcpdump -i en0 -n host 20.207.73.82
sudo tcpdump -i en0 -n port 443 and host 20.207.73.82
sudo tcpdump -i en0 -n -w /tmp/gh.pcap host 20.207.73.82

16. -i picks the interface, -n stops name lookups, host and port are filter expressions in the Berkeley Packet Filter language described by Steven McCanne and Van Jacobson at the USENIX conference in winter 1993, and -w writes a capture file you can send to somebody.
17. mtr and nmap are not shipped with macOS. Install with brew install mtr nmap. mtr needs sudo for ICMP mode; mtr --tcp --port 443 github.com probes with TCP instead. nmap -Pn -p 443 --reason github.com reports why it reached its conclusion, which is the single most useful nmap flag for diagnosis.
18. Linux equivalents, briefly: ip addr for ifconfig, ip route for netstat -rn, ip neigh for arp -a, ip route get for route get, ss -tunap for lsof -i, and resolvectl status for scutil --dns on systems using systemd-resolved.

WORDS29.8.6 remember these#

1. Flag — a short option after a command — a switch, usually one dash and a letter, that alters one behaviour.
2. HEAD request — headers only, no body — the HTTP method curl -I sends.
3. Exit code — the number a command leaves behind — readable with echo $?, and for curl it names the failing layer.
4. Berkeley Packet Filter — the language for choosing packets — the expression syntax used by tcpdump, from a 1993 USENIX paper.
5. Homebrew — the usual way to install extra tools on a Mac — a package manager providing mtr, nmap, bind and much else.
6. Reverse lookup — number to name — dig -x or host, using the in-addr.arpa tree, and frequently absent or wrong.

29.9 The decision tree for “I cannot reach a site”#

PLAIN29.9.1 in simple words#

1. A decision tree is a list of steps where each answer sends you to a different next step.
2. The point is that you stop deciding what to do next. The tree decides.
3. That matters because fault finding goes wrong when you are annoyed and start guessing.
4. Follow it in order. Do not skip ahead because you have a hunch.
5. Every branch ends either in a cause, or in a clear instruction about who to contact and what to tell them.

PLAIN29.9.2 a picture in your head#

1. It is the flowchart on the wall of a repair shop.
2. Junior staff follow it and get the same answer a senior person would.
3. The chart is not clever. It is just written down, so nobody forgets step four when they are in a hurry.
4. Your own memory under pressure is the unreliable part, not your knowledge.

Where this comparison breaks: a repair chart assumes one machine with fixed parts. A network path is shared with strangers and changes shape between two steps, so a tree can send you down a branch that was correct thirty seconds ago and is wrong now. Re-run the step that surprised you before you trust it.

PLAIN29.9.3 a worked example#

1. The procedure, written to be followed without thinking.

 1 Load a different site (example.com).
     works -> go to 2
     fails -> go to 10
 2 dig +short github.com
     gives an address -> go to 3
     fails -> go to 12
 3 nc -vz -G 5 <that address> 443
     connects -> go to 4
     refused  -> service or port issue, go to 8
     silence  -> go to 5
 4 curl -sSI https://github.com
     works -> the fault was the browser or a cache
     TLS error -> certificate or clock, go to 9
 5 Test from mobile data or another network.
     works there -> path fault, go to 6
     fails there too -> go to 8
 6 Test another device on your own Wi-Fi.
     also fails -> not your Mac, go to 7
     works -> fault is on your Mac, check utun and firewall
 7 traceroute -n and TCP traceroute to port 443.
     record where it stops, then go to 11
 8 Check the service status page and an outage tracker.
     confirmed outage -> stop, wait, nothing to fix
     nothing reported -> go to 11
 9 Check date and time, then the certificate chain.
10 Restart Wi-Fi, then check rung 1 and rung 2 commands.
     still fails -> local network or router fault
11 Log the fault for an hour with the script in 29.12.
     then contact the ISP with the evidence in 29.10
12 dig @1.1.1.1 and dig @8.8.8.8 for the same name.
     one works -> your configured resolver is the fault
     both fail -> port 53 is blocked or routing is broken

PLAIN29.9.4 what is really happening inside#

1. Step 1 is first because it has the highest eliminating power for two seconds of effort. It clears your machine, your Wi-Fi, your router and your provider’s access network in one go.
2. Step 2 is second because naming failures produce confusing symptoms everywhere else, so you remove them early.
3. Step 3 is the bisection point of the whole tree. Above it, everything is application. Below it, everything is network.
4. Step 3 has three outcomes, not two, and the difference between refused and silent is the single most valuable distinction in the tree.
5. Step 5 exists to close the “is it down” question before you spend an hour on your own network.
6. Steps 10 and 12 are the two dead ends that you can actually fix yourself.
7. Step 11 is the honest ending. Sometimes the correct output of a diagnosis is a well-evidenced report to somebody with access you do not have.

TECHNICAL29.9.5 the engineer’s version#

1. Dead ends and the correct action at each.

2. Stop and contact the internet provider when all of the following hold: your local chain passes, another destination on the same network works, DNS resolves correctly, a connection to the target port times out with no reply, the same target succeeds from another provider, and the condition persists across at least twenty minutes of logging.
3. What to give them so it is not dismissed: your account or circuit identifier, your public address at the time from curl -s ifconfig.me or similar, the target address and port, a numeric traceroute, the exact curl or nc output including the timeout, a coordinated universal time stamp, and one sentence stating that the same target works from a different provider at the same moment.
4. Phrase the request as a specific technical question. “Please check for a routing or filtering issue between AS-of-yours and 20.207.73.82 on TCP 443” gets escalated. “The internet is not working” gets a modem reboot script.
5. Ask them explicitly to check the return path towards your prefix, not only the forward path. Return-path faults are invisible from your side and are a common cause of exactly this symptom.
6. If they claim it is a customer-side issue, the counter-evidence is the working test to a different destination on the same line, which proves your line carries traffic.

WORDS29.9.6 remember these#

1. Decision tree — a written list of if-this-then-that steps — a diagnostic procedure with defined branch conditions and terminal states.
2. Dead end — a branch where you personally can do no more — a terminal state whose action is escalation rather than repair.
3. Escalation — passing the fault to whoever can act — a handover accompanied by evidence sufficient to reproduce the finding.
4. Circuit identifier — the number your provider knows you by — the account or service identifier that lets a technician find your line.
5. Prefix — a block of addresses — a range written as an address plus a slash and a length, such as 20.192.0.0/10.

29.10 Writing up a fault so someone else can act#

PLAIN29.10.1 in simple words#

1. A report is not a complaint. It is a set of instructions for reproducing what you saw.
2. Four things make a report work: what you did, what you expected, what happened, and what you have already ruled out.
3. The last one is the one people leave out, and it is the one that saves the most time.
4. “The internet is broken” gives the reader nothing to check, so they send you the standard script: restart the router.
5. A traceroute plus a command output gives them something to paste into their own tools, so it goes to somebody technical.
6. Keep every line copy-and-pasteable. Never describe output in your own words when you could paste it.

PLAIN29.10.2 a picture in your head#

1. Two people go to a mechanic.
2. The first says “it makes a funny noise sometimes”.
3. The second says “at 40 kilometres per hour, in third gear, on a left turn only, there is a knock from the front right, and it started on Tuesday”.
4. The second person gets a fix. The first gets a bill for an hour of guessing.
5. The information in the second sentence is exactly the information a good fault report contains: conditions, location, timing, and what does not trigger it.

Where this comparison breaks: a mechanic can drive the car. Your reader cannot sit at your desk, so anything you did not write down does not exist.

PLAIN29.10.3 a worked example#

1. A template you can fill in.

Subject: TCP 443 to 20.207.73.82 times out from my line

WHAT I DID
  From macOS on my home connection, 13 Aug 2026, 09:40 UTC:
  curl -v --max-time 15 https://github.com

WHAT I EXPECTED
  A TCP connection, then an HTTP 200 response.

WHAT HAPPENED
  * Trying 20.207.73.82:443...
  curl: (28) Failed to connect after 15001 ms
  No RST, no ICMP unreachable. Total silence.

WHAT I HAVE RULED OUT
  - DNS: dig +short github.com -> 20.207.73.82, in 12 ms
  - Local network: ping 192.168.0.1 -> 0% loss
  - Internet access: https://example.com loads normally
  - Service down: same URL loads instantly on mobile data
  - My device: same failure from a second device on this Wi-Fi

EVIDENCE
  traceroute -n reaches 104.44.20.42 (hop 12) then * * *
  My public address at the time: <address>
  Circuit / account: <identifier>

WHAT I AM ASKING
  Please check for a routing or filtering problem towards
  20.207.73.82 on TCP 443, including the return path to my
  prefix. Happy to run any command you name.

PLAIN29.10.4 what is really happening inside#

1. Why this format works is not politeness. It is about who can act.
2. A first-line support agent has a script and a set of buttons. Your report must contain something their script recognizes, or something so specific that the script does not fit and they must pass it on.
3. A network engineer needs three things to look at their own tools: a source, a destination, and a time. Give all three or they cannot look.
4. The “ruled out” list is doing a specific job. It pre-empts the four questions they would otherwise ask you, each costing a day of email.
5. Pasting exact output matters because your summary may hide the detail that identifies the fault. “It did not work” and “no RST, no ICMP” are wildly different findings.
6. State facts and separate them from your theory. Put any theory in one line at the end, marked as a guess. Reports that assert a cause get argued with; reports that present evidence get investigated.

TECHNICAL29.10.5 the engineer’s version#

1. Minimum evidence set for a reachability report:

2. Capture the public address at the time of the fault, since it may change: curl -s https://ifconfig.me or dig +short myip.opendns.com @resolver1.opendns.com.
3. Always paste numeric output. traceroute -n avoids reverse lookups that might themselves fail and confuse the reader.
4. Include one control test that succeeded. Without it, the first response will be a line test rather than a routing investigation.
5. Where a packet capture is appropriate, attach a file made with sudo tcpdump -i en0 -n -w /tmp/gh.pcap host 20.207.73.82. Retransmitted SYN packets with no response are unambiguous proof of a silent drop and are the strongest artefact you can supply.
6. Use coordinated universal time throughout and say so. Provider logs are kept in it, and a mismatch of five and a half hours makes your evidence unsearchable.
7. Keep the theory to one labelled line, for example: “Hypothesis, unproven: asymmetric routing or a policy filter between your network and AS8075.”

WORDS29.10.6 remember these#

1. Reproduction steps — how to see it yourself — an exact command sequence that produces the reported behaviour.
2. Control test — the thing that worked — a comparison measurement proving the surrounding system is healthy.
3. Artefact — a file that carries evidence — a capture, a log or a saved output, attached rather than described.
4. First-line support — the scripted layer — the tier whose tools cannot see routing, whose job is to filter and escalate.
5. Coordinated universal time — the one clock everyone shares — the reference time zone used in logs, printed by date -u.

29.11 When it is not your problem#

PLAIN29.11.1 in simple words#

1. Sometimes the fault really is at the other end, or really is your provider, and the right action is to stop working and wait.
2. There are three places to look, and each one has a weakness you must know.
3. A status page is written by the company that owns the service.
4. An outage tracker counts complaints from members of the public.
5. Your provider’s own channels tell you about local work and known faults.
6. None of the three is proof. Together with your own tests, they are enough.

PLAIN29.11.2 a picture in your head#

1. A status page is the shop’s own sign in the window saying “we are open”.
2. An outage tracker is a crowd outside the shop, some of whom came on the wrong day and some of whom wanted a different shop.
3. Your own test is walking to the door and pulling the handle.
4. Only one of those three actually touches the door.

Where this comparison breaks: a shop sign is updated by someone who can see the shop. A status page is often updated by a process that requires a human decision and an internal agreement that an incident exists, which can lag the outage by half an hour or more.

PLAIN29.11.3 a worked example#

1. Three checks and what each is worth.

2. GitHub publishes a status page, hosted on Atlassian’s Statuspage product, which Atlassian acquired in 2016. It reports per-component state such as Git Operations, API Requests, Webhooks, Actions and Pages.
3. That component split matters. In the reader’s session plain git kept working while the web interface and the programming interface did not. A status page showing “Git Operations operational” and something else degraded would be consistent with that.
4. Downdetector, launched in 2012, plots reports per minute. A sharp spike across many providers means a service problem. A spike from one provider only means a provider problem.
5. If the tracker shows nothing and your cross-check from another network works, the problem is yours or your provider’s. That is the common case.

PLAIN29.11.4 what is really happening inside#

1. How to tell a regional problem from a personal one, in three moves.
2. Move one: another destination on the same line. Works means your line is fine and the fault is destination-specific.
3. Move two: the same destination from another provider. Works means the destination is fine and the fault is provider-specific or path-specific.
4. Move three: ask one other person on the same provider in the same city. If they fail too, it is regional and the provider will already have a ticket.
5. Combine them into a small truth table and the answer falls out without argument.

6. Two honest limits. A status page saying “all systems operational” never contradicts your own failed request, because the service may be fine and unreachable from you. And a tracker with no reports never proves an absence, because trackers under-report small providers and small countries.

TECHNICAL29.11.5 the engineer’s version#

1. Verification independent of anybody’s page:

curl -sS -o /dev/null -w '%{http_code} %{remote_ip}\n' \
     --max-time 10 https://www.githubstatus.com
dig +short api.github.com
curl -sSI --max-time 10 https://api.github.com | head -3

2. Query the status host from your own machine and note the resolved address; if the status page itself is unreachable from your network, that is another data point about your path, not about the service.
3. For provider-wide issues, check whether the provider’s own portal and speed-test host respond, and whether a public looking glass in your country can reach your prefix.
4. Autonomous system level tools separate provider from path: whois -h whois.cymru.com " -v 20.207.73.82" returns the announcing autonomous system, which for Microsoft is AS8075. Doing the same for your own public address names your provider’s autonomous system, and that pair is what an engineer needs to investigate.
5. Outage aggregation has known limits: crowd trackers measure complaints, not reachability, so they lag by minutes, over-represent consumer services and large providers, and cannot see business-hours-only faults.
6. Where experts disagree: some operators argue public status pages should be automated from probes, others that human confirmation prevents false alarms and customer panic. The practical consequence for you is that a page can be green during a real outage, so never treat it as a veto over your own measurements.

WORDS29.11.6 remember these#

1. Status page — the operator’s own report — a published per-component state, updated by an internal incident process.
2. Outage tracker — a count of public complaints — a crowd-sourced signal that correlates with outages but does not measure them.
3. Regional fault — broken for a city or a provider — a failure whose blast radius is a geography or an autonomous system.
4. Truth table — every combination of results — a grid mapping test outcomes to conclusions, removing the need for judgement.
5. Whois — who owns this number — a directory service naming the holder of an address block or autonomous system number.

29.12 Intermittent faults, the hardest kind#

PLAIN29.12.1 in simple words#

1. An intermittent fault is one that comes and goes.
2. It defeats every test in this chapter, because every test is one shot.
3. You run a command, it passes, and you conclude the problem is gone. Ten minutes later it is back.
4. The only cure is to stop taking single measurements and start recording, continuously, with the time written next to every result.
5. Then the pattern appears: every evening, every hour, only under load, only after the router has been up for two days.
6. A pattern is worth more than a hundred one-shot tests, because it points at a mechanism.

PLAIN29.12.2 a picture in your head#

1. A doctor cannot find your headache in a ten-minute appointment.
2. So they ask you to keep a diary for two weeks: time, what you ate, how much you slept.
3. The diary finds what the appointment could not, because the fault is a pattern in time, not a state in the room.
4. Network logging is that diary, written automatically once a minute.

Where this comparison breaks: a diary records your own experience, which is continuous. A one-minute network probe samples, so a fault lasting twenty seconds can hide completely between two samples. Sample faster than the fault.

PLAIN29.12.3 a worked example#

1. A logging script. Save it as netlog.sh, then run chmod +x netlog.sh and ./netlog.sh.

#!/bin/bash
TARGET=github.com
IP=20.207.73.82
LOG=$HOME/netlog.csv
echo "time,dns_ms,tcp,http,rtt_ms" >> "$LOG"
while true; do
  T=$(date -u +%FT%TZ)
  D=$(dig +short "$TARGET" | head -1)
  C=$(nc -z -G 5 "$IP" 443 >/dev/null 2>&1 && echo ok || echo fail)
  H=$(curl -sS -o /dev/null --max-time 10 \
      -w '%{http_code}' "https://$TARGET" 2>/dev/null)
  R=$(ping -c 1 -W 1000 1.1.1.1 2>/dev/null \
      | awk -F'time=' '/time=/{print $2+0}')
  echo "$T,$D,$C,${H:-none},${R:-loss}" >> "$LOG"
  sleep 60
done

2. Line by line. #!/bin/bash tells the system which interpreter runs the file. TARGET, IP and LOG hold the values you may want to change.
3. echo "time,..." >> "$LOG" writes a header row and appends rather than overwriting, because >> adds to the end.
4. while true; do ... done repeats forever until you press control and C.
5. T=$(date -u +%FT%TZ) records the moment in coordinated universal time in a sortable format such as 2026-08-13T09:40:12Z.
6. D=$(dig +short "$TARGET" | head -1) records what the name resolved to, keeping only the first answer, so you can see if the address changed.
7. The nc -z -G 5 line tests the connection with a five-second limit and records ok or fail. Sending output to /dev/null throws away the chatter and keeps only the success or failure.
8. The curl line records the HTTP status code, or nothing if it failed, and ${H:-none} substitutes the word none when the value is empty.
9. The ping line takes one measurement to a known-good address and awk extracts the number after time=, giving you a baseline. ${R:-loss} writes loss when there was no reply.
10. sleep 60 waits a minute. Lower it to 10 for a fast-moving fault, but be aware that frequent probes can look like scanning to some networks.
11. The result is a comma-separated file you can open in a spreadsheet, sort by time, and plot.

PLAIN29.12.4 what is really happening inside#

1. Why a log beats a test. A test answers “is it broken now”. A log answers “when is it broken, and what else is true then”.
2. Look for three patterns in the file.
3. Time of day. Failures clustered in the evening point at congestion, since that is when a shared line is busiest.
4. Correlation with your own activity. Failures only during large downloads point at your router’s connection table filling up.
5. Uptime correlation. Failures starting a fixed time after every router restart point at a memory or table exhaustion problem in the router.
6. Also watch the resolved address column. If the address changes and the failures follow one particular address, the fault is specific to one front end at the destination, which is a completely different report to write.
7. Continuous path monitoring is the companion tool. mtr sends probes repeatedly to every hop and shows loss per hop over time, which catches a flapping link that a single traceroute misses.

TECHNICAL29.12.5 the engineer’s version#

1. Continuous path measurement:

sudo mtr -n --report --report-cycles 100 20.207.73.82
sudo mtr -n --tcp --port 443 --report-cycles 100 github.com

2. --report runs a fixed number of cycles and prints a summary table. --report-cycles 100 sets that number. --tcp --port 443 probes with TCP SYN packets to the port you actually care about, which is far more representative than ICMP when a filter is port-specific.
3. Read mtr loss columns from the bottom up. Loss at an intermediate hop that does not persist to the final hop is control-plane rate limiting, not path loss. Only loss that continues to the destination is real.
4. mtr was written by Matt Kimball in 1997 as Matt’s Traceroute and has been maintained by Roger Wolff since 1998 under the name My Traceroute.
5. Sampling theory applies. To characterize a fault of duration D with confidence, sample at an interval well below D. A 60-second interval will miss most 20-second events and will under-report their frequency.
6. Prefer recording raw results rather than pass or fail. Store connect time in milliseconds, not a boolean, because degradation precedes failure and only the numeric series shows the trend.
7. To run the logger unattended on macOS, use a launchd property list in ~/Library/LaunchAgents rather than cron, because macOS may not run cron reliably under modern power management. On Linux, a systemd timer or a cron entry is the normal choice.
8. Correlate with local link quality on macOS using sudo wdutil info for Wi-Fi signal and noise figures. Apple deprecated the older airport command-line tool during macOS 14 and it stopped working in a 14.x update in 2024; treat that detail as version-dependent.
9. Apple added networkQuality in macOS 12 Monterey in 2021, which measures throughput and responsiveness under load in round trips per minute. Running it during a suspected congestion window is a useful second signal.

WORDS29.12.6 remember these#

1. Intermittent fault — comes and goes — a failure whose probability of observation depends on when you sample.
2. Sampling interval — how often you measure — the gap between probes, which must be shorter than the shortest fault you hope to catch.
3. Flapping — a link going up and down repeatedly — instability that a single traceroute cannot detect but a continuous probe will.
4. Correlation — two things happening together — evidence of a shared cause, not proof of one causing the other.
5. Comma-separated values — a plain text table — one record per line with fields separated by commas, openable by any spreadsheet.
6. Launchd — the macOS way to run things on a schedule — Apple’s service manager, configured with property list files, replacing cron in practice.

29.13 What to do next in this specific situation#

PLAIN29.13.1 in simple words#

1. The reader’s fault has a shape: naming works, the connection to one destination on port 443 dies in silence, and the same destination answers from another network.
2. Four hypotheses survive that shape, and no test done so far separates them.
3. So the next steps are chosen for one reason only: each one splits those four hypotheses further.
4. The order below is by cost. Cheap and informative first.
5. Do not skip to contacting the provider. Do it after step 6, with evidence, or you will be sent round the reboot loop.

PLAIN29.13.2 a picture in your head#

1. You know the parcel gets as far as one particular sorting office and then vanishes.
2. Next you would send a second parcel to a different address served by the same office, and a third to the same address by a different route.
3. If the second arrives and the third does not, the office is fine and the address is the problem. If both fail, the office is the problem.
4. That is exactly what the steps below do, with addresses and ports.

Where this comparison breaks: you can watch a parcel leave. Here you must capture your own outgoing packets to know they left at all, which is why a packet capture appears in the list.

PLAIN29.13.3 a worked example#

1. The ordered list.
2. Step 1. Confirm the failure is still happening and is not a memory. Run curl -v --max-time 20 https://github.com and keep the output.
3. Step 2. Test the same destination on a different port. Try 22 and 80. If one port opens and 443 does not, the destination is reachable and the fault is port-specific, which is a very different report.
4. Step 3. Test another Microsoft-hosted address, so that most of the path is the same but the machine at the end is not. If that works, the fault is specific to the one address rather than to the whole path.
5. Step 4. Run a TCP traceroute to port 443, which follows the same kind of packet as your real connection instead of the different kind ordinary traceroute uses.
6. Step 5. Capture packets while trying, to prove your machine really sent the connection request and really received nothing.
7. Step 6. If your router allows it, run a ping and a trace from the router itself. That removes your Mac and your Wi-Fi from the experiment entirely.
8. Step 7. Log for an hour with the script from the previous section, so you know whether it is constant or comes and goes.
9. Step 8. Contact the provider with the template from earlier in this chapter, and ask the specific questions listed below.

PLAIN29.13.4 what is really happening inside#

1. Why step 2 matters. A filter usually targets a port. If port 22 opens to the same host and 443 does not, you have proved the route works and a policy is selecting on port.
2. Why step 3 matters. It separates “this path is broken” from “this one machine is not accepting me”. Both look identical when you only test one address.
3. Why step 4 matters. Ordinary traceroute on a Mac sends a different kind of packet to unusual port numbers. Some networks treat that differently from real web traffic, which is why a trace can die where your connection would have succeeded, or the reverse.
4. Why step 5 matters. Until you have seen the outgoing packets, “nothing happened” also covers “my own machine never sent anything”, which a local firewall or a tunnel interface could cause.
5. Why step 6 matters. If the router fails the same way, every question about your Mac is closed, and the provider cannot blame your device.
6. Why step 7 matters. Providers respond very differently to “it failed once” and “it failed for 47 of 60 consecutive minute-by-minute probes”.

TECHNICAL29.13.5 the engineer’s version#

1. The commands, in order.

# 2. different ports, same host
nc -vz -G 5 github.com 22
nc -vz -G 5 github.com 80
nc -vz -G 5 github.com 443

# 3. different address, same network path
OTHER=$(dig +short www.microsoft.com | tail -1)
nc -vz -G 5 "$OTHER" 443
curl -sSI --max-time 10 https://www.microsoft.com

# 4. TCP traceroute to the port that matters
sudo traceroute -n -P TCP -p 443 20.207.73.82
sudo nmap -Pn -p 443 --traceroute --reason 20.207.73.82
sudo mtr -n --tcp --port 443 --report-cycles 50 20.207.73.82

# 5. prove what left the machine
sudo tcpdump -i en0 -n host 20.207.73.82 and port 443

2. On the TCP traceroute: macOS traceroute -P TCP exists but its handling of the final hop is unreliable, because a SYN-ACK or RST from the destination is not always recognized as arrival. Prefer nmap --traceroute or mtr --tcp and treat the macOS built-in as a second opinion.
3. Expected capture signature for a silent drop: repeated outgoing SYN packets to 20.207.73.82:443 with increasing gaps of roughly 1, 2, 4, 8 and 16 seconds under the retransmission rules of RFC 6298, and zero inbound packets from that address. That is the artefact to attach to a ticket.
4. Compare against a working target in the same capture window by adding a second curl to a host that works; a capture containing both a successful and a failed handshake side by side is very hard to dismiss.
5. From the router, if it offers a diagnostics page or a shell, run its own ping and traceroute to 20.207.73.82. Many consumer routers expose exactly these two tools. A failure there removes the Mac, the utun interfaces and the local firewall from consideration in one step.
6. Questions to put to the provider, in this wording:

7. Also ask whether your line is behind carrier-grade network address translation and, if so, which public address you were mapped to at the timestamp you supply. Without that mapping, the far end cannot look you up at all.
8. If the provider confirms nothing and the fault persists, the remaining practical mitigations are a different resolver that returns a different front-end address, an IPv6 path if one can be obtained, or a tunnel that moves your egress to another network. Note that these are workarounds that avoid the fault; none of them fixes it or identifies it.

WORDS29.13.6 remember these#

1. TCP traceroute — a trace that uses real connection packets — probes carrying TCP SYN to a chosen port, so filters that select on port are visible.
2. Egress — where your traffic leaves for the internet — the point and address from which your packets enter the wider network.
3. Capture signature — the recognizable shape in a packet capture — here, retransmitted SYNs with no inbound reply.
4. Carrier-grade NAT — many customers sharing one public address — provider-side address translation, which hides which subscriber a session belonged to.
5. Workaround — avoids the problem without fixing it — a change that restores service by routing around the fault, leaving the cause in place.

29.98 Common wrong ideas#

1. Wrong: the site did not load, so the site is down. Right: you proved only that you could not reach it. “Down” is a claim about everybody, and one success anywhere refutes it in ten seconds.
2. Wrong: it worked on mobile data, so my Wi-Fi is the problem. Right: switching to mobile data changes about thirteen variables at once. It proves the destination is alive and nothing about which variable mattered.
3. Wrong: ping works, so the service is fine. Right: ping tests one message type to one address. The service uses a different protocol on a specific port and may be down while ping answers.
4. Wrong: ping fails, so the host is down. Right: very many networks drop ICMP on purpose, and routers deprioritize it. Confirm with a TCP test before concluding anything.
5. Wrong: the stars at the end of my traceroute show where the fault is. Right: silence at the end of a trace is normal. Routers commonly rate-limit or suppress the replies traceroute depends on.
6. Wrong: loss shown at hop 6 means the path is losing packets. Right: read the last hop. Intermediate loss that does not persist to the destination is control-plane rate limiting on that router.
7. Wrong: DNS resolved, so DNS is not involved. Right: DNS resolved to one address out of many possible ones. A different resolver may hand you a different front end that works.
8. Wrong: the status page says all systems operational, so it is my fault. Right: a status page reports service health, not your reachability, and it updates only after a human decides an incident exists.
9. Wrong: a timeout and a connection refused are both just failures. Right: refused means a machine answered and said no. A timeout means nothing answered at all. They point at completely different causes.
10. Wrong: I tested it three times, so I have three pieces of evidence. Right: three identical tests from one vantage point are one observation repeated. Evidence comes from changing a variable, not from repetition.

29.99 Chapter summary in 20 lines#

1. Reachability is a property of a pair, a source and a destination, never of a destination alone. That one sentence prevents most wasted hours.
2. “The site is down” is a universal claim; “I cannot reach it” is about you. One success from anywhere refutes the first; your own failure proves the second.
3. The one-line test that separates them is a request to the same destination from a network that shares nothing with yours.
4. The reader ran that test with mobile data and the site loaded instantly, so GitHub was up and the fault was on the reader’s own path.
5. That switch changed at least thirteen things: provider, public address, autonomous system, forward path, return path, peering point, resolver, resolved address, address family, middleboxes, path MTU, device stack, time.
6. Only one of them needed to be the cause, so the test had huge power over one question and none at all over which difference mattered.
7. Good diagnosis is a controlled experiment: change one variable, state the hypothesis, and know the eliminating power of the test before running it.
8. The ladder orders the chain: interface, router, public address by number, name resolution, TCP port, full request, path discovery.
9. For every rung, know three things: what a pass proves, what a pass does not prove, and what a failure means. Most mistakes are over-reading a pass.
10. On the reader’s ladder, rungs 1 to 4 passed, rung 5 failed with silence, rung 6 never ran, and rung 7 stopped at hop 12 inside Microsoft’s Pune site.
11. DNS passing while TCP fails is a very informative pair, because the two used different destinations, protocols and ports; only path, address and port remain as candidates.
12. Silence is the least informative failure mode. No RST, no ICMP, nothing. Consistent with a silent drop on that path, not with the server being down.
13. The trace’s * * * from hop 13 proves nothing; ICMP time-exceeded is routinely rate-limited, as with the Cisco default of one per 500 ms.
14. Five cross-checks cover every component: other network, other device, other destination, remote shell, and a tunnel. Each eliminates a different family.
15. Ping is ICMP echo, RFC 792 from 1981, written by Mike Muuss in December
    1983. It is the most over-trusted tool in networking; use it to measure.
16. TTL in a ping reply lets you count hops; time is a round trip and cannot be halved reliably; loss must be read at the final hop only.
17. On macOS know these flags: ping -c -i -s -t -W, traceroute -n -q -w -m -P, dig +short +trace @server, nc -vz -G, curl -v -I -sS --resolve --max-time -w. On Linux, ping -t means TTL instead, and -W is seconds.
18. A fault report needs what you did, what you expected, what happened and what you ruled out, with exact output, a UTC timestamp and a control test.
19. Intermittent faults need logging, not testing: a minute-by-minute log, mtr in report mode, and correlation with time of day and load.
20. The reader’s remaining steps are TCP traceroute to port 443, a second Microsoft address, a test from the router itself, a packet capture showing retransmitted SYNs with no reply, and a precise question to the provider about the route and the return path towards their own prefix.