Traceroute Decoded - Reading Your Own Path

27.0 What this chapter gives you#

1. You will be able to explain why there is no “show me the route” command in the internet protocol, and why traceroute is a trick rather than a feature.
2. You will be able to describe the time-to-live field exactly: how many bits it has, what it counts, and what happens at each router.
3. You will be able to guess which operating system sent you a packet by looking at one number in it, and to say why that guess can be wrong.
4. You will be able to draw the full sequence of probes and replies that produces a traceroute, and say what each printed line really is.
5. You will be able to choose between the three main traceroute methods and say which one gets furthest through modern filtering, and why.
6. You will be able to explain why one hop shows several addresses, and point at the exact hops in the reader’s own trace where this happened.
7. You will be able to read a row of stars honestly, and to say out loud that stars at the end of a trace prove nothing on their own.
8. You will be able to read every hop of the reader’s real trace, including every router name, city code and interface name.
9. You will be able to say precisely what a traceroute cannot tell you, which is a longer and more important list than what it can.
10. You will be able to run a much better traceroute next time, with the exact flags, and to get a second opinion from the far end of the path.

27.1 Traceroute is a clever trick, not a feature#

PLAIN27.1.1 in simple words#

1. When you send data across the internet, it passes through many machines before it arrives.
2. Each of those machines is a router: a box whose only job is to take a packet in one door and push it out the right other door.
3. The internet protocol has no field that records the route. Packets do not carry a list of where they have been.
4. Routers do not report to you. A router does not know or care who you are. It looks at the destination, picks a door, and forgets you instantly.
5. Traceroute gets the answer anyway, by cheating.
6. It sends packets that are designed to fail. It sends them so they die at the first router, then the second, then the third.
7. Every time a packet dies, the router that killed it sends back a short complaint. That complaint has the router’s own address on it.
8. Nobody meant for this to be possible. It is a side effect of two safety rules that were written for completely different reasons.

PLAIN27.1.2 a picture in your head#

1. Imagine you want to know the route your letters take to a friend in another city, and the postal service will not tell you.
2. But there is a rule: if a letter has been handled too many times without being delivered, an office must destroy it and post you a note saying so.
3. The note is stamped with that office’s own name and town, because the rule also says the office must identify itself.
4. So you write a letter and mark it “destroy after one handling”. Back comes a note from your local sorting office. You now know step one.
5. Twenty letters later you have the whole route, and you never once asked anyone a question they were allowed to refuse.

Where this comparison breaks: a real postal route is stable, so twenty letters would follow the same path. On a network each of your twenty probes can legitimately take a different road. A traceroute is a sketch assembled from many separate journeys, not a photograph of one. That single fact explains most of the confusing things in this chapter.

PLAIN27.1.3 a worked example#

1. Here is the reader’s own trace, exactly as their machine printed it. We are going to spend most of this chapter on these twenty lines.

 1  192.168.0.1
 2  172.31.0.17
 3  137.97.29.249
 4  172.26.22.235
 5  172.16.18.33 / 172.26.14.75 / 172.26.22.234
 6  172.26.14.75 / 172.16.18.33
 7  ae66-0.del01-96cbe-1b.ntwk.msn.net (104.44.196.187)
 8  be23.rwa02.bom01.ntwk.msn.net (104.44.55.163)
    po22.rwa04.bom01.ntwk.msn.net (104.44.51.55)
 9  be1010.owr02.bom01.ntwk.msn.net (104.44.53.119)
    be1013.owr01.bom01.ntwk.msn.net (104.44.53.47)
10  be5.ibr02.pnq21.ntwk.msn.net (104.44.31.62)
11  ae104-0.rwa03.pnq21.ntwk.msn.net (104.44.20.52)
    be1.ibr01.pnq20.ntwk.msn.net (104.44.16.51)
12  ae106-0.rwa04.pnq20.ntwk.msn.net (104.44.20.42)
13-20  * * *   (no reply)

2. Twenty lines. Twelve of them answered. Eight of them are silence.
3. Every single line is a death notice. Something threw the reader’s packet away and sent back a note about it.
4. The router names on hops 7 to 12 were not sent by the routers either. Those came from a separate name lookup done afterwards by the reader’s own machine, on the addresses in brackets.
5. So even the names are a second-hand fact. We will come back to that, because it matters when a name is wrong or missing.

PLAIN27.1.4 what is really happening inside#

1. There were two serious attempts to build a real “record the route” feature into the internet, and both failed for good reasons.
2. The first is called the record route option. It is part of the original internet protocol from 1981. A packet can carry a small list, and each router writes its address into the list.
3. It does not work in practice. The space for options in the packet header is only 40 bytes, which holds at most nine addresses. Real paths are longer.
4. The second attempt was a proper traceroute protocol, published in 1993. It would have used one packet instead of dozens. Almost nobody implemented it.
5. So the working method is the trick. It relies on two rules that every router already obeys for its own reasons.
6. Rule one: every packet carries a counter, and every router must reduce that counter by one. When it hits zero the packet is destroyed. This exists to stop packets circling forever when routing goes wrong.
7. Rule two: when a router destroys a packet for that reason, it must send a short error message back to whoever sent it. This exists so the sender can find out its packets are dying.
8. Traceroute exploits both. It sets the counter deliberately low, so the packet dies on purpose at a chosen distance, and the error message that comes back carries the address of the killer.

TECHNICAL27.1.5 the engineer’s version#

1. The IPv4 header defined in RFC 791, September 1981, is 20 bytes without options. There is no field in it that accumulates a path.
2. The Record Route option is IP option number 7, defined in RFC 791. Options live in the space between byte 20 and byte 60 of the header, so the whole option area is at most 40 bytes.
3. Packets with IP options are process-switched or punted to the control plane on most modern hardware routers, rather than handled by the forwarding silicon. Many operators drop them outright as a denial-of-service defence.
4. RFC 1393, “Traceroute Using an IP Option”, by G. Malkin of Xylogics, January 1993, is an Experimental specification. It proposed a new IP option and a new ICMP message so that one packet could map a whole path.
5. The mechanism that actually works is emergent behaviour from two normative requirements. RFC 1812, “Requirements for IP Version 4 Routers”, edited by Fred Baker, June 1995, section 5.3.1, requires that a router “MUST decrement the TTL field of every datagram by exactly one”, and that when the result is zero the router MUST discard the datagram and MUST send an ICMP Time Exceeded message.
6. The complementary requirement is in RFC 792, “Internet Control Message Protocol”, by Jon Postel, September 1981, which defines that Time Exceeded message as type 11, code 0.
7. Traceroute itself was written by Van Jacobson in 1987, from a suggestion by Steve Deering. The manual page has said for decades: “Implemented by Van Jacobson from a suggestion by Steve Deering.” Guy Almes and Matt Mathis are credited with having had the same idea independently.
8. Van Jacobson worked in the Network Research Group at Lawrence Berkeley National Laboratory in California, which released traceroute along with tcpdump and other network tools.

9. Observe it with traceroute on macOS, Linux and BSD, tracert on Windows, tracepath on Linux, and mtr everywhere.

WORDS27.1.6 remember these#

1. Router — a box that passes packets along — a layer 3 forwarding device that makes a next-hop decision per packet from a routing table.
2. Traceroute — a tool that lists the machines on the way out — a utility that elicits ICMP Time Exceeded responses by manipulating the IP TTL field.
3. Probe — one test packet — a single datagram sent with a specific TTL and a specific identifier so its response can be matched to it.
4. Record Route option — an old idea for packets to log their own path — IP option 7 from RFC 791, limited to nine addresses by the 40-byte option area.
5. Emergent behaviour — a use nobody designed for — functionality arising from the interaction of unrelated normative requirements.

27.2 The time to live field, properly#

PLAIN27.2.1 in simple words#

1. Every packet on the internet carries a small number called the time to live, almost always shortened to TTL.
2. It is one byte. That means it can hold any whole number from 0 to 255.
3. The name is misleading. It sounds like a clock. It is not a clock.
4. It counts hops, not seconds. A hop is one router handling the packet.
5. Every router that forwards the packet subtracts one from it.
6. When the number reaches zero, the packet is thrown away. It goes no further, no matter how important it was.
7. This is a safety belt. Without it, a mistake in the routing tables could make a packet loop between two routers forever, and the network would slowly fill up with immortal garbage.
8. Different computers start the number at different values. That turns out to be a small accidental fingerprint.

PLAIN27.2.2 a picture in your head#

1. Think of a bus ticket with a row of boxes on it, and a number printed on the front saying how many boxes are left.
2. Every time you board a bus, the driver punches a box and reduces the number by one.
3. When the number reaches zero the driver refuses you and puts you off the bus, wherever you happen to be.
4. The ticket does not expire at a time of day. It expires after a count of journeys. A slow bus and a fast bus cost exactly the same: one punch.
5. Now imagine the rule says the driver who puts you off must post a postcard to whoever bought your ticket, saying which stop it happened at.

Where this comparison breaks: a router that holds a packet for a whole second still only subtracts one. The 1981 specification really did intend a clock, in seconds, and asked routers to subtract at least one per second of holding time. No router implemented the clock part, and in 1995 the rule was rewritten to say plainly: subtract exactly one, every time. The name is a fossil.

PLAIN27.2.3 a worked example#

1. The reader’s machine is a Mac. macOS starts outgoing packets with TTL 64.
2. Follow one packet from the reader’s flat toward hop 12 of their own trace.

Packet leaves the Mac with TTL = 64

  at 192.168.0.1      64 -> 63   forward
  at 172.31.0.17      63 -> 62   forward
  at 137.97.29.249    62 -> 61   forward
  at 172.26.22.235    61 -> 60   forward
  at hop 5 router     60 -> 59   forward
  at hop 6 router     59 -> 58   forward
  at 104.44.196.187   58 -> 57   forward   (Delhi)
  at 104.44.55.163    57 -> 56   forward   (Mumbai)
  at 104.44.53.119    56 -> 55   forward   (Mumbai)
  at 104.44.31.62     55 -> 54   forward   (Pune)
  at 104.44.20.52     54 -> 53   forward   (Pune)
  at 104.44.20.42     53 -> 52   forward   (Pune)

3. Now run the trick backwards. Suppose a reply comes back to you with a TTL of 52 in it. You guess it started at 64 and crossed 12 routers.

4. From that you can often guess what kind of machine sent it. A reply arriving with 118 was probably sent by Windows. One arriving with 52 was probably sent by something built on Unix, such as Linux or macOS.
5. This is a guess, not a measurement. Say so out loud when you use it.

PLAIN27.2.4 what is really happening inside#

1. The TTL sits at a fixed place in the packet header, so a router can find it without understanding anything else about the packet.
2. On arrival the router reads it. If it is already 0 or 1, the packet cannot be forwarded, because forwarding would require subtracting one and reaching zero or below.
3. In that case the router drops the packet and builds an error message to send back to the source address printed on the dead packet. Otherwise it subtracts one and forwards.
4. There is a hidden cost here. The header carries a checksum, a small number that proves the header was not corrupted in transit.
5. Change the TTL and the checksum is now wrong, so the router must fix the checksum too, on every packet, at full line rate.
6. Note carefully who does the subtracting. Only devices that forward the packet. A plain network switch does not touch the TTL, because it works at a lower level and is not a router. This is why a large office network with many switches still shows as one hop.

TECHNICAL27.2.5 the engineer’s version#

1. In IPv4 the TTL is an 8-bit unsigned field at byte offset 8 of the header, the ninth byte, immediately before the protocol field. Range 0 to 255.
2. RFC 791, September 1981, described it in time units and required a decrement of at least one per hop and per second of queuing.
3. RFC 1812, June 1995, section 5.3.1 replaced that with the flat rule: a router MUST decrement by exactly one. The time semantics are dead.
4. In IPv6 the field was renamed to Hop Limit, still 8 bits, at byte offset 7 of the 40-byte fixed header. The rename admits what the field always was.
5. The IANA IP Parameters registry states: “The current recommended default time to live (TTL) for the Internet Protocol (IP) is 64”, citing RFC 791 and RFC 1122.
6. Linux exposes the default as the sysctl net.ipv4.ip_default_ttl, documented as “Default value of TTL field (Time To Live) for outgoing (but not forwarded) IP packets”, with a documented default of 64.
7. Common starting values, all of them product defaults rather than standards, and all of them changeable by an administrator:

8. Guessing the sender from a received TTL is called passive OS fingerprinting and is implemented in tools such as p0f and in the ttl matcher of several firewalls. It is weak evidence, for four reasons.
9. First, a tunnel changes the count. The reader’s machine showed several utun interfaces, which are point-to-point tunnel devices used by VPN and similar software. Traffic through a tunnel gets a different hop count.
10. On macOS you can read the local default with sysctl net.inet.ip.ttl, and macOS traceroute uses that same value as the default maximum hop count, documented as “net.inet.ip.ttl hops (the same default used for TCP connections)”.

WORDS27.2.6 remember these#

1. TTL — a hop counter carried in every packet — an 8-bit IPv4 header field decremented by exactly one at each forwarding router, per RFC 1812 5.3.1.
2. Hop — one router handling the packet — one layer 3 forwarding decision.
3. Hop Limit — the IPv6 name for TTL — the 8-bit field at offset 7 of the IPv6 fixed header, semantically identical.
4. Header checksum — a small proof the header is intact — the 16-bit ones complement checksum over the IPv4 header only, updated incrementally.
5. Passive OS fingerprinting — guessing the sender’s system from clues — the inference of the originating stack from TTL, window size and option order.

27.3 The mechanism, step by step#

PLAIN27.3.1 in simple words#

1. Traceroute does the same small thing over and over, with one number changed each time.
2. Step one. Send a packet toward the final destination, but set its hop counter to 1.
3. The first router receives it, subtracts one, gets zero, and must throw it away. It sends back an error message that says “your packet ran out of hops at my place”.
4. That error message has the router’s own address as its sender. That is how the reader learned that hop 1 is 192.168.0.1.
5. Step two. Send another packet, exactly the same, but with the hop counter set to 2. It survives the first router and dies at the second.
6. The second router complains, and now the reader knows hop 2 is 172.31.0.17. Repeat with 3, then 4, then 5, until you reach the destination or give up.
7. Three packets are sent for each hop by default, not one, so a slow or lost probe does not hide a router.
8. Every line you see on screen is therefore a different packet. The trace is not one journey observed. It is dozens of separate short journeys.

PLAIN27.3.2 a picture in your head#

1. Picture a long corridor of doors, one behind the other, and you standing at the near end. You cannot see past the first door.
2. You have a supply of paper darts. Each dart has a number written on it.
3. The rule of the corridor is that each person who catches a dart writes one less on it and throws it to the next person, unless the number would go to zero.
4. If the number would go to zero, that person keeps the dart, writes their own name on a card, and walks the card back to you.
5. So you throw a dart marked 1 and the nearest person hands you a card. Throw one marked 2 and the second person does. Twenty darts give you the whole list in order.
6. You also time each card. You start a stopwatch when the dart leaves your hand and stop it when the card arrives.

Where this comparison breaks: the person walking the card back may take a completely different corridor, which you never see, so the stopwatch measures a loop and not a distance. Also, some people are under orders never to walk cards back, and some are so busy they return only one card a second. Both turn into stars on your screen.

PLAIN27.3.3 a worked example#

1. Here is the full sequence for the first three hops of the reader’s trace. Nine packets out, nine replies in.

2. Notice that the destination address on all nine packets was the same: the GitHub address 20.207.73.82. Only the TTL changed.
3. Now the whole process as a diagram. X marks where a probe dies.

 YOU              R1        R2        R3         TARGET
  |               |         |         |            |
  |--TTL=1------->X         |         |            |
  |<--Time Exceeded         |         |            |
  |   src = R1    |         |         |            |
  |               |         |         |            |
  |--TTL=2------->|-------->X         |            |
  |<--Time Exceeded---------|         |            |
  |   src = R2    |         |         |            |
  |               |         |         |            |
  |--TTL=3------->|-------->|-------->X            |
  |<--Time Exceeded-------------------|            |
  |   src = R3    |         |         |            |
  |               |         |         |            |
  |--TTL=4------->|-------->|-------->|----------->|
  |<--reply from the target itself, different type-|
  |               |         |         |            |

4. The last line is different on purpose. When a probe finally survives to the destination, the destination does not send a Time Exceeded message, because the packet did not run out of hops. It sends something else, and what it sends depends on which traceroute method you used.
5. Timing. Each of the three probes at each hop is timed on its own, which is why a normal traceroute prints three separate millisecond figures per line.
6. The reader’s captured trace does not show those figures. It records only the addresses. We will come back to that gap in section 27.7, because it is a real limitation of the evidence they collected.

PLAIN27.3.4 what is really happening inside#

1. The error message that comes back is not empty. It carries a copy of the dead packet inside it.
2. That copy is the key to the whole system. Without it, traceroute could not tell which of its many probes a given reply belongs to.
3. Replies arrive from different routers, in any order, some late, some never. Each reply must be matched to exactly one probe.
4. The trick is that the router quotes the dead packet back at you. Inside the error message you find the original header, including whatever identifying number traceroute put in it.
5. Classic traceroute puts that identifier in the destination port number. It uses a different port for every single probe.
6. So probe number 7 goes to port 33440, and when a reply comes back quoting port 33440, traceroute knows this is the answer to probe 7, sent at a known moment, with a known TTL.
7. This clever identifier scheme has one bad side effect, and it causes most of the confusion in the reader’s trace. Changing the port for every probe changes what load-balancing routers see, so different probes get sent down different links. Section 27.5 is entirely about that.

TECHNICAL27.3.5 the engineer’s version#

1. The response of interest is ICMP type 11, code 0. RFC 792 gives the code meanings verbatim as “time to live exceeded in transit” for code 0 and “fragment reassembly time exceeded” for code 1.
2. In IPv6 the same event is ICMPv6 type 3, code 0, “Hop limit exceeded in transit”, defined in RFC 4443, March 2006, by Conta, Deering and Gupta.
3. The ICMP error payload contains the IP header of the offending datagram plus at least the first 8 bytes of its payload, which is enough to capture the UDP or TCP source and destination ports, or the ICMP identifier and sequence number.
4. RFC 1812 section 4.3.2.3 upgraded that to say the error should contain as much of the original datagram as possible without exceeding a total ICMP message size of 576 bytes.
5. Source address selection matters for reading a trace. RFC 1812 section 4.3.2.4 requires the source address of a router-originated ICMP message to be an address of the interface over which the ICMP message is transmitted.
6. On a point-to-point link, that interface is the same one the probe arrived on. This is why traceroute conventionally shows the ingress interface address of each router, not a loopback address or a router identifier.
7. That single fact explains the interface names in the reader’s hops 7 to 12. ae66-0 and be23 are names of the specific interfaces the probes came in on, not names of the routers.
8. The macOS manual page states the defaults as “-q nqueries: Set the number of probes per ttl to nqueries (default is three probes)”, “-p port: default is 33434”, and “-w: default 5 sec.”.

WORDS27.3.6 remember these#

1. ICMP — the internet’s error and status messaging — Internet Control Message Protocol, RFC 792, IP protocol number 1.
2. Time Exceeded — the “your packet ran out of hops” message — ICMP type 11 code 0 in IPv4, ICMPv6 type 3 code 0 in IPv6.
3. Quoted packet — the copy of the dead packet inside the error — the original IP header plus at least 8 payload bytes, used for demultiplexing.
4. Ingress interface — the door the packet came in through — the interface whose address a router normally uses as the ICMP error source.
5. Round-trip time — how long the there-and-back took — the measured interval between probe transmission and matching ICMP response reception, per probe.

27.4 The variants, and which one to use#

PLAIN27.4.1 in simple words#

1. The trick with the hop counter works no matter what kind of packet you send. The counter is in the outer wrapper, which every packet has.
2. So there are several traceroutes, differing only in what they put inside the wrapper.
3. The original sends a small packet to a strange, unused port number on the destination. This is the classic method and it is still the default on Mac and Linux.
4. The Windows version sends a ping instead. A ping is the simplest possible “are you there” message.
5. The third version sends the first packet of a real connection attempt, aimed at a real service port, such as the one used by secure websites.
6. That second difference is the whole point. Networks are much more willing to pass traffic that looks like normal web traffic than traffic that looks like a diagnostic tool.
7. For the reader’s problem, which was a web connection failing, the third version is clearly the right one, and it is not the default.

PLAIN27.4.2 a picture in your head#

1. Imagine three people trying to reach the same office in a guarded building.
2. The first wears a plain uniform nobody recognizes, and asks for room 33434, which does not exist. Guards find this suspicious and often turn them away.
3. The second shouts “hello, is anyone there?” from the doorway. Guards at many buildings have standing orders to ignore that shout entirely.
4. The third walks in wearing normal clothes and asks for the reception desk, room 443, which is the room everyone visits all day long.
5. The third person gets much further, because they look exactly like the thousands of legitimate visitors the guards are there to let in.

Where this comparison breaks: the guards are not usually deciding that they dislike diagnostics. Very often nobody blocked anything, and the difference is simply that the third kind of traffic is what the path is built and tuned to carry. Also, asking for reception has a real cost: you make the far end start work on a connection it will never finish.

PLAIN27.4.3 a worked example#

1. The three commands, for the reader’s exact target, on macOS.

# 1. classic UDP, the default. No special permission needed.
traceroute 20.207.73.82

# 2. ICMP echo, the same method Windows tracert uses.
sudo traceroute -I 20.207.73.82

# 3. TCP to port 443, the one to use for a web problem.
sudo traceroute -P TCP -p 443 20.207.73.82

2. What each looks like at the moment it reaches the destination.

3. Notice that the classic method succeeds by provoking an error. It picks a port nothing is listening on precisely so the destination answers “nothing here”, which proves you reached it.
4. The TCP method succeeds by getting an actual answer from an actual service. If the destination replies at all, you have proven far more than a map.

PLAIN27.4.4 what is really happening inside#

1. The important idea is that the intermediate hops behave identically in all three cases. Time Exceeded is generated for any packet whose counter runs out, regardless of what is inside.
2. But whether a probe reaches a middle hop at all depends on every device before it.
3. A firewall five hops along that drops odd UDP traffic will make everything past it look dead, even though the routers there are perfectly healthy.
4. Swap to TCP port 443 and the same probes sail through, and the hops appear.
5. There is one more family worth knowing: tools that repeat the whole trace continuously and build statistics, rather than running once and stopping.
6. Running once gives you a snapshot. Running for two minutes tells you whether a hop loses one packet in a hundred, which a snapshot can never show.

TECHNICAL27.4.5 the engineer’s version#

1. UDP traceroute, the 1987 Van Jacobson design, sends UDP datagrams to destination ports starting at 33434 and increasing by one per probe, giving the familiar 33434 to 33534 window.
2. Termination is by ICMP type 3 code 3, Destination Unreachable, Port Unreachable, from the target’s own stack.
3. ICMP traceroute sends ICMP Echo Request, type 8 code 0, and terminates on Echo Reply, type 0 code 0. This is the default of Windows tracert, and is available on macOS as -I, documented as “Use ICMP ECHO instead of UDP datagrams. (A synonym for -P icmp)”.
4. TCP traceroute sends a TCP SYN to a chosen port, typically 80 or 443, and terminates on SYN-ACK from an open port or RST from a closed one.
5. The macOS manual page documents -P proto as “Send packets of specified IP protocol. The currently supported protocols are: UDP, TCP, GRE and ICMP”. Raw socket creation means -I and -P TCP need root.
6. Why TCP to 443 gets further, in order of practical importance:

7. mtr, originally written by Matt Kimball around 1996 and maintained by Roger Wolff from October 1998, combines traceroute and ping. It re-probes continuously and reports per-hop loss percentage, best, worst, average and standard deviation. Modern builds support ICMP, UDP and TCP probes.
8. Paris traceroute was introduced in “Avoiding traceroute anomalies with Paris traceroute” by Brice Augustin, Xavier Cuvellier, Benjamin Orgogozo, Fabien Viger, Timur Friedman, Matthieu Latapy, Clemence Magnien and Renata Teixeira, at the Internet Measurement Conference in October 2006.
9. What it fixes: classic traceroute varies the destination port per probe, which changes the flow identifier that load balancers hash on, so consecutive probes take different paths and the printed path is a blend of several real paths that may never have existed as one route.
10. Paris traceroute holds the flow identifier constant across probes by varying a field the load balancer does not hash, classically the UDP checksum field, compensated so the checksum still validates.
11. Recommendation for the reader’s exact situation, a failing HTTPS connection to 20.207.73.82: use TCP to port 443. It is the only method whose probes are treated by every device on the path the same way the real failing traffic is treated. Anything else measures a different journey.

WORDS27.4.6 remember these#

1. Probe type — what kind of packet the tool sends — the transport and port combination that determines both filtering treatment and ECMP hashing.
2. Port Unreachable — “nothing is listening there” — ICMP type 3 code 3, the normal termination signal of UDP traceroute.
3. SYN — the first packet of a connection request — a TCP segment with the SYN flag set, opening the three-way handshake.
4. Paris traceroute — a traceroute that keeps probes looking identical — a variant holding the ECMP flow identifier constant across probes, from the 2006 Internet Measurement Conference paper.
5. MTR — a traceroute that never stops — a continuous hybrid of traceroute and ping reporting per-hop loss and latency statistics.

27.5 Why one hop shows several addresses#

PLAIN27.5.1 in simple words#

1. Look at hop 5 of the reader’s trace. It has three addresses on one line.
2. Neither is true. It is normal, healthy, and a sign of a well-built network.
3. Remember that three separate packets were sent for that hop. Three packets, three journeys, three answers.
4. Big networks do not join two places with one cable. They join them with several cables side by side, for capacity and for safety.
5. When several equally good roads exist, a router shares traffic across them.
6. So probe one went left, probe two went straight, probe three went right. Each landed on a different router, and each of those routers answered.
7. It means the network has spare capacity in that place. If one of those cables was cut, traffic would keep flowing on the others.
8. Five of the reader’s hops show this. Hops 5, 6, 8, 9 and 11. That is not a fault, and it is not a sign of anything wrong.

PLAIN27.5.2 a picture in your head#

1. Imagine a wide river with three parallel bridges between the same two towns, and a traffic officer at the near end sending cars over all three.
2. You want to map the route. You send three cars, each told to stop at the far end of the first bridge and report which bridge they were on.
3. Now suppose the middle bridge is a little longer, with an extra toll booth halfway. Cars told to stop after the same number of booths stop in different places on different bridges.
4. That is why the reader’s hop 11 shows two routers in two different cities. The parallel roads are not exactly the same length.

Where this comparison breaks: the officer chooses at random and a real router almost never does. It computes a number from the addresses and ports on each packet and uses that to pick a bridge, so two packets of one conversation always get the same bridge and cannot arrive out of order. Traceroute defeats that on purpose by changing the port on every probe.

PLAIN27.5.3 a worked example#

1. Here are the reader’s five multi-address hops, exactly as recorded.

2. Hop 5. 172.16.18.33, 172.26.14.75 and 172.26.22.234. All three probes took different links. Three answers, three addresses.
3. Hop 6. 172.26.14.75 and 172.16.18.33. Both of these already appeared at hop 5.
4. That looks impossible until you remember the bridges. On one parallel road that router is five hops away; on another it is six.
5. Hop 8. Two Mumbai routers named rwa02 and rwa04. Different machines in the same city doing the same job.
6. Hop 9. Two Mumbai routers named owr01 and owr02. Same pattern, one step further in.
7. Hop 11. The most interesting one. One probe was answered by a router in Pune site 21, another by a router in Pune site 20.
8. So at the same hop count, two probes were in two different buildings. The two parallel paths through Pune are not the same length.

PLAIN27.5.4 what is really happening inside#

1. The technique is called equal-cost multi-path, usually shortened to ECMP. It means: when the routing table offers two or more equally good next steps, use all of them.
2. The router must decide, per packet, which of them to use, and there are two ways to decide.
3. The first way is per-packet: alternate strictly, one packet each. This uses the links perfectly evenly and it is almost never used, because packets of one conversation arrive out of order and confuse the receiver.
4. The second way is per-flow: work out which conversation the packet belongs to, and always send that conversation the same way. This is what virtually every network does.
5. To identify the conversation, the router takes a handful of fields from the packet and mixes them into a single number. Then it takes the remainder of that number divided by the number of links.
6. The fields are usually the source address, the destination address, the protocol, the source port and the destination port. Five things, so this is called the five-tuple.
7. Now the punchline. Classic traceroute changes the destination port on every single probe, because it uses the port as a serial number. Change the port, change the mixed number, change the chosen link.
8. Your web browser does the opposite. One TCP connection keeps the same five fields for its whole life, so it stays on one link the entire time.
9. This is the single most important thing to understand about reading a traceroute: the tool is not following your traffic. It is sampling all the roads your traffic could have taken.

            +--> R-a  172.16.18.33  ---+
            |                          |
 hop 4 -----+--> R-b  172.26.14.75  ---+---> onward
            |                          |
            +--> R-c  172.26.22.234 ---+

 probe 1 hashes to R-a, probe 2 to R-b, probe 3 to R-c
 all three answer, so hop 5 prints three addresses

TECHNICAL27.5.5 the engineer’s version#

1. ECMP is standard behaviour in every modern routing platform. Inside a single operator, it is usually driven by an interior gateway protocol such as OSPF or IS-IS installing multiple equal-metric next hops.
2. The hash input is configurable. Typical defaults include the IPv4 source and destination addresses, the protocol number, and the layer 4 source and destination ports. Some platforms add the incoming interface index or a per-device seed so that two routers in series do not make correlated choices, which is called hash polarization.
3. Load balancing also happens below layer 3, on link aggregation groups. A bundle of physical links presented as one logical interface hashes frames across its members, often on MAC and IP fields.
4. That distinction matters for reading names. An interface called ae66-0 or be23 is a bundle. Traffic is already being spread across the physical members of that bundle, invisibly, without any change to the hop count.
5. So the reader’s trace shows two layers of load balancing at once: ECMP across distinct routers, which is visible as extra addresses, and link bundling inside each hop, which is completely invisible.
6. Classic traceroute increments the UDP destination port per probe, mutating the five-tuple and therefore the hash, which is the anomaly Paris traceroute was written to eliminate.
7. Paris traceroute keeps the five-tuple constant and varies the UDP checksum field instead, adjusting payload bytes so the checksum remains valid. Routers do not hash on the checksum, so all probes follow one path.
8. The reader’s hop 11 is the classic textbook case of the anomaly Paris traceroute describes. Reading hops 10, 11 and 12 as a single chain gives a route that probably never existed as one route.

9. Proven by this data: at hop 11 the probes reached at least two different routers in two different Pune sites.
10. Inferred, not proven: that these form two clean parallel chains as shown. A single-run traceroute cannot separate the chains. Paris traceroute with several fixed flow identifiers, or a multipath detection run that enumerates every parallel path, would settle it.

WORDS27.5.6 remember these#

1. ECMP — using several equally good roads at once — equal-cost multi-path forwarding across multiple next hops of identical routing metric.
2. Five-tuple — the five fields that identify a conversation — source address, destination address, protocol, source port, destination port.
3. Per-flow load balancing — keep each conversation on one road — hashing the flow identifier to select a next hop, preserving packet order.
4. Per-packet load balancing — alternate every packet — round-robin next-hop selection, giving even utilization at the cost of reordering.
5. Link aggregation — several cables acting as one — LAG or bundle interfaces, hashing frames across members below the routing layer.
6. Hash polarization — two routers making the same bad choice — correlated next-hop selection when devices in series use identical hash inputs.

27.6 Stars, and how to read them honestly#

PLAIN27.6.1 in simple words#

1. A star in a traceroute means one thing only: no reply arrived for that probe within the waiting time.
2. It does not mean the packet stopped there. It does not mean a router is down. It does not mean anything is broken.
3. There are five common reasons for a star, and only one of them is a fault.
4. Reason one. The router is configured not to send these messages at all. Some operators disable them on purpose.
5. Reason two. The router sends them, but only a few per second, and yours was over the limit. This is very common and it is the default on many systems.
6. Reason three. Something between you and the router blocks the reply on the way back, or blocks your probe on the way out.
7. Reason four. A tunnel technology carries your packet across several machines while showing you only one hop.
8. Reason five. The destination itself is silent by policy, which is normal for large web services.
9. Stars in the middle of a trace, with normal hops after them, always mean a quiet router and never mean a fault. The packets clearly got through.
10. Stars at the end of a trace are the hardest to read, and they are extremely common. Most traces of most large services end in stars.

PLAIN27.6.2 a picture in your head#

1. Go back to the corridor of doors and the paper darts.
2. You throw a dart marked 13 and no card comes back.
3. Six things could have happened, and from where you stand they look identical.
4. Nobody was at position 13, so the dart flew on and nobody kept it.
5. Someone was there but has orders never to walk cards back to strangers.
6. Someone was there, would normally reply, but has already walked one card this second and refuses to walk another.
7. Someone was there and wrote a card, but a doorman between you tore it up.
8. There was a covered walkway between position 12 and position 18, so positions 13 to 17 do not exist as far as darts are concerned.
9. Or the corridor really does end in a wall at position 13.
10. You cannot tell which. Standing at your end and staring harder does not help. You need a different experiment.

Where this comparison breaks: in the corridor, silence is unusual. On the internet, silence is the normal, expected, default state of most routers near a large service. Treat a silent tail as the baseline, not as a surprise.

PLAIN27.6.3 a worked example#

1. The reader’s trace ends like this.

12  ae106-0.rwa04.pnq20.ntwk.msn.net (104.44.20.42)
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *

2. What is proven: no ICMP Time Exceeded message reached the reader’s machine for any probe with a hop counter of 13 or more, within the wait time.
3. What is not proven: that the packets stopped at hop 13. That is a guess, and there is no evidence for it in this trace.
4. Compare with a control. Run the same trace to a target known to answer, on the same connection, at the same moment.
5. If that trace also ends in eight stars, the stars belong to the reader’s own network or to the general environment, not to this destination.
6. If it completes cleanly, the stars are specific to this path, which narrows things a great deal without proving anything about the cause.
7. The reader did not run that control. It is the single cheapest improvement available, and section 27.13 shows how.

PLAIN27.6.4 what is really happening inside#

1. Generating a Time Exceeded message is expensive for a router, in a way that forwarding is not.
2. Forwarding happens in dedicated hardware: a chip that looks up an address and moves the packet, millions of times a second, without a general-purpose processor involved.
3. Building an error message is different work. The router must copy part of the dead packet, build a new packet around it, work out a source address, and send it. That is software, on the router’s ordinary processor.
4. So every serious router limits how much of it can be spent on courtesy messages to strangers. Not out of malice. Out of survival.
5. On Linux the default limit is one such message per second to any given destination, and carrier gear is usually tighter still.
6. Now count. Traceroute sends three probes to the same hop, back to back, in a few milliseconds. Under a one-per-second limit, two of them get nothing.
7. This is why partial stars, like one answer and two stars on the same line, are so common. It is a rate limit, almost every time.
8. Tunnels are the other big cause. If an operator carries your packet inside a wrapper across five internal routers, and the wrapper is set up not to copy your hop counter, those five routers never see your counter reach zero. They vanish from the trace entirely.

TECHNICAL27.6.5 the engineer’s version#

1. Linux rate limiting is exposed as sysctls, with documented defaults:

2. The mask value 6168 is binary with bits 3, 4, 11 and 12 set, meaning ICMP types 3, 4, 11 and 12. Type 11 is Time Exceeded. So Time Exceeded is rate limited by default on Linux, at one per second per destination.
3. RFC 1812 section 4.3.2.8 requires it: a router MUST implement rate limiting of ICMP error messages to avoid being used in a denial-of-service attack. Silence is standards-compliant behaviour, not misconfiguration.
4. Carrier platforms add control-plane policing, a hard rate limit on traffic punted to the route processor, usually classed so that routing protocol packets are protected and ICMP generation is not.
5. MPLS is the main cause of hidden hops. RFC 3032, “MPLS Label Stack Encoding”, January 2001, by Rosen and others, specifies that when an IP packet is first labelled, the label’s TTL is set from the IP TTL, and on the final pop the IP TTL is set from the label TTL.
6. Operators frequently disable that copying, so the label TTL starts at 255 instead of inheriting yours. Then the whole tunnel decrements a counter your probes never see, and the tunnel interior is invisible.
7. This is an implementation detail with different names per vendor: on Cisco IOS it is no mpls ip propagate-ttl, on Junos it is no-propagate-ttl under the MPLS protocol stanza.
8. RFC 4950, “ICMP Extensions for Multiprotocol Label Switching”, August 2007, by Bonica, Gan, Tappan and Pignataro, allows a label switching router to attach its label stack to ICMP Time Exceeded and Destination Unreachable messages, so a traceroute can display MPLS labels when the operator allows it.
9. Honest ranking of causes of a silent tail, most likely first, for a trace toward a large cloud provider:

10. Note that the genuine fault is last on that list. That ordering is the whole lesson of this section.
11. For the reader’s hops 13 to 20: the evidence is consistent with every one of those six causes, and distinguishes between none of them.

WORDS27.6.6 remember these#

1. Star — no reply arrived in time — a probe timeout, recorded as an asterisk, carrying no information about the cause.
2. Rate limiting — only a few replies per second — an enforced minimum interval between generated ICMP messages, required by RFC 1812 4.3.2.8.
3. Control plane — the router’s thinking part — the route processor running protocols and generating ICMP, distinct from the forwarding hardware.
4. Control-plane policing — protecting the thinking part — a hard rate limit on traffic punted from the data plane to the route processor.
5. MPLS — a tunnelling technology used inside carriers — Multiprotocol Label Switching, which can hide interior hops by not propagating the IP TTL.
6. Black hole — a place where packets vanish silently — a forwarding or filter state that discards traffic without generating any ICMP notification.

27.7 Why the times look strange#

PLAIN27.7.1 in simple words#

1. A normal traceroute prints three times on each line, in milliseconds. One for each probe.
2. People expect those numbers to grow steadily down the page, because each hop is further away. Often they do not.
3. You will regularly see hop 9 answer faster than hop 8. That looks like the packet went backwards in time.
4. It did not. Each number is a full round trip: out to that router, and all the way back to you. It is not the time for that one step of the journey.
5. And the way back can be completely different from the way out. You never see the way back.
6. There is a second cause, and it is bigger than most people expect. Some routers are simply slow at producing the reply, even though they forward traffic at full speed.
7. A router that takes 30 milliseconds to write a complaint will show a high number, while the packets it forwards are delayed by almost nothing.
8. So a slow-looking hop in the middle of a trace, with fast hops after it, is proof that the router is slow at complaining and nothing else.
9. The rule to remember: you cannot subtract two lines to get the time of the link between them.

PLAIN27.7.2 a picture in your head#

1. Imagine measuring the distance to towns along a road by sending a runner to each one with a message, and timing until a reply comes back to you.
2. The reply might come back by a different road, or by train, or by post. Your stopwatch includes all of that.
3. Also, each town has one clerk who writes the replies, and clerks vary. One town’s clerk answers in seconds. The next town’s clerk finishes lunch first.
4. Now try to work out the distance between town four and town five by subtracting your two stopwatch readings.
5. The answer is meaningless. You have subtracted two different return journeys and two different clerks.

Where this comparison breaks: the outward journey really is shared, so times do usually trend upward over many hops, and a jump that persists across every later hop is real evidence of a long link. The caution is against reading small differences, not the overall shape.

PLAIN27.7.3 a worked example#

1. First, an honest statement about the evidence. The reader’s captured trace contains no times at all. It records only hop numbers and addresses.
2. So the example here is a typical trace of the same shape, to show what the reader would have seen.

 7  104.44.196.187   18.412 ms  17.998 ms  18.203 ms
 8  104.44.55.163    41.902 ms  42.118 ms  41.777 ms
 9  104.44.53.119    88.640 ms  39.115 ms  39.402 ms
10  104.44.31.62     42.881 ms  43.006 ms  42.744 ms
11  104.44.20.52     42.560 ms  44.881 ms  43.019 ms
12  104.44.20.42     43.702 ms  43.244 ms  43.610 ms

3. Read hop 9. One probe took 88 ms and the other two took 39 ms. That is not a congested link. If it were, all three would be slow.
4. It is one reply that queued behind other work on that router’s processor. The forwarding path was fine, as hops 10, 11 and 12 prove by being fast.
5. Read hop 10. It is 0.7 ms faster than hop 9, despite being further away. That is well within noise and means nothing.
6. Read the jump between hop 7 and hop 8: 18 ms to 42 ms, and it sticks for every later hop. That is real. Something long sits between them.
7. That is the correct way to use these numbers. Look for jumps that persist, ignore single slow probes, and never subtract.

PLAIN27.7.4 what is really happening inside#

1. Break one printed number into its parts. It contains four things and you can separate none of them.
2. Part one: the time for your probe to travel from you to that router.
3. Part two: the time that router took to notice the dead packet, hand it to its processor, and build the reply. This can be microseconds or tens of milliseconds.
4. Part three: the time for the reply to travel back to you, along a path you cannot see and which need not be the reverse of the outward path.
5. Now consider why the return path is invisible. Internet routing is chosen independently by each network for traffic entering it. The road from A to B and the road from B to A are separate decisions made by different people.
6. Paths being different in the two directions is called asymmetry, and it is the normal case between large networks, not the exception.
7. So when hop 8 shows 42 ms and hop 9 shows 39 ms, the honest reading is: hop 9’s reply happened to come home by a quicker road, or hop 9’s processor was less busy. Both are ordinary.

TECHNICAL27.7.5 the engineer’s version#

1. The measured value is the interval between the probe’s transmission timestamp and the reception timestamp of the matching ICMP response, taken in user space by the traceroute process.
2. That value decomposes as forward path delay, ICMP generation latency, and return path delay, plus scheduling jitter on the measuring host itself.
3. ICMP generation latency is the dominant unexplained term. Time Exceeded generation is a control-plane function. On platforms with hardware forwarding, the probe is punted from the data plane to the route processor, queued behind protocol traffic, and answered at low priority.
4. Observed generation latencies of 10 to 100 ms on a busy carrier router are routine while the same router forwards transit traffic with under 100 microseconds of added delay.
5. This is why an intermediate hop can show a far higher RTT than hops beyond it. The high value describes the router’s own responsiveness, not the path.
6. Path asymmetry means the return leg is chosen by BGP policy in the far network and may traverse entirely different transit providers, exchange points and even continents.
7. Consequently, per-link latency cannot be derived by differencing adjacent hop RTTs. The difference mixes two different return paths and two different generation latencies, and can be negative.
8. Useful practice for reading times, in order:

9. Critical caution when reading mtr loss columns: loss at an intermediate hop that does not appear at later hops is rate limiting, not packet loss. Only loss that continues to the final hop is real.

WORDS27.7.6 remember these#

1. Round-trip time — there and back — the interval from probe transmission to matching response reception, including generation latency.
2. Path asymmetry — the way back is not the way out — independent forward and reverse routing decisions between autonomous systems.
3. Data plane — the fast part that moves packets — hardware forwarding, typically an ASIC, operating at line rate.
4. Punt — hand a packet to the slow part — transferring a packet from the forwarding hardware to the route processor for software handling.
5. Jitter — variation in delay — the statistical spread of one-way or round-trip latency across repeated measurements.

27.8 What a traceroute cannot tell you#

PLAIN27.8.1 in simple words#

1. This is the most important section in the chapter, so read it twice.
2. A traceroute is a weak instrument. It answers one narrow question, and people routinely use it to answer five questions it cannot touch.
3. It cannot tell you the way back. You see only the outward direction.
4. It cannot tell you whether the destination is working. Silence from a destination is normal for almost every large service.
5. It cannot tell you where a block is. The last hop that answered is the last hop that was willing to answer, which is a different thing.
6. It cannot tell you anything at all about what lies past the last reply. Not whether there are more routers, not whether your packets got there.
7. It cannot prove that a hop showing high times is causing your problem.
8. It cannot prove the path is the path your real traffic takes, because your real traffic does not change its port number on every packet.
9. Now say this plainly, because it is the reader’s own case.
10. The reader’s traceroute, on its own, would NOT have proved a fault. Not the stars, not the twelve hops, not any part of it.
11. Everything in that trace is compatible with a completely healthy network in which GitHub simply does not answer diagnostic probes.
12. Something else proved the fault. The trace only narrowed the region.

PLAIN27.8.2 a picture in your head#

1. A traceroute is a torch shone down a corridor, at night, from one end.
2. It lights up the people who choose to wave back. It does not light up the corridor.
3. Someone standing still and silent is invisible. So is a wall. So is an open door with nothing beyond it.
4. Now imagine you conclude “the corridor ends at position 13” because nobody waves past 12.
5. You have confused “nobody waved” with “nothing is there”. That is the error almost everyone makes with traceroute.

Where this comparison breaks: the torch is worse than it sounds, because it also does not show you the route the wave took to come back. Two people standing next to each other can wave back along completely different routes, and their apparent distance from you will differ for that reason alone.

PLAIN27.8.3 a worked example#

1. Here is the reader’s situation, split into what was proven and what was merely suggested.

2. Now the observations that came from outside the traceroute, in the same session, and what those proved.

3. Compare the two tables. The second one carries almost all of the weight.
4. The decisive fact is the combination: the same request to the same address succeeded over a different network, at the same time.
5. That rules out the destination being down, because it served the phone.
6. It also rules out DNS, because both used the same address.
7. What remains is that something on the path used by the home connection discarded the traffic without saying so.
8. The traceroute contributed one thing to that conclusion: it showed which networks were involved, so the problem could be described precisely instead of vaguely.

PLAIN27.8.4 what is really happening inside#

1. The deep reason traceroute is weak is that it measures the wrong traffic.
2. Its probes are not your traffic. They have different ports, often a different protocol, a different size, and a deliberately broken hop counter.
3. Every device on the path is entitled to treat them differently from your real traffic, and many do.
4. So a perfect traceroute does not prove your real connection will work, and a completely silent traceroute does not prove it will fail.
5. There is a second, subtler weakness. Traceroute reports on the forward direction only, but a connection needs both directions to work.
6. A failure in which your packets arrive perfectly and the replies are discarded looks, from your end, exactly like a failure in which your packets never arrive.
7. Traceroute cannot separate those two cases. Nothing you run from one end can. You need a measurement from the other end.
8. This is why the honest diagnostic sequence is: use traceroute to describe the region, then use a test with real traffic to establish the fault, then get a second viewpoint from elsewhere to locate it.

TECHNICAL27.8.5 the engineer’s version#

1. Enumerated limits of a single-ended forward traceroute:

2. The probe traffic differs from production traffic in transport, port, payload size, packet rate and TTL, so per-hop treatment can differ under access control lists, QoS classification, policy-based routing and stateful inspection.
3. The last responding hop is the last hop willing and able to generate an ICMP error that reached you. It carries no information about topology beyond itself.
4. A common and serious error is naming the operator of the last responding hop as the cause. In the reader’s case that would mean blaming Microsoft’s Pune router, which the data does not support in any way.
5. Correct statements for the reader’s evidence, split by strength:
6. Proven. Twelve hops responded, ending at 104.44.20.42, a Microsoft address inside 104.40.0.0/13, announced by AS8075. No response arrived for TTL values 13 through 20.
7. Proven, from outside the trace. A TCP connection to 20.207.73.82 port 443 produced no response of any kind in 15 seconds, and the identical request over a mobile network succeeded immediately.
8. Supported, not proven. Traffic on this path was silently discarded somewhere at or beyond the last responding hop.
9. Not supported by any evidence here. That a specific named party blocked it, that the discard point is hop 13, or that the trace itself demonstrates a fault.

WORDS27.8.6 remember these#

1. Single-ended measurement — testing from one side only — a measurement with no observation point in the far network, blind to the reverse path.
2. Last responding hop — the last router that answered — the furthest TTL for which an ICMP error was received, with no topological meaning beyond that.
3. Silent drop — discarded without notice — a discard action that generates no ICMP error, defeating all ICMP-based diagnosis.
4. Liveness — whether something is actually working — reachability and service availability, which ICMP silence cannot establish either way.
5. Looking glass — a window into someone else’s network — a public read-only interface for running ping, traceroute and BGP queries from an operator’s own routers.

27.9 The reader’s trace, decoded hop by hop#

PLAIN27.9.1 in simple words#

1. The trace has four zones, and once you see them the whole thing reads easily.
2. Zone one is hop 1. That is the box in the reader’s own flat.
3. Zone two is hops 2 to 6. That is the reader’s internet provider, using addresses that are not visible from the outside world.
4. Inside zone two, hop 3 stands out. It has an address of the kind used on the public internet, sitting between two hops that do not.
5. Zone three is hops 7 to 12. Every one of those belongs to Microsoft, which owns GitHub, and the names say which city each is in.
6. Zone four is hops 13 to 20, which is silence.
7. So the journey visible in this trace is: home, provider, Microsoft, quiet.

PLAIN27.9.2 a picture in your head#

1. Think of posting a parcel and watching the tracking page fill in.
2. First scan: your own doorstep.
3. Next few scans: your local courier’s vans and depots, with internal codes that mean nothing to anyone outside that company.
4. Then a scan at a handover point, where the parcel changes companies.
5. Then several scans inside the second company, whose codes are different and happen to include city names you recognize.
6. Then the tracking page stops updating.

Where this comparison breaks: a tracking page stops updating because nobody scanned the parcel, and that is exactly right here too. But a parcel is one object with one history. This trace is twenty different parcels, and some of the scans belong to journeys that never met.

PLAIN27.9.3 a worked example#

1. The whole trace in one table. Names are shown without the shared ending .ntwk.msn.net, which every one of them carries.

2. Rows marked a, b and c are different probes at the same hop count. They are not separate hops.

PLAIN27.9.4 what is really happening inside#

1. Hop 1, 192.168.0.1. The reader’s own router, in the flat. It is the default gateway: the machine every packet leaving the home goes to first. The address is in 192.168.0.0/16, a range reserved for private use, so millions of homes use this exact number at the same time.
2. Hop 2, 172.31.0.17. The first machine inside the provider. Still a private address, from the 172.16.0.0/12 range, which covers everything from 172.16.x.x to 172.31.x.x. This is the provider’s access network.
3. Hop 3, 137.97.29.249. A public address, and the only one before Microsoft. It belongs to the reader’s provider. Its presence here tells you the provider mixes public and private numbering inside its own core.
4. Hop 4, 172.26.22.235. Back to private. This is not a mistake and not a loop. Providers commonly number their internal links privately even when the same routers also hold public addresses on other interfaces.
5. Hop 5, three addresses. 172.16.18.33, 172.26.14.75 and 172.26.22.234. Three probes, three parallel links, three routers. Note that .234 is one below hop 4’s .235, which is what you expect from two interfaces on the same small link subnet.
6. Hop 6, two addresses. 172.26.14.75 and 172.16.18.33. Both appeared at hop 5. The same routers are five hops away by one road and six by another, so they answer at both distances.
7. Hop 7, 104.44.196.187. The handover. The name ends in ntwk.msn.net, which is Microsoft’s backbone naming domain, and the site code is del01, meaning Delhi. This is where the provider hands the traffic to Microsoft.
8. Hop 8, two Mumbai routers. 104.44.55.163 and 104.44.51.55, named rwa02 and rwa04 at site bom01. Mumbai. Two parallel paths, both inside Microsoft.
9. Hop 9, two more Mumbai routers. 104.44.53.119 and 104.44.53.47, named owr02 and owr01. Still Mumbai, one layer deeper in.
10. Hop 10, 104.44.31.62. Named ibr02.pnq21. Pune. Both parallel paths converge here, because only one address appeared.
11. Hop 11, two Pune routers in two buildings. 104.44.20.52 at site pnq21 and 104.44.16.51 at site pnq20. The paths diverge again, and the two roads are not the same length.
12. Hop 12, 104.44.20.42. Named rwa04.pnq20. Pune. The last machine on the whole internet that said anything at all to the reader’s computer.

TECHNICAL27.9.5 the engineer’s version#

1. Address classification, with the governing document:

2. RFC 1918, “Address Allocation for Private Internets”, February 1996, reserves 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. The /12 spans 172.16.0.0 to 172.31.255.255, so all of hops 2, 4, 5 and 6 fall inside it.
3. Proven for the Microsoft hops: 104.44.196.187 lies inside 104.40.0.0/13, a prefix announced by AS8075, registered to Microsoft Corporation. Every hop from 7 to 12 shares that block.
4. GitHub was acquired by Microsoft in 2018, and github.com in this session resolved to 20.207.73.82, inside 20.192.0.0/10, also AS8075. So the Microsoft hops are not a detour; they are the destination network.
5. On hop 4 and hop 5c: 172.26.22.235 and 172.26.22.234 differ by one. Two readings are possible and a traceroute cannot separate them.
6. Reading one: they are the two ends of a single point-to-point link, addressed as a /31 under RFC 3021, “Using 31-Bit Prefixes on IPv4 Point-to-Point Links”, December 2000, which gives exactly two usable addresses, or as a /30 giving two usable out of four.
7. Reading two: they are unrelated interfaces that merely sit close together in the same /24 the operator uses for core links.
8. Mark this clearly as inferred. Nothing in a traceroute reveals a prefix length.
9. Hop 3 being public while hops 2, 4, 5 and 6 are private is worth stating carefully. It does not imply the packet left the ISP at hop 3 and came back. A single router can hold public addresses on some interfaces and private addresses on others, and the ICMP source is the interface the reply left by.
10. Reverse DNS: the names on hops 7 to 12 come from PTR lookups performed by the reader’s own resolver, 1.1.1.1, after the trace collected the addresses. Hops 1 to 6 show no names because the ISP publishes no PTR records for that private space, which is normal and correct.
11. Verify any of this yourself with whois 104.44.196.187, dig -x 104.44.196.187 @1.1.1.1, or an RDAP query against the ARIN registry.

WORDS27.9.6 remember these#

1. Default gateway — the router your machine sends everything to — the next hop for destinations outside the local subnet.
2. Private address — an address only meaningful inside one network — a block from RFC 1918, never routed on the public internet.
3. Reverse DNS — turning an address back into a name — a PTR record lookup under in-addr.arpa, published at the address holder’s discretion.
4. Autonomous system — one network under one routing policy — an AS such as AS8075 for Microsoft, identified by number in BGP.
5. Point-to-point link — a cable with exactly two ends — a link addressed with a /30 or, under RFC 3021, a /31.

27.10 Reading carrier router names#

PLAIN27.10.1 in simple words#

1. Large network operators give every router a name, and the name is built from parts that describe the machine.
2. The parts usually say: which cable the packet came in on, which building the router is in, which city that is, and what job the router does.
3. Once you can read them, a trace stops being a wall of characters and becomes a sentence about geography.
4. Important warning first. There is no standard for this. No specification says how to name a router.
5. Every operator invents its own scheme, and schemes change over time inside the same company.
6. So everything in this section is a convention: a pattern people follow because it is useful, not a rule anyone is obliged to obey.
7. Read names as strong hints, never as proof. If a name says Mumbai and the times say otherwise, believe the times.

PLAIN27.10.2 a picture in your head#

1. Think of a hospital room label: 2W-14B-ICU.
2. Floor 2, west wing, room 14, bed B, intensive care.
3. Nobody outside that hospital could invent that code, but anyone inside reads it instantly and knows exactly where to walk.
4. Another hospital across the road uses a completely different code for the same idea, and both are perfectly sensible.
5. Router names work exactly like that, and ntwk.msn.net is one hospital’s labelling system.

Where this comparison breaks: a hospital label is put there by the hospital for its own use, and so is a router name, but you are reading it from outside with no key. Some parts are guessable and some, like a device serial code, are not. Say which is which rather than inventing meanings for the rest.

PLAIN27.10.3 a worked example#

1. Take the reader’s hop 7 and split it at the dots.

ae66-0 . del01-96cbe-1b . ntwk . msn . net
  |            |            |     |     |
  |            |            |     |     +-- top-level domain
  |            |            |     +-------- Microsoft's domain
  |            |            +-------------- network naming zone
  |            +--------------------------- site and device
  +---------------------------------------- interface on that device

2. Read it right to left, the way domain names are actually structured.
3. net is the top-level domain. msn is a Microsoft domain, in use since the Microsoft Network launched in 1995. ntwk is short for network, the zone Microsoft uses for its backbone infrastructure.
4. del01 is the site: Delhi, facility 01. 96cbe-1b is a device identifier that is not publicly documented, so do not guess at it.
5. ae66-0 is the interface: aggregated Ethernet bundle 66, logical unit 0.
6. Putting it in a sentence: “logical unit 0 of bundle 66 on a device at Microsoft’s Delhi site 1”.

PLAIN27.10.4 what is really happening inside#

1. The name did not travel with the packet. The router never told anyone its name.
2. What happened is that traceroute collected an address, then asked DNS to turn that address back into a name. That is a reverse lookup.
3. The operator publishes those names in DNS voluntarily, one per interface address, usually generated automatically from its own inventory system.
4. That is why the interface appears in the name at all. There is one name per interface address, not one name per router.
5. So a single router shows up under different names at different hops, depending on which door your probe came in through.
6. It also means the names can be stale. An automatic system that is not run after a change leaves an old name in place for months.
7. And it means the names can be absent, which is what happened on the reader’s hops 1 to 6.

TECHNICAL27.10.5 the engineer’s version#

1. City codes in ntwk.msn.net follow IATA airport codes, the three-letter codes used on luggage tags. The three in the reader’s trace, verified:

2. The trailing digits are facility numbers within the metro, so pnq20 and pnq21 are two distinct Pune sites, not two routers in one building.
3. Role codes, as used by Microsoft. These are a convention of one operator, not an industry standard, and other carriers use entirely different letters:

4. Reading the roles as a shape: ibr devices carry traffic between regions over long-haul links, rwa devices aggregate within a region, and owr devices sit further out toward peers and customers.
5. Interface prefixes all mean the same underlying thing, several physical links bonded into one logical link, but the word differs by vendor:

6. Seeing ae, be and po in one trace is normal and suggests a multi-vendor network, which every large operator runs deliberately.
7. The -0 in ae66-0 and ae104-0 is the logical unit or subinterface number, Junos style. be23 and po22 carry no unit, which is the Cisco style of writing the same idea.
8. Full decode of every hostname in the reader’s trace. All nine end in .ntwk.msn.net, which is omitted here for width.

9. Hop 7 is the odd one out: it has no three-letter role code, only del01-96cbe-1b. Inferred, not proven: this is an edge or peering device named from an inventory identifier rather than a backbone role code, which is consistent with it being the handover point from the reader’s ISP.
10. Honest limits of name reading, in order of importance: names are published voluntarily and may be absent; they may be stale after a change; the codes are one operator’s private scheme; and geolocation databases built on such names inherit every one of those errors.
11. Cross-check a name before relying on it. Compare the round-trip time with the distance the name implies. A hop named Mumbai answering in 2 ms from Delhi is a stale or wrong name, not a physics breakthrough.

WORDS27.10.6 remember these#

1. IATA code — the three-letter airport code — the International Air Transport Association location identifier, widely reused for network site names.
2. Site code — which building the router is in — the operator’s facility identifier, usually a city code plus a number.
3. Role code — what job the router does — a naming convention indicating the device’s place in the network hierarchy, private to each operator.
4. Bundle interface — several cables acting as one — a link aggregation group, called ae on Junos, Bundle-Ether on IOS-XR, Port-channel on IOS.
5. Subinterface — a logical slice of one interface — a numbered unit on a physical or bundle interface, often carrying one VLAN.

27.11 The trace as a story#

PLAIN27.11.1 in simple words#

1. Read the whole thing as one journey and it becomes memorable.
2. A packet leaves a laptop in a flat and reaches a plastic box on a shelf.
3. The box passes it to the provider’s access network, the equipment that serves the street and the neighbourhood.
4. It crosses the provider’s own core, several machines using addresses that only mean something inside that company.
5. It reaches the provider’s public edge, the point where that company touches the rest of the internet.
6. It is handed to Microsoft in Delhi.
7. It travels down the country to Mumbai, then inland to Pune.
8. And then nothing more is heard from it.

PLAIN27.11.2 a picture in your head#

1. Picture a parcel leaving a house and reaching the corner post box.
2. A van takes it to the local depot, then a bigger van to the regional hub.
3. At the hub it is handed to a different company entirely, with different vans and different paperwork.
4. That company flies it from Delhi to Mumbai, then drives it to Pune.
5. At Pune the tracking page stops. The parcel is somewhere in that building or somewhere beyond it, and nobody will say which.

Where this comparison breaks: a parcel is one object, and this is not. Twenty probes went out and twelve zones answered. Also, no van driver in this story was ever obliged to scan anything for you, and most of them did it as a courtesy that they can withdraw at any time.

PLAIN27.11.3 a worked example#

1. The journey in one diagram.

 flat  ->  home router  ->  ISP access  ->  ISP core
 [1]       192.168.0.1      172.31.0.17     172.26.x
                                            172.16.x
                                               |
                                               v
                                        ISP public edge
                                        137.97.29.249
                                               |
                                               v
   Microsoft DELHI  ->  MUMBAI  ->  PUNE  ->  silence
   104.44.196.187      104.44.5x     104.44.2x   * * *
      hop 7            hops 8, 9    hops 10-12  13 to 20

2. Now the distances. These are straight-line figures and are approximate.

3. Where the fastest possible figure comes from: light in glass fibre travels at roughly 200,000 km per second, which is about 5 microseconds per kilometre one way, so 10 microseconds per kilometre there and back.
4. Real cable never runs straight, and real routers add delay, so expect real figures well above the floor.

5. The typical figures are approximate working numbers for Indian domestic routes, not measurements from this session.

PLAIN27.11.4 what is really happening inside#

1. Notice what the story reveals about the reader’s provider: it did not hand the traffic to Microsoft locally. It carried it to Delhi first.
2. Notice also that Microsoft carried it a long way inside its own network, from Delhi to Pune, roughly 1,180 km in a straight line.
3. That is normal for a large content network. Once traffic is inside, moving it on private long-haul fibre is cheap and fully controlled.
4. Inferred, not proven: that the reader is physically near Delhi. The Delhi handover is equally consistent with a provider that backhauls traffic from elsewhere to a Delhi interconnection point.
5. What is proven is only the sequence of sites: the handover happened at a Delhi-named Microsoft device, and the last reply came from a Pune-named one.
6. The destination address itself, 20.207.73.82, is also a Microsoft address, so the trace was heading toward the same network the whole time from hop 7 onward.

TECHNICAL27.11.5 the engineer’s version#

1. The handoff at hop 7 is an interconnection: either a private interconnect or a public internet exchange port, between the reader’s ISP and AS8075.
2. A traceroute cannot distinguish those two. Both appear as one hop with the far side’s address.
3. Once inside AS8075, forwarding is Microsoft’s own decision. The Delhi to Mumbai to Pune ordering reflects the internal topology, and the ibr role code at hops 10 and 11 is consistent with inter-region long-haul devices.
4. Rough latency budget for the whole visible path, using floor values plus realistic equipment and routing overhead:

5. Total expected round trip at hop 12 is therefore roughly 40 to 80 ms, which is unremarkable for a domestic Indian path to a cloud edge.
6. This estimate cannot be checked against the reader’s capture, because that capture contains no timing data. Note that as a gap in the evidence rather than filling it in.

WORDS27.11.6 remember these#

1. Access network — the equipment serving your street — the last-mile aggregation between customer premises and the provider core.
2. Interconnection — where two networks touch — a private interconnect or internet exchange port between two autonomous systems.
3. Backhaul — carrying traffic to a bigger site before handing it off — long transport from access aggregation to a core or peering location.
4. Long-haul — a link between distant cities — high-capacity inter-city fibre, typically terrestrial DWDM within a country.
5. Latency floor — the fastest the distance allows — propagation delay at roughly 200,000 km per second in fibre, about 5 microseconds per kilometre.

27.12 What the silence after hop 12 means#

PLAIN27.12.1 in simple words#

1. Eight hop numbers, twenty-four probes, not one answer.
2. There are five plausible explanations, and they are not equally likely.
3. One. Microsoft’s inner network does not send these diagnostic replies to outsiders, by policy. Very likely.
4. Two. Those routers do send them, but only a few per second, and traceroute asked too fast. Very likely.
5. Three. The destination itself never answers this kind of probe. Very likely, and true of almost every large web service.
6. Four. Something on the way discarded the reader’s packets silently. Possible.
7. Five. Something discarded the replies on the way back. Possible.
8. The first three are ordinary behaviour that would look exactly the same on a perfectly working connection.
9. So the honest conclusion from the trace alone is: no conclusion.

PLAIN27.12.2 a picture in your head#

1. You knock on twenty doors down a corridor. Twelve open. Eight do not.
2. Behind those eight doors could be an empty room, a person who ignores knocks, a person already busy answering someone else, or a wall.
3. You will not learn which by knocking harder.
4. But you have learned something real: the corridor definitely continues to door twelve, and door twelve is in Pune.

Where this comparison breaks: knocking on a real door has one meaning. A traceroute probe carries a hop counter, and a device can forward it perfectly while refusing to answer, so a silent door here may be one your packets are sailing straight past.

PLAIN27.12.3 a worked example#

1. Ranked honestly, most likely first, for a trace toward a large cloud provider’s inner network.

2. Five of the six most likely explanations are not faults.
3. And note that explanation 6, the only real fault, is indistinguishable from the other five using this tool.
4. What the trace did contribute: it proved the traffic reached Microsoft’s network and travelled inside it as far as a Pune device.
5. That narrows the region enormously. It rules out the reader’s own router, the reader’s provider, and any question of a wrong address.
6. It does not identify the cause. Not even slightly.

PLAIN27.12.4 what is really happening inside#

1. The reason silence is uninformative is that all six causes produce the same observation at the reader’s end: no packet arrives.
2. To distinguish them you need an observation that cannot be produced by politeness, only by a working or a broken path.
3. That is what the curl test provided. It did not ask for a courtesy reply. It attempted a real connection to a real service on port 443.
4. A working path produces an answer to that within milliseconds. A refusing server produces a refusal. A missing route produces an error message.
5. The reader got none of those. Fifteen seconds, and nothing of any kind.
6. Then the same request, to the same address, over mobile data, succeeded immediately.
7. That pair of results is what settled it, and Chapter 28 works through exactly why.
8. Keep the two roles separate in your head: the traceroute describes the region, and the connection test establishes the fault.

TECHNICAL27.12.5 the engineer’s version#

1. Proven. No ICMP Time Exceeded arrived for TTL 13 through 20 within the configured wait, across 24 probes.
2. Not proven, and not supported by this data alone. That forwarding stops at hop 13; that any specific party filtered anything; that the destination is unreachable.
3. Discriminating experiments, from cheapest to most conclusive:

4. The rate-limiting hypothesis is testable directly: rerun with -q 1 and a pause between probes. If hops appear that were stars at -q 3, the cause was rate limiting, not filtering.
5. The MPLS hypothesis is testable by looking for a discontinuity: a sudden large RTT step across one hop boundary, or ICMP extensions carrying label information as specified in RFC 4950.
6. The direction question cannot be answered from one end at all. It requires a probe originated inside or near the destination network, which is what a looking glass or a remote measurement platform provides.
7. Final honest reading of the reader’s evidence, in one sentence: the traffic entered Microsoft’s network in Delhi and was last seen alive at a Pune device, after which it was discarded silently somewhere, by something, in a direction that this trace cannot determine.

WORDS27.12.6 remember these#

1. Discriminating test — an experiment whose outcomes differ by cause — a test designed so that competing hypotheses predict different observations.
2. Silent discard — dropping without notification — a filter or forwarding action producing no ICMP error, common on security policy denies.
3. Narrowing — reducing the region under suspicion — eliminating candidate causes without identifying the actual one.
4. Control test — the same test against a known-good target — a baseline run that separates environment-specific from target-specific behaviour.
5. Bidirectional visibility — seeing both directions — measurement from both ends, required to determine which direction of a path fails.

27.13 How to run a much better traceroute#

PLAIN27.13.1 in simple words#

1. The default traceroute is a 1987 tool with 1987 defaults, and it is the wrong tool for a modern web problem.
2. Four changes make it far better, and all four are one-letter flags.
3. Change the probe type to match the traffic that is actually failing. For a website, that means a real web connection to port 443.
4. Turn off name lookups. During a network fault, name lookups are slow or broken and they make the tool look stuck when it is not.
5. Shorten the waiting time, so a silent path finishes in seconds instead of minutes.
6. Then run the same command again against something known to work, so you have a comparison.
7. After that, run a tool that keeps probing, because one snapshot cannot show a fault that comes and goes.
8. Finally, get somebody else to trace back toward you, because you cannot see the return path from where you are standing.

PLAIN27.13.2 a picture in your head#

1. A doctor who takes one temperature reading, of one patient, with a broken thermometer, has learned very little.
2. Take the reading with the right instrument. Take it again on a healthy person to check the instrument. Take it repeatedly over time. And ask a colleague at the other hospital what they see.
3. Those four steps are exactly the four commands below.

Where this comparison breaks: a patient stays still. A network path changes between measurements, so two runs a minute apart can legitimately differ, and that difference is itself information rather than an error.

PLAIN27.13.3 a worked example#

1. The commands, for macOS, in the order to run them.

# 1. The real test: TCP to the port that is failing.
sudo traceroute -P TCP -p 443 -n -q 1 -w 2 -m 30 20.207.73.82

# 2. The control: identical flags, a target known to answer.
sudo traceroute -P TCP -p 443 -n -q 1 -w 2 -m 30 1.1.1.1

# 3. For comparison, the ICMP method Windows uses by default.
sudo traceroute -I -n -q 3 -w 2 -m 30 20.207.73.82

# 4. The classic UDP method, which needs no special permission.
traceroute -n -q 3 -w 2 -m 30 20.207.73.82

# 5. Continuous, with per-hop loss, over TCP port 443.
sudo mtr -T -P 443 -n --report --report-cycles 100 20.207.73.82

2. What each flag does.

3. Why -q 1 with TCP: each TCP probe leaves a half-open connection at the target until it times out. Sending fewer is both faster and politer.
4. Why -n matters most during a fault: without it, every hop triggers a reverse name lookup, and if DNS is affected the tool appears to hang.
5. Worst case with these flags is 30 hops times 1 probe times 2 seconds, which is 60 seconds. With the defaults it is 450 seconds.

PLAIN27.13.4 what is really happening inside#

1. A looking glass is a web page or login service run by a network operator that lets outsiders run a few read-only commands on that operator’s own routers.
2. Typically you can run ping, traceroute, and queries about which routes that operator knows.
3. Operators provide them so that other engineers can diagnose problems involving their network without having to phone anybody.
4. Why it matters here: a looking glass inside or near the destination network can trace back toward the reader’s address.
5. That gives you the return path, which is the one thing a traceroute from your own machine can never show you.
6. Where to find them: most large operators publish one, and there are public directories of them maintained by the peering community. Measurement platforms such as RIPE Atlas do the same job from thousands of probes.
7. A cheaper version of the same idea, which the reader already had available: run the test from a phone on mobile data. That is a second viewpoint on a completely different path.

TECHNICAL27.13.5 the engineer’s version#

1. Flag reference, quoting the macOS manual page:

2. -P TCP and -I require raw sockets, so run them with sudo.
3. mtr flags used above: -T selects TCP probes, -P 443 sets the port, -n disables name lookup, --report prints a summary rather than a live display, --report-cycles 100 sets how many rounds to run.
4. Reading mtr output correctly is the single highest-value skill here: loss at an intermediate hop that does not persist to the final hop is ICMP rate limiting, not packet loss. Only loss that continues to the last hop is real.
5. For flow-stable path discovery use paris-traceroute, or scamper from CAIDA in its multipath detection mode, or dublin-traceroute. These map the parallel paths instead of blending them.
6. For a reproducible record, save output with a timestamp, for example by redirecting to a file named with the date, and keep the control run alongside it. A trace without a control run is much weaker evidence.
7. Recommended minimum evidence set for reporting a path fault to a provider: the TCP traceroute to the failing target, the identical traceroute to a working control, an mtr report of at least 100 cycles, the curl -v output showing the failure, and the same curl from a second network.

WORDS27.13.6 remember these#

1. Looking glass — a public window onto an operator’s routers — a read-only interface offering ping, traceroute and BGP route queries.
2. Control run — the same test on something known to work — a baseline measurement isolating target-specific behaviour from environmental noise.
3. Reverse traceroute — a trace run toward you from elsewhere — a measurement originated near the destination, revealing the return path.
4. RIPE Atlas — a global network of small measurement probes — a distributed active measurement platform operated by the RIPE NCC.
5. Half-open connection — a connection request never completed — a TCP session left in SYN-RECEIVED at the target, consuming state until it times out.

27.98 Common wrong ideas#

1. Wrong: traceroute asks the network to report the path. Right: there is no such request. It provokes error messages by sending packets designed to die.
2. Wrong: the trace shows the route your traffic takes. Right: it shows a blend of routes taken by dozens of separate probes, each of which may be hashed onto a different link.
3. Wrong: TTL means time, so a slow link uses more of it. Right: it counts hops. One router costs exactly one, whether it holds the packet for a microsecond or a second.
4. Wrong: stars mean the packet stopped there. Right: stars mean no reply arrived. The packet may have sailed straight through a router that simply does not answer.
5. Wrong: several addresses on one hop is a bug. Right: it is load balancing across parallel links, and it is a sign of a well-built network.
6. Wrong: subtract two hops’ times to get the latency of the link between them. Right: each time is a full round trip on a different return path, with different reply-generation delays. The subtraction is meaningless.
7. Wrong: a slow hop in the middle is causing your problem. Right: if the hops after it are fast, that router is merely slow at generating replies, and its forwarding is fine.
8. Wrong: the last responding hop is where the fault is. Right: it is the last hop that was willing and able to answer, which says nothing about what lies beyond it.
9. Wrong: the reader’s traceroute proved a fault. Right: it proved nothing of the kind. Every line of it is compatible with a healthy network. The curl timeout plus success on mobile data is what established the fault.
10. Wrong: router hostnames are a standard you can rely on. Right: they are each operator’s private convention, published voluntarily, sometimes stale, and always to be cross-checked against measured times.

27.99 Chapter summary in 20 lines#

1. IP has no field that records a path, so there is no way to ask for one.
2. Traceroute works by abusing the TTL field, a safety counter meant only to stop packets looping forever.
3. TTL is 8 bits, counts hops not seconds, and every forwarding router must decrement it by exactly one, per RFC 1812 section 5.3.1.
4. Common starting values are 64 on Linux and macOS, 128 on Windows and 255 on much network gear, which lets you guess a sender’s system, weakly.
5. Send TTL 1 and the first router discards the packet and returns ICMP Time Exceeded, type 11 code 0, sourced from its own interface address.
6. Repeat with TTL 2, 3, 4 and so on. Three probes per hop is the default, and each round-trip time is measured per probe.
7. Van Jacobson wrote traceroute in 1987 at Lawrence Berkeley Laboratory, from a suggestion by Steve Deering.
8. The classic method sends UDP to ports from 33434 upward; Windows tracert sends ICMP echo; TCP traceroute sends a SYN to a real port such as 443.
9. TCP to 443 gets furthest through modern filtering because it looks exactly like the traffic the path is built to carry. It is the right tool here.
10. Multiple addresses on one hop are equal-cost multi-path load balancing. The reader’s hops 5, 6, 8, 9 and 11 all show it, and all are normal.
11. Classic traceroute changes its destination port per probe, which changes the load-balancing hash. Paris traceroute, from 2006, fixes exactly that.
12. Stars mean only that no reply arrived. Causes include policy, rate limiting, filtering, MPLS hiding hops, and a silent destination.
13. Linux rate limits ICMP Time Exceeded to one per second per target by default, which alone explains most partial star lines.
14. Stars at the end of a trace are extremely common and prove nothing. The reader’s hops 13 to 20 are consistent with six different causes.
15. Round-trip times can go down as you go further out, because they include reply-generation time on a slow control plane and an unseen return path.
16. You cannot subtract two hops’ times to get a link latency.
17. The reader’s trace reads: flat, home router, ISP access, ISP core with one public address at hop 3, Microsoft’s Delhi edge, Mumbai, Pune, silence.
18. Names decode as ntwk.msn.net for Microsoft’s backbone, IATA city codes del, bom and pnq for Delhi, Mumbai and Pune, role codes ibr, rwa and owr, and interface prefixes ae, be and po all meaning bonded link bundles.
19. Say it plainly: the reader’s traceroute alone would not have proved a fault. It narrowed the region and nothing more.
20. What settled it was a real connection attempt that received no response of any kind, and the same request succeeding over mobile data.