How to Read a Traceroute: Stars, Latency Jumps, Asymmetric Paths, and MPLS Hops
Every afternoon around three, the video calls between a bookkeeper working from home outside Chicago and the firm's office in London started to stutter. Voices froze mid-word and faces dissolved into blocks of color, and after a week of it the bookkeeper opened a command prompt, ran tracert against the firm's video server, and got a screen of output that looked damning. Hop 6 took 244 milliseconds while its neighbors took around 30 and 100, two rows said "Request timed out," and a router whose name placed it in New York answered in exactly the time it takes to reach London. The bookkeeper sent a screenshot to the internet provider with a note blaming hop 6. Every one of those details turned out to be normal, and the real problem sat on the very first line, where anyone who knew how to read a trace could have found it.
Data crosses the internet the way a parcel moves through a delivery network, handed from one sorting center to the next until it arrives. The sorting centers are routers, computers whose whole job is to read the destination on each piece of data and pass it one step closer, and each handoff is called a hop. Every device on a network has an IP (Internet Protocol) address, the number other devices use to reach it, and every router along the way has addresses of its own. Traceroute is a tool that lists those routers in order, along with how long each took to respond, and it comes built into Windows (where it's called tracert) and macOS, installs in a moment on Linux, and runs from any browser through our free traceroute tool. A trip from a house near Chicago to a server in London usually passes through ten to twenty hops run by three or four different companies, and traceroute gives each hop one numbered line.
Traceroute finds those routers with a clever use of a safety feature. Each packet, the small unit that data travels in, carries a counter called TTL (time to live), and every router that handles the packet subtracts one from it before passing it along. When the counter hits zero, the router throws the packet away and sends a short note back to the sender saying the packet expired in transit, which keeps a packet caught between two misconfigured routers from circling forever. That note travels by ICMP (Internet Control Message Protocol), the behind-the-scenes messaging system networks use for errors and status reports, and the same protocol ping uses. Traceroute sends its first probe with a TTL of 1, so the very first router discards it and replies, revealing its address. The next probe gets a TTL of 2 and expires one router further out, and the process repeats until a probe reaches the destination and the destination itself answers.
By default, traceroute sends three probes to every hop and times each one, which is why each line shows three numbers in milliseconds (thousandths of a second, written ms). Each number is a round trip: the probe's journey out to that router plus the note's journey back to you. The versions differ in what they send. Windows tracert uses the same echo requests as ping, while traceroute on macOS and Linux sends UDP (User Datagram Protocol, a lightweight way of sending data without setting up a connection) to high-numbered ports where nothing should be listening, and the destination signals that a probe arrived by replying that nobody is home on that port. The macOS and Linux versions can switch to other kinds of probes, and on Linux they can even imitate ordinary web traffic using TCP (Transmission Control Protocol, the connection-based method most internet traffic uses), which matters when a firewall lets one kind through and drops the rest. On Linux, install the traceroute package first if the command isn't found.
Running a trace (swap meet.example.com for the site you want to reach)
# Windows (Command Prompt)
tracert meet.example.com
pathping meet.example.com traces once, then measures each hop for a few minutes
# macOS and Linux (Terminal)
traceroute meet.example.com
traceroute -I meet.example.com ICMP probes like Windows (on Linux, put sudo in front)
sudo traceroute -T -p 443 meet.example.com Linux only: TCP probes to the web port
mtr meet.example.com a live trace that keeps updating (install it first)The sudo in front of a command runs it with administrator rights, which Linux requires for the ICMP and TCP modes. The last command, mtr (short for My Traceroute), repeats the trace every second and keeps running totals, and it comes up again below. Here is the trace the bookkeeper sent, with the addresses and names swapped for ranges reserved for examples like this one.
The bookkeeper's trace from outside Chicago to a video server in London
Tracing route to meet.example.com [203.0.113.40]
over a maximum of 30 hops:
1 2 ms 1 ms 2 ms 192.168.1.1
2 11 ms 10 ms 12 ms 100.64.0.1
3 12 ms 12 ms 13 ms 10.172.4.9
4 13 ms 13 ms 14 ms ae-4.cr1.chi1.example.net [198.51.100.17]
5 32 ms 31 ms 32 ms ae-1.cr2.nyc2.example.net [198.51.100.42]
6 244 ms 239 ms 251 ms be-7.br1.nyc2.example.net [198.51.100.61]
7 * * * Request timed out.
8 101 ms 100 ms 101 ms et-0-1.cr1.nyc3.example.org [192.0.2.130]
9 101 ms 101 ms 102 ms et-2-0.cr4.lon1.example.org [192.0.2.134]
10 * * * Request timed out.
11 102 ms 101 ms 102 ms meet.example.com [203.0.113.40]
Trace complete.Reading starts at the top. Hop 1 is the home router, the box that connects the house to the internet, and like most home routers it uses a private address, 192.168.1.1, that works only inside the house; the router swaps in a public address on the way out, the job our NAT (Network Address Translation) explainer covers. Hops 2 and 3 belong to the ISP (internet service provider), and they also show addresses that would never work on the open internet. The 100.64.0.1 at hop 2 comes from a range set aside for carrier-grade NAT, which providers use to share one public address among many customers, and the 10.172.4.9 at hop 3 comes from the same private ranges offices use inside their buildings; our guide to the reserved IP ranges explains both. Private addresses in the middle of a trace are common and harmless, since a router can tell you its address without that address being reachable from anywhere else.
Each line shows the address of the router's connection that faced you when the probe arrived, so a router with forty connections appears under whichever one your traffic came in on. Those connections between routers are usually tiny subnets with room for just two addresses, the subject of our guide to point-to-point links, and a router whose links have no addresses of their own answers from a single address it keeps as its identity. From hop 4 on, the lines carry names as well as numbers, because tracert looks up each address in DNS (Domain Name System, the internet's directory of names and addresses) using a reverse lookup, known as a PTR (pointer) record. Network operators pack these names with clues. In ae-4.cr1.chi1.example.net, chi1 stands for a building in Chicago, cr1 for core router number one, and ae-4 for the bundle of cables the probe arrived on, so hop 4 sits in Chicago and hop 5, with nyc2 in its name, sits in New York.
Treat those names as hints, because someone typed them by hand and nothing forces anyone to update them when equipment moves, a kind of drift our guide to keeping DNS in sync with your IP inventory describes in detail. Patterns like site, role, and number show up in well-run networks of every size, as our post on naming conventions explains. Our free reverse DNS tool looks up the name for any hop, and the ASN lookup tool shows which company runs the network an address belongs to, using its ASN (autonomous system number), the ID number every independently run network on the internet holds.
The times grow as the trace moves outward, and most of that growth is distance. Light in fiber-optic cable travels at about 200,000 kilometers per second, roughly two-thirds of its speed in empty space, which works out to about one millisecond of round trip for every 100 kilometers of cable. Cables follow highways, rail lines, and seabeds instead of straight lines, so Chicago to New York, about 1,150 kilometers apart on a map, typically costs 18 to 20 milliseconds there and back, and crossing the Atlantic from New York to London adds roughly 70. In the bookkeeper's trace, the step from 13 milliseconds at hop 4 to 32 at hop 5 is the trip to New York, and the step from the low thirties to 101 is the ocean. Distance makes a good first test for any jump: if the names say the trace just crossed a continent or an ocean, tens of milliseconds is simply the length of the cable.
Hop 6 looks like the culprit, at 244 milliseconds between hops in the thirties and around 100, and it's almost certainly harmless. A router does two very different jobs. It forwards traffic using specialized chips built to move enormous numbers of packets per second, and it writes those expiry notes using a small general-purpose processor that also runs the router's own software and treats your probe as the lowest item on its list. When that processor is busy, the note waits and the hop looks slow, even as the traffic passing through it, including every probe headed for hops 7 through 11, moves at full speed. The test is simple: a delay that matters shows up at that hop and at every hop after it, all the way to the destination. Hop 6's extra 200-plus milliseconds vanish by hop 8, so the router at hop 6 is slow to talk about itself and quick to forward everything else, while a real slowdown looks like a staircase, with times that step up at one hop and stay up through the end.
The two rows of stars follow the same logic. Each asterisk is a probe that got no reply before traceroute stopped waiting, and Windows adds "Request timed out" when all three come back empty. Routers skip the expiry note for several ordinary reasons. Some carriers configure routers to never send one, to keep the inner layout of their network private, and many routers cap how many notes they send per second and quietly drop the rest, which can turn one probe into a star while its two siblings get answers. Firewalls often discard the probes outright, much as a Windows computer ignores ping by default, a habit our guide to why a ping sweep misses half your network explains. In the bookkeeper's trace, hop 7 is the first router of the carrier that crosses the Atlantic, and hop 10 belongs to the London data center that hosts the video server, and both keep quiet as a matter of policy. Hops answered after each of them, which proves the probes passed straight through.
So stars followed by answering hops are harmless, and only the last lines of a trace deserve worry. A trace that ends in line after line of stars has stopped at something that won't answer, which could mean a real break or a destination that ignores traceroute's style of probe. Many web servers drop the UDP probes that macOS and Linux send by default while answering ping and web traffic happily, so before concluding anything, run the trace again with ICMP probes (-I) or TCP probes to port 443 (-T -p 443), and if the destination answers either one, it's reachable and simply picky about what it replies to. Short codes can also appear in place of a time on macOS and Linux, and Windows spells them out in words: !H means a router reported that the destination host can't be reached, !N means it has no route to the destination's whole network, and !X means a filter rejected the probe on purpose. Those codes are definite answers, and the router that sent one is the place to start looking.
Every time in a trace includes the trip home, and the trip home can take a completely different road. Traceroute only discovers the path from you to the destination, because the routers it hears from are the ones where its probes expired. The expiry notes travel back by whatever route the replying router's network prefers, and networks choose for their own reasons. The internet is a patchwork of tens of thousands of independently run networks, each with its own ASN, that agree to carry one another's traffic, and many of them hand traffic off to the next network at the nearest exit to save the cost of carrying it themselves, a habit engineers call hot-potato routing. As a result, the path to London and the path back often cross the ocean on different cables, through different cities, and sometimes through different companies. When a hop looks slow, its expiry note may simply have taken the long way home.
Paths can split going out, too. Large networks often run two or four parallel links between the same pair of cities and spread traffic across all of them, a setup called equal-cost multipath, and because traceroute's probes differ slightly from one another, back-to-back probes can take different branches. On macOS and Linux this shows up as two or three addresses printed on the same line, and from one run to the next, a hop can switch between them. When the branches differ in length, a trace can even seem to visit one router twice or skip a step. A variant called Paris traceroute keeps every probe's identifying details the same so they all stay on one branch, which gives a steadier picture of a network that balances its load.
Since a single trace shows only half the journey, the most useful thing to add is a second trace taken from somewhere else. Many providers run looking glasses, public web pages that let anyone run a traceroute from the provider's own routers, so you can trace back toward your own address from near the destination and compare the two directions. Our free traceroute tool offers another vantage point: it runs from IPCraft's server in Ashburn, Virginia, one of the busiest data center hubs on the internet, traces up to 20 hops with three probes each, and shows each hop as a numeric address you can feed into the reverse DNS tool. If our server reaches a site cleanly while your own trace stalls, the trouble probably sits between you and the wider internet. If both traces stall in the same company's network, the problem is likely theirs, and you have evidence from two places to show the provider. When the destination replies from a different address than the one the tool asked for, it says so and explains that this is normal for anycast, where many servers around the world share a single address, a common setup for CDNs (content delivery networks, which keep copies of websites close to their visitors) and public DNS services. Large sites also send each visitor to a nearby server, so your trace and ours may end at different addresses for the same name.
That leaves hop 8, the router whose name says New York and whose time says London. Many large carriers move traffic across their networks using MPLS (Multiprotocol Label Switching), which works something like an express lane. The router at the entrance to the carrier's network reads each packet's destination once, attaches a short label to it, and sends it down a tunnel toward the exit, and the routers inside the tunnel read only the label. That lets the carrier plan routes precisely, and it means the routers in the middle don't need to know the way to every address on the internet. The tunnel shows up in a traceroute in one of two ways, depending on how the carrier sets it up.
Some carriers stop the TTL counter from dropping inside the tunnel, so the whole tunnel counts as a single hop. The trace jumps from a router in New York straight to one in London with nothing in between, as though the ocean held no equipment at all, and the time jumps by the entire crossing at once. Other carriers let the counter drop at each router inside, and those routers then have to send expiry notes even though they don't know the way back to you. They hand the note to the tunnel, which carries it forward to the exit router, which sends it home, so every note from inside the tunnel travels to the far end and back. Every hop inside the tunnel therefore shows about the same time, the time to the tunnel's exit. Hop 8 is a router in New York inside a tunnel that ends in London, reporting London's 101 milliseconds, and a run of hops with nearly identical times, often much higher than the cities in their names would suggest, usually means a tunnel.
Some versions of traceroute will show the labels themselves. Routers that support a standard extension attach the label to their expiry notes, and traceroute -e on Linux, or mtr with its own -e option, prints it beside the hop, so a line marked MPLS:L=24005 tells you the probe was inside a tunnel when it expired. Windows tracert doesn't display labels, but the matching times give tunnels away regardless.
Tunnels closer to home have a similar effect. A VPN (virtual private network) wraps your traffic in an encrypted tunnel to the office or to the VPN company's servers, so a trace from a laptop with a VPN switched on usually shows the tunnel's far end as one of the first hops and nothing of the home router and provider equipment the traffic actually crossed. Our guide to VPN and overlay network addressing explains how the tunnel takes over. If a trace looks implausibly short, check whether a VPN is running.
Everything above applies to IPv6 (Internet Protocol version 6, the newer addressing system with much longer addresses), where the counter goes by the name hop limit instead of TTL. Run tracert -6 on Windows, traceroute6 on macOS, or traceroute -6 on Linux, and expect the IPv6 path to differ from the one taken by IPv4 (version 4, the familiar four-part addresses), since many networks route the two separately. Our guide to IPv6 address planning covers the addresses themselves.
What each pattern in a trace usually means
pattern usually means
stars mid-trace, later hops answer a router that stays quiet or limits replies (harmless)
one slow hop, later hops normal a router slow to reply, fast to forward (harmless)
times step up at a hop and stay up real delay at that hop or the link before it
a big jump between cities or oceans distance, about 1 ms of round trip per 100 km
several hops with nearly identical times an MPLS tunnel sending replies via its exit
one hop that spans a huge distance an MPLS tunnel hiding the routers inside
two or three addresses on one line traffic balanced across parallel paths
stars to the end, but ping works a destination that drops the probes; try -I or -T
!H, !N, or !X a router reporting the destination unreachable or blocked
loss at one hop only (mtr, pathping) a router skipping replies (harmless)
loss from one hop through the end real loss starting at that hopA single trace is a snapshot, so for problems that come and go, like the bookkeeper's afternoon stutter, a tool that keeps probing works better. The mtr tool runs traceroute continuously and keeps a running tally of lost probes and average times for every hop, and Windows includes pathping, which traces once and then spends a few minutes measuring every hop. Read their loss figures with the same rule as everything else: loss at a single hop that doesn't carry on to the hops after it is a router skipping replies, and loss that begins at one hop and continues through every hop to the destination is real. When you contact a provider, send a report that covers several minutes while the problem is happening, from both directions if you can, and point to the hop where the loss or delay starts and stays, which gives their engineers something they can act on.
Inside your own network, traceroute is just as useful and often easier to act on, because the routers belong to you. A trace from a branch office to headquarters that stalls at 10.20.255.6 points at a specific device, as long as someone can say which device that is, and the answer depends on records: which subnet holds the address, what that subnet connects, and which router owns it. In IPCraft, an IPAM (IP address management) system, searching for an address takes you to the subnet that contains it, along with its location and VLAN (virtual local area network), and to the record for the address itself, with its hostname, description, and reverse DNS name, so a hop in a trace becomes a named router in a known closet within seconds. Record your point-to-point links and router identity addresses the same way you record servers, since those are the addresses traces report. Our concepts guide covers subnets, records, and locations, and our guide to tracing an IP address back to a physical device follows the trail from the record to the switch port.
Back outside Chicago, the provider's reply was polite and specific: hop 6 was a busy router, hop 7 belonged to a carrier that never answers, and hop 8 sat inside a tunnel to London. They suggested running pathping during a bad call, and the next afternoon the bookkeeper did. Loss started at hop 1, the home router, at around 9 percent, and carried through every line to London unchanged, because hop 1 was a Wi-Fi connection reaching up two floors from a router in the basement, and the first trace had been taken on a calm morning. A long network cable fixed the calls that week, and reading the first line before the sixth would have saved the week before it.