DNS is the single point of failure nobody monitors: lessons from SkyBell, Marketo, Facebook and Dyn

On the morning of 2 November 2026, SkyBell’s video doorbells started glowing blue instead of green, its app logged everyone out, and its website and support site stopped loading. The company’s status log for the next thirty hours tells the story in four lines: an outage with the cloud server; services “may be both server and DNS related”; “an issue with DNS and DNS server settings”; and, at 4:15 pm Pacific on 3 November, services “slowly coming back online” in waves. The servers were there the whole time. The names that pointed to them were not.

DNS, the system that turns a name into an address, is the step before every other step. A request for your site cannot reach a load balancer, a CDN, a server or a failover region until a resolver has answered the question of where to send it. Everything you have built to stay up sits behind that answer, and the answer depends on a handful of records, one registration and a provider most companies have never tested losing.

This article looks at four famous outages with that one cause, sorts them into the four ways DNS fails, and sets out six controls that would have prevented most of them. None of the controls cost more than an afternoon.

Four outages, one cause

Marketo, 25 July 2017: the registration lapsed. Marketo’s domain most likely expired on 23 July, by ThousandEyes’ reading of the registry data. Two days later, at about 4 am Pacific, traffic for marketo.com began flowing to a parking address that dropped it. Every customer whose marketing forms, tracking links and emails ran through the company’s domain lost them for six hours, until 10:08 am. The company put it down to a combination of human and process error, and later reported 99.17% uptime for the month once the six hours were counted. A company whose product is sending email was unreachable because a renewal notice went unanswered.

Sorenson Communications, June 2016: the registration lapsed, and lives depended on it. Sorenson’s video relay service lets deaf and hard-of-hearing users make calls, including to 911. Its domain registration was deactivated on 6 June 2016 and service was not fully restored until 8 June. The FCC’s enforcement bureau found the outage preventable; the September 2017 settlement came to roughly $3 million, most of it restitution to the relay fund.

Facebook, 4 October 2021: the route to the name servers vanished. A command intended to measure backbone capacity instead disconnected every data centre from the internet. Facebook’s DNS servers are designed to stop advertising their addresses when they cannot reach the data centres behind them, on the sound principle that a name server that cannot see the service should not answer for it. With the whole backbone gone, every one of them withdrew. In the engineering team’s words, the DNS servers became unreachable even though they were still operational. Facebook, Instagram and WhatsApp were gone for about six hours, and the engineers who could fix it had to travel to the sites and get through physical security first.

Dyn, 21 October 2016: the provider was attacked. A Mirai botnet of compromised cameras and routers hit Dyn’s managed DNS in three waves from 11:10 UTC, and Twitter, Netflix, Reddit, Spotify, GitHub, PayPal and around 1,200 other sites and services became unreachable for much of the day, in the largest DNS-driven outage on record. Dyn’s own infrastructure never failed completely. The sites that stayed up, or came back first, were the ones already serving their zones from a second provider as well.

Cloudflare 1.1.1.1, 14 July 2025: a setting was wrong. The world’s second-largest public resolver stopped answering for 62 minutes because of a configuration change made five weeks earlier for an unrelated service, which lay dormant until a later change activated it. Cloudflare’s report is explicit that it was not an attack. Any device set to use only 1.1.1.1 could not resolve anything for an hour; devices with a second resolver configured barely noticed.

The four ways DNS fails

Sorted, the stories reduce to four mechanisms, and each has a different control.

1. The registration lapses. The registry stops publishing your name servers, or the registrar points the name at a parking page. Cause: a renewal notice to a mailbox nobody reads, an expired card on auto-renew, a departed employee who owned the account. Marketo, Sorenson, and a long list of others including Foursquare in 2010 and Sony Online Entertainment in 2014.
2. The name servers become unreachable. The servers run, but the network path to them is gone: a withdrawn route, a firewall change, a data-centre outage that takes the authoritative servers down with the site because both live in the same place. Facebook, and most small companies whose DNS is hosted on the same box as the website.
3. The DNS provider fails. A denial-of-service attack, an outage at the provider, or an account compromise. Dyn in 2016; the CISA emergency directive of January 2019 was issued after a campaign of registrar-account hijacks that redirected government domains to attacker-controlled servers.
4. A record or setting is wrong. A typo in an IP address, a change made to the wrong zone, a TTL set to a day on a record that needs to change in minutes, a resolver setting on the servers themselves. SkyBell this month, Cloudflare’s resolver last year, and the majority of the DNS incidents that never make the news because they last twenty minutes.

Why recovery is slower than the fix

A DNS fix rarely takes effect when you make it. Every resolver that asked for your name during the failure cached whatever it got, and keeps serving that answer until the record’s time to live expires. A record with a 24-hour TTL that lapses or goes wrong can keep some users on the bad answer for a day after you have corrected it. Marketo’s six hours were partly the registrar’s processing time and partly the world’s caches catching up.

The control is cheap. Keep the TTL on records you may need to change in a hurry, the website’s A or CNAME record and the mail server’s MX, at five to fifteen minutes. Resolution is no slower in practice, because resolvers refresh in the background, and a wrong answer now clears in the time it takes to make a coffee. Raise the TTL before a planned change if you want to reduce query load, and lower it again a day before the next one.

Six controls, in order of how often they would have helped

1. Auto-renew, on a card that will not expire, to a mailbox that is read. The registrar account should use a shared role address, such as domains@ your company, that forwards to two people, not a founder’s personal email. Put the domain’s expiry date and the card’s expiry date in the company calendar with reminders at ninety and thirty days. Renew for the longest term the registrar offers. This control alone covers Marketo, Sorenson, Foursquare and Sony.

2. Registrar lock and multi-factor authentication on the registrar account. The lock blocks transfers and name-server changes until explicitly released. Multi-factor authentication on the account stops the credential theft that CISA’s directive responded to. Both are switches in the registrar’s control panel.

3. Authoritative DNS on more than one network. At minimum, name servers that do not share a data centre, a provider or a route with your website. For a site that matters, two providers serving the same zone, kept in sync by zone transfer or a small script. This is the Dyn lesson, and the Facebook lesson: Facebook’s name servers were correctly built to withdraw when they lost sight of the service, but every one of them lost sight at once because they all depended on the same backbone.

4. Low TTLs on records you may need to change, higher on the rest. Five to fifteen minutes on the website and mail records. The root and name-server records can stay at hours.

5. A change process for DNS records. Two people for any change to a production zone, a note of what changed and when, and a check from outside the network immediately afterwards. Most wrong-record outages are a change nobody double-checked made by someone in a hurry.

6. Monitoring that would notice. Three layers. An external uptime check every minute resolves the name on every run and alerts when resolution fails, which is the layer almost everyone has and almost everyone has pointed only at the home page. A calendar reminder for expiry, which is the layer almost nobody has. And for names the business depends on, a scheduled query to each authoritative server directly, alerting when any stops answering or disagrees with the others, which catches a provider failure before it reaches your users. Our guide to how often monitoring should check covers the interval; for DNS, one minute is the right answer because the caches make every minute of a bad answer last longer.

The recovery runbook, written before you need it

Because a DNS failure stops your own email and status page as well as your site, the runbook cannot live only on those systems. Keep a printed or offline copy with: the registrar, its account email and who holds the second factor; the DNS provider and its support line; the current zone file exported monthly; the TTLs; and the order to check things in, which is registration first, then the name servers’ reachability, then the records, then the resolvers your own servers use. SkyBell’s thirty hours and Facebook’s six were both lengthened by the people with the fix being unable to reach the systems to apply it.

Where BCDRPulse fits

Every BCDRPulse uptime check resolves your verified hostname from Cloudflare’s network before it connects, so a name that stops resolving fails the check and raises the same alert as a dead server: every five minutes on the Free plan, every minute on Pro and Business, by email and webhook. A security scan of the hostname reports the certificate and header state the world sees, which is the fastest way to confirm after a DNS change that users are reaching the server you meant. The platform does not watch registrar expiry dates or compare authoritative servers; those are the calendar entry and the script above, and they take an hour to set up once.

Checklist

• Registrar account on a shared, monitored mailbox with multi-factor authentication and registrar lock on every domain
• Auto-renew on, longest term, card expiry and domain expiry in the shared calendar with reminders
• Name servers on at least two independent networks, or two providers in sync
• TTLs of five to fifteen minutes on the website and mail records
• Two-person change rule for production zones, with an external check after every change
• External uptime checks every minute on the hostname, alerts to two people
• Printed recovery sheet with registrar, provider, zone export and the order of checks

The bottom line

Marketo’s outage cost six hours and a reputation; Sorenson’s cost two days and three million dollars; Dyn’s took a tenth of the internet’s biggest names offline for an afternoon; Facebook’s and SkyBell’s showed that a company can have every server running and still be gone. In every case the fix was known and cheap, and in every case nobody had treated the name as part of the infrastructure. It is the first part. Treat it that way for an afternoon and it stops being the single point of failure.