Cloudflare 524 Error: But the Page Is Working — Why It Happens
A Cloudflare 524 means Cloudflare gave up waiting for your origin to send a response within its 100-second limit — it is not a verdict that the page is broken. The page can load fine for real users and still 524 for Googlebot or in your logs, because users are served from Cloudflare's edge cache while the crawler's request is forced back to an origin that cannot answer in 100 seconds. Both facts hold at once: the cache hides the slowness from humans, and the crawl exposes it.
This is the exact contradiction a Stack Overflow question has been stuck on since 2015: "I received 524 error in Google webmaster tool but the page is working fine. let me know why did cloudflare throw this 524 error ?" It has 9,927 views, a score of 7, and a single answer — posted nine years later — that just points at another question and shrugs about "connection issues." Nobody has answered the contradiction itself. This post does.
What a 524 actually means
Cloudflare's documentation calls error 524 "A timeout occurred" and describes a first-byte deadline, not a whole-page deadline. Cloudflare connects to your origin, sends the request, and waits for the origin to return HTTP response headers. If those headers have not arrived within 100 seconds, Cloudflare drops the connection and serves the 524 page. The default is 100 seconds on every plan; Enterprise customers can ask Cloudflare to extend it to 600 seconds on specific paths.
Two details trip people up. First, the clock runs from the request reaching the origin to the first byte of the response — not the last byte of the body. A page that streams its body slowly after quick headers never 524s; a page stuck in a database query for 101 seconds before emitting headers does. Second, the 524 is Cloudflare's verdict, not your origin's: the origin is usually still working when Cloudflare has already given up.
Why the page works for you but 524s for Google
The insight that makes the contradiction disappear: "the page" is not one thing. It is a cacheable asset sitting in front of an origin, and those two layers can be in opposite states at the same instant. A real user hits a warm edge cache or a fast origin path; Googlebot hits a cold cache and a slow origin path. Same URL, same minute, two different results.
Five mechanisms produce this, and you usually have more than one active at a time:
- Cache hit vs uncached crawl. Real users get the HTML from Cloudflare's edge, which never touches your origin and so can never time out. Googlebot often misses: it crawls on its own schedule, requests URLs humans never visit, and can arrive right after a cache expiry, forcing a slow origin fetch.
- Cold starts. Serverless and containerized origins pay a cold-start tax on the first request after idle. Crawlers hit at unpredictable, low-traffic hours — exactly when the origin is cold.
- TTFB spikes. A lock, a saturated connection pool, a stalled third-party API — any of these can push time-to-first-byte past 100 seconds intermittently. Users ride the fast path; the crawler lands inside the spike.
- Long-running synchronous work. Exports, reports, image processing — anything that runs inside the request path can legitimately exceed 100 seconds. Users trigger it rarely and wait it out; a crawler hits it once and it 524s.
- Partial responses. Because 524 is a first-byte deadline, an origin that stalls mid-body can leave Cloudflare holding no headers — a 524 even though the page would eventually finish.
How to fix it
The fix is not to raise timeouts — on free and Pro plans you can't, and on Enterprise a bigger timeout just masks the problem. The fix is to stop making a crawler's request depend on a slow origin. Two moves do that: cache the HTML at the edge, and bring the origin's TTFB down.
Edge-caching HTML means a crawler's request is answered from Cloudflare's cache — no origin round-trip, no 100-second clock. For public or static pages, a Worker that caches HTML for known crawlers is the surgical option: humans pass through untouched, bots get a cached copy, and a cold crawl never reaches a slow origin.
export interface Env {
ORIGIN_HOST: string; // the origin Cloudflare fronts, e.g. "app.example.com"
}
const BOT_UA =
/googlebot|bingbot|slurp|duckduckbot|baiduspider|yandex|facebookexternalhit|twitterbot|ahrefsbot|semrushbot/i;
const HTML_TTL_SECONDS = 60 * 60;
export default {
async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
const url = new URL(request.url);
const userAgent = request.headers.get("user-agent") ?? "";
// Humans pass straight through — only crawler HTML is cache-managed.
if (!BOT_UA.test(userAgent)) {
return passthrough(request, env);
}
// The cache is GET-only: a bodyless HEAD must never poison the GET entry.
if (request.method !== "GET") {
return passthrough(request, env);
}
// Never let personalized or authenticated crawler responses into the shared cache.
if (request.headers.has("cookie") || request.headers.has("authorization")) {
return passthrough(request, env);
}
const cacheKey = new Request(url.href, {
method: "GET",
headers: { accept: "text/html" },
});
const cache = caches.default;
try {
const hit = await cache.match(cacheKey);
if (hit) {
const served = withSecurityHeaders(stripCookies(hit));
served.headers.set("X-Cache-Status", "HIT");
return served;
}
} catch {
// Cache unavailable — degrade to a direct origin fetch below.
}
try {
const origin = await fetch(buildOriginUrl(url, env.ORIGIN_HOST), request);
const contentType = origin.headers.get("content-type") ?? "";
const vary = origin.headers.get("vary") ?? "";
// Only cache clean 200 HTML with no cookies, no revalidation markers,
// and no request-header Vary: the Workers Cache API key is URL-only and
// ignores request headers, so any such Vary would poison the entry
// (206 responses are rejected by cache.put anyway).
const cacheable =
origin.status === 200 &&
contentType.includes("text/html") &&
!origin.headers.has("set-cookie") &&
!/private|no-store|no-cache|must-revalidate|max-age=0|s-maxage=0/i.test(
origin.headers.get("cache-control") ?? ""
) &&
!vary.includes("*") &&
!/accept|origin|referer|sec-ch-ua|user-agent|cookie|authorization|range/i.test(vary);
if (cacheable) {
const copy = stripCookies(new Response(origin.body, origin));
copy.headers.set("Cache-Control", `public, max-age=${HTML_TTL_SECONDS}`);
ctx.waitUntil(cache.put(cacheKey, copy.clone()).catch(() => {}));
const served = withSecurityHeaders(copy);
// caches.default does not set cf-cache-status — this Worker sets its own
// header so cache behaviour is observable from the outside.
served.headers.set("X-Cache-Status", "MISS");
return served;
}
return origin;
} catch {
return new Response("Origin unavailable", { status: 502 });
}
},
};
async function passthrough(request: Request, env: Env): Promise<Response> {
try {
return await fetch(buildOriginUrl(new URL(request.url), env.ORIGIN_HOST), request);
} catch {
return new Response("Origin unavailable", { status: 502 });
}
}
function buildOriginUrl(url: URL, originHost: string): URL {
const origin = new URL(url.href);
origin.protocol = "https:";
// Set host + port in one step: origin.hostname rejects values containing ":".
origin.host = originHost;
return origin;
}
function stripCookies(response: Response): Response {
const headers = new Headers(response.headers);
headers.delete("set-cookie");
headers.delete("set-cookie2");
return new Response(response.body, {
status: response.status,
statusText: response.statusText,
headers,
});
}
function withSecurityHeaders(response: Response): Response {
const headers = new Headers(response.headers);
headers.set("X-Content-Type-Options", "nosniff");
headers.set("X-Frame-Options", "DENY");
headers.set("Referrer-Policy", "strict-origin-when-cross-origin");
return new Response(response.body, {
status: response.status,
statusText: response.statusText,
headers,
});
}
Two production notes. caches.default is Cloudflare's edge cache, so the copy the Worker stores serves the next crawler with no origin fetch. And the ORIGIN_HOST binding keeps your origin hostname out of the code, so one Worker runs against staging and production by swapping a variable.
Edge caching buys you time; it does not fix a slow origin. Bring TTFB down in parallel:
- Find the slow query or endpoint and fix it — add an index, cache the result, or move synchronous work to a background job.
- Warm the cache. A cron that pings your key URLs every few minutes keeps the edge warm, so the next crawler gets a HIT.
- Keep the origin warm. A keep-alive or a minimum-instance setting removes the cold-start tax crawlers keep paying.
How to verify
Prove the contradiction with three curl commands. The first times the crawler path through the Worker; the second times the origin directly by pinning its IP with --resolve; the third watches the cache warm up.
What to look for: once the edge cache is warm, bot-2's TTFB should be well below bot-1's. If the origin-direct TTFB is slow while the cached bot TTFB is fast, the edge cache is masking a slow origin — the contradiction in a single number. And the retry loop in step 3 should end on x-cache-status: hit — proof the Worker is shielding crawlers from the 524.
# 1. Time the CACHED crawler path (bot UA through the Worker). Run twice:
# once the edge cache is warm the second request reports X-Cache-Status: HIT
# (step 3 verifies the warm-up reliably).
curl -s -o /dev/null -A "googlebot" \
-w 'bot-1: http=%{http_code} ttfb=%{time_starttransfer}s total=%{time_total}s cache=%{header:x-cache-status}\n' \
https://www.example.com/slow-page
curl -s -o /dev/null -A "googlebot" \
-w 'bot-2: http=%{http_code} ttfb=%{time_starttransfer}s total=%{time_total}s cache=%{header:x-cache-status}\n' \
https://www.example.com/slow-page
# 2. Hit the origin directly. --resolve pins the origin IP, bypassing Cloudflare.
# A slow ttfb here plus a fast bot-2 above = the edge cache masks a slow origin.
curl -s -o /dev/null \
--resolve www.example.com:443:203.0.113.10 \
-w 'origin: http=%{http_code} ttfb=%{time_starttransfer}s total=%{time_total}s\n' \
https://www.example.com/slow-page
# 3. Watch the crawler cache warm up. cache.put runs asynchronously and the
# cache is per-datacenter, so retry until the Worker reports HIT.
for i in 1 2 3 4 5; do
s=$(curl -s -D - -o /dev/null -A "googlebot" https://www.example.com/slow-page \
| grep -i '^x-cache-status:' | tr -d '\r' | tr '[:upper:]' '[:lower:]')
echo "attempt $i: $s"
[ "$s" = "x-cache-status: hit" ] && break
sleep 2
done
On a live 524 page, the response also carries a cf-ray header with a Ray ID — hand that ID to Cloudflare support or your host, and it pinpoints the exact request in the logs.
If you're chasing an adjacent serverless failure — a Turso database that returns SERVER_ERROR: 404 right after creation — the create-to-first-query gap bites in the same cold-path spots, covered here: Turso SERVER_ERROR 404 right after creating a database. More edge-and-serverless troubleshooting on the C2CZ home page.
Sources
- Cloudflare Support docs — "Cloudflare 5xx errors", Error 524 "A timeout occurred": developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/
- Stack Overflow question 28316121, "CloudFlare 524 Error: But the page is working" (9,927 views, score 7, no accepted answer): stackoverflow.com/questions/28316121
- Cloudflare Workers Cache API: developers.cloudflare.com/workers/runtime-apis/cache/