TTFB Optimization: Under 600ms for AI Crawlers

What Is TTFB and Why AI Crawlers Care

Time to First Byte (TTFB) measures the duration from when a client sends an HTTP request to when it receives the first byte of the response from the server. It encompasses DNS resolution, TCP connection, TLS handshake, and server processing time.

For human users, TTFB is one component of the overall page load experience. For AI crawlers, TTFB is far more critical — it is the gatekeeper that determines whether the crawler can even begin to receive your content within its time budget.

AI crawlers like OAI-SearchBot (ChatGPT), PerplexityBot, and ClaudeBot operate under strict constraints. They crawl thousands of pages per session, allocating limited time to each page. When your server takes 1,200ms to respond, that is 1,200ms the crawler cannot spend receiving content, parsing HTML, or visiting other pages on your site. A slow TTFB effectively reduces your crawl budget — the number of pages the AI bot visits and indexes during a session.

For detailed coverage of server-side performance factors, see our server response optimization guide. For the broader performance picture, read Core Web Vitals and AI SEO.

TTFB Components: Where Time Is Spent

TTFB is not a single measurement — it is the sum of several sequential steps:

1. DNS resolution (10-100ms)

Converting your domain name to an IP address. This can be slow if your DNS provider has limited global presence or if the record has a short TTL requiring frequent lookups.

Optimization: Use a fast DNS provider (Cloudflare DNS, Route 53, Google Cloud DNS). Set reasonable TTL values (300-3600 seconds). Enable DNS prefetching for your domain.

2. TCP connection (10-100ms)

Establishing a TCP connection between the client and server. Distance between them directly affects this — each additional 1,000km adds roughly 6-10ms of latency.

Optimization: A CDN places edge servers closer to the requester. Keep-alive connections reduce reconnection overhead for subsequent requests.

3. TLS handshake (20-100ms)

If using HTTPS (which you should), the TLS handshake adds another round trip. TLS 1.3 requires one fewer round trip than TLS 1.2.

Optimization: Ensure your server supports TLS 1.3. Use OCSP stapling to avoid certificate status lookups. Enable 0-RTT resumption for repeat connections.

4. Server processing (50-2000ms+)

This is where the largest gains are available. Server processing includes: executing application code, querying databases, rendering templates, and building the HTML response. On a WordPress site with no caching, this can easily take 500-2000ms.

Optimization: Full-page caching, object caching, database optimization, and application-level performance tuning.

Server-Side Caching Strategies

Caching is the most impactful TTFB optimization. Instead of processing every request from scratch, caching serves pre-built responses.

Full-page caching

The most aggressive and effective approach. The server generates the HTML response once and caches the complete output. Subsequent requests receive the cached HTML without executing any application code.

Implementation options:

• Reverse proxy cache (Varnish, Nginx FastCGI cache): Sits in front of your application server and intercepts requests before they reach application code. Can reduce TTFB from 800ms to 5-15ms.
• Application-level cache (Redis page cache, WP Super Cache): Stores rendered pages in memory or on disk. Faster than re-rendering but slightly slower than a reverse proxy.
• CDN edge cache: Stores full pages at edge locations worldwide. See the CDN section below.

Cache invalidation: The main challenge with full-page caching. Set up automatic purging when content is updated. Most modern caching systems support tag-based invalidation — update a blog post and only that post's cache is purged.

Object caching with Redis or Memcached

For dynamic pages that cannot be fully cached (logged-in user pages, personalized content), object caching speeds up the processing stage:

• Database query results — Cache frequently executed queries. A product listing page that queries the same 50 products for every visitor benefits enormously.
• API responses — Cache external API calls that do not change frequently.
• Computed values — Navigation menus, sidebar content, and footer data that are the same across requests.

Redis is generally preferred over Memcached for its data structure support, persistence options, and pub/sub capabilities for cache invalidation.

Stale-while-revalidate

This caching pattern serves stale (expired) content immediately while asynchronously refreshing the cache in the background:

Cache-Control: max-age=60, stale-while-revalidate=3600

This means: serve the cached version for 60 seconds. After 60 seconds, serve the stale version while fetching a fresh copy. This eliminates TTFB spikes that occur when cache expires and the first request must wait for a fresh response.

CDN Configuration for AI Bots

A Content Delivery Network places cached copies of your pages at edge servers around the world. For AI crawlers, this is transformative.

Why CDNs matter for AI crawlers

Most AI crawlers operate from data centers in the United States — primarily in Virginia (us-east-1) and California (us-west-2). If your origin server is in Europe, every AI crawler request crosses the Atlantic Ocean, adding 80-150ms of network latency each way. A CDN with US edge nodes eliminates this latency for cached content.

CDN options for AI SEO

| CDN | Best For | Edge Cache TTL | AI Bot Handling | |---|---|---|---| | Cloudflare | Most websites, easy setup | Configurable per URL | Bot management with AI-specific rules | | Vercel Edge Network | Next.js / Jamstack sites | Automatic with ISR | Built-in edge caching | | AWS CloudFront | Custom infrastructure | Configurable per behavior | Lambda@Edge for bot-specific logic | | Fastly | High-traffic, real-time purge | Instant purge support | VCL for advanced bot routing |

CDN caching rules for AI bots

Configure your CDN to cache HTML pages (not just static assets) for AI crawlers. Most CDNs allow you to set different caching rules based on the User-Agent header:

# Cloudflare Page Rule example
URL: example.com/*
Cache Level: Cache Everything
Edge Cache TTL: 1 hour

For AI-specific optimization, consider caching pages longer for bot requests than for human visitors. AI crawlers do not need real-time content freshness — a 1-hour or even 24-hour cache is perfectly acceptable.

Early hints (103 status)

Some CDNs support HTTP 103 Early Hints, which sends preload instructions to the client before the full response is ready. While AI crawlers may not process early hints, this feature benefits human users and contributes to better overall performance metrics.

Database and Application Optimization

When caching cannot fully cover your pages (dynamic content, authenticated pages), optimizing the application itself reduces TTFB:

Database query optimization

• Add indexes for columns used in WHERE clauses and JOIN conditions. A missing index can turn a 5ms query into a 500ms full table scan.
• Avoid N+1 queries — Use eager loading (JOIN or subquery) instead of executing separate queries for related data in a loop.
• Use EXPLAIN to analyze slow queries and identify performance bottlenecks.
• Connection pooling — Reuse database connections instead of opening a new connection for each request. PgBouncer (PostgreSQL) and ProxySQL (MySQL) handle this.

Application-level optimization

• Reduce middleware — Each middleware layer adds processing time. Remove unused middleware from request pipelines.
• Optimize template rendering — Compiled templates (Twig, Blade) are faster than interpreted ones. Pre-compile templates during deployment.
• Use HTTP/2 or HTTP/3 — These protocols reduce connection overhead through multiplexing and header compression.

PHP-specific optimizations (WordPress, Laravel)

• OPcache — Enable and configure PHP OPcache to store compiled bytecode in memory. This eliminates the compilation step on every request.
• PHP 8.x — Upgrade to PHP 8.x for JIT compilation and significant performance improvements over PHP 7.x.
• Disable unused plugins — Each active WordPress plugin adds to the processing time. Audit and remove plugins that are not essential.

Hosting Infrastructure Choices

Your hosting infrastructure sets the floor for TTFB — no amount of application optimization can overcome a slow server.

Hosting tiers and typical TTFB

| Hosting Type | Typical TTFB | AI Crawler Suitability | |---|---|---| | Shared hosting | 500-2000ms | Poor — inconsistent performance | | VPS (basic) | 200-800ms | Acceptable with caching | | Managed cloud (Vercel, Netlify) | 50-200ms | Excellent for static/SSG sites | | Dedicated/cloud server | 100-400ms | Good — full control over optimization | | Edge-rendered (Cloudflare Workers) | 10-50ms | Optimal — lowest possible TTFB |

Key infrastructure decisions

Server location: Choose a server location close to your primary audience AND close to AI crawler data centers (US East is optimal). If your audience is in Europe, use a European origin with a CDN that has strong US edge presence.

Server resources: Ensure adequate CPU and RAM. During AI crawler spikes (when multiple bots visit simultaneously), insufficient resources cause TTFB spikes. Monitor server resource utilization during known crawler activity periods.

HTTP protocol: Enable HTTP/2 at minimum, HTTP/3 (QUIC) if your infrastructure supports it. HTTP/3 eliminates TCP handshake latency, reducing TTFB for initial connections.

Measuring and Monitoring TTFB

Track TTFB from multiple perspectives:

Lab testing:

• WebPageTest — Test from multiple locations worldwide. Use the "First Byte" metric in the waterfall chart. Test from Virginia (us-east-1) to simulate AI crawler location.
• Chrome DevTools Network panel — The "Waiting (TTFB)" column in the Network tab shows per-request TTFB.
• curl timing — curl -o /dev/null -w "%{time_starttransfer}" https://example.com gives you TTFB from the command line.

Field data:

• Navigation Timing API — performance.timing.responseStart - performance.timing.requestStart gives real-user TTFB.
• CrUX (Chrome User Experience Report) — Provides TTFB data from real Chrome users.
• Server access logs — Analyze request duration from your web server logs to identify slow endpoints.

AI-specific monitoring: Check your server logs for AI crawler User-Agents (OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot) and compare their TTFB against human visitor TTFB. If AI bots consistently get slower responses, investigate whether bot detection or WAF rules are adding latency.