Crawler traffic was eating the rendering budget
On a multi-tenant e-commerce platform with dozens of stores and around a million products, I measured where the rendering work went: about three out of four rendered pages were served to bots, not shoppers, and five SEO crawlers alone accounted for half of it. I blocked those by user agent in nginx across every store, blocked AI and social-media crawlers on the image endpoint (plus an old scraper pulling tens of thousands of images a day), put smaller bots on a crawl delay and took internal search out of the index. I evaluated rate limiting and dropped it: the remaining volume didn’t justify it.
Result−49.5% page-rendering load, with no new infrastructure.