Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all
The website next-episode.net faced indexing issues as Google misreported "429 Too Many Requests" as "5xx" errors. Whitelisting Google Crawlers' IPs resolved the issue, with no new errors reported since.
The website next-episode.net experienced issues with Google Search Console reporting "Server error (5xx)" for several pages, despite the site functioning normally. The owner investigated the server, Cloudflare, and analytics logs but found no evidence of 5xx errors. Over time, the number of reported 5xx URLs increased, prompting further investigation. The owner discovered that the requests were returning a "429 Too Many Requests" status due to a custom rate limiting feature that restricts excessive requests. This unexpected increase in Google's crawling activity led to the misreporting of the 429 status as a 5xx error in GSC. To address the issue, the owner whitelisted Google Crawlers' IPs and the entire Google ASN (15169) in the rate limiter. After implementing these changes, there have been no new 5xx errors reported in GSC for over two weeks.
- The website faced indexing issues due to Google misreporting "429 Too Many Requests" as "5xx" errors.
- The owner discovered the problem stemmed from a custom rate limiting feature.
- Whitelisting Google Crawlers' IPs and ASN helped resolve the issue.
- Monitoring showed no new 5xx errors for over two weeks after the fix.
- The incident highlights the importance of understanding server response codes and their implications for SEO.
Related
Cloudflare blocking my IP (2023)
The Cloudflare Community discusses a user facing "verify you are human" prompts on Cloudflare-protected sites. Cloudflare advises contacting site owners for resolution, clarifying they don't block IPs. User frustration ensues.
Google Search Ranks AI Spam Above Original Reporting in News Results
Google Search faces challenges as AI-generated spam surpasses original reporting in news results. Despite efforts to combat this issue, plagiarized articles with AI-generated illustrations dominate search rankings, raising concerns among SEO experts and original content creators.
Link-Busters Sent a Billion DMCA Takedown Requests to Google Search
Link-Busters sends over a billion DMCA takedown requests to Google Search to combat shadow libraries. Penguin Random House leads the effort, with 75% success rate in removing infringing URLs.
AI crawlers need to be more respectful
Read the Docs has reported increased abusive AI crawling, leading to high bandwidth costs. They are blocking offenders and urging AI companies to adopt respectful practices and improve crawler efficiency.
Google listed my restaurant's number as its British HQ
A London restaurant owner faced distress after Google mistakenly listed their contact details as the company's UK headquarters, resulting in 300 weekly calls. Google has now corrected the error.
4 commentsNot trying to criticize if this was already checked. Just something I’d try to double check out of being overly cautious.
It should return 429 so Google can reduce the requests.
I'm not even sure why you would rate limit in the first place? IPS are not unique. One company gateway or university for example has plenty of users.
Rate limit requests from users you know and make sure every public API is properly cached.
I'll take a look for 429s. Cheers.
Related
Cloudflare blocking my IP (2023)
The Cloudflare Community discusses a user facing "verify you are human" prompts on Cloudflare-protected sites. Cloudflare advises contacting site owners for resolution, clarifying they don't block IPs. User frustration ensues.
Google Search Ranks AI Spam Above Original Reporting in News Results
Google Search faces challenges as AI-generated spam surpasses original reporting in news results. Despite efforts to combat this issue, plagiarized articles with AI-generated illustrations dominate search rankings, raising concerns among SEO experts and original content creators.
Link-Busters Sent a Billion DMCA Takedown Requests to Google Search
Link-Busters sends over a billion DMCA takedown requests to Google Search to combat shadow libraries. Penguin Random House leads the effort, with 75% success rate in removing infringing URLs.
AI crawlers need to be more respectful
Read the Docs has reported increased abusive AI crawling, leading to high bandwidth costs. They are blocking offenders and urging AI companies to adopt respectful practices and improve crawler efficiency.
Google listed my restaurant's number as its British HQ
A London restaurant owner faced distress after Google mistakenly listed their contact details as the company's UK headquarters, resulting in 300 weekly calls. Google has now corrected the error.