Hi Google, please stop pooping the bed: a desperate plea from the indie web

I have no specific information here, never seen/used Shepherd until I saw a list on there a few days ago. Disclaimer, I work for Google, but not on anything related to this space, and this is based on my previous job where we did some SEO for an ecommerce site.

The example list given just looks a lot like spam when you squint. It's a list of affiliate links to buy products, and there are many HN threads talking about the abundance of affiliate link aggregators being a blight on the web. The commentary does look useful, but distinguishing between good commentary and bad commentary is hard, whereas distinguishing between a site designed to extract affiliate commission vs one more about the content is easier.

The comparison given to the other results here is frustrating, I know, but probably not a valid experiment. All the major search engines change results based on the user using them, or the IP address, or the region, or whatever, so it's impossible to know what others see. The developer of a book-focused shopping site is likely to get very skewed results for a book related query. My results were noticeably better.

The author says that a Bookshop.org list they created that links back to Shepherd is ranking #2, and this kinda makes sense to me. Bookshop.org sells the books, it makes sense that would rank above a site that only links to (and makes money from) sites that sell books.

SEO, and people getting annoyed at not ranking, has been a thing for 25+ years, I don't think this instance is any different.

It looks like shepherd.com is a book review site that doesn't have any reviews, just ratings. It links to Amazon.

Is this really the sort of content Google should be returning?

Edit: it seems I missed the link to the actual book reviews because the link text is uninformative: "Chosen by 1 person - see why." (Sometimes it goes to reviews, sometimes not.)

And the word "review" never appears on the pages that have reviews. Seems like bad SEO?

If you're looking for book reviews, here's a website with some pretty great reviews: https://www.thepsmiths.com/ (Content warning: the authors are conservative.)

One thing I find especially bizarre about the current situation: the pages that are making it to the top aren’t even pretending to be real content. If they were difficult-to-detect LLM-generated pages, that would be one thing. But they’re generally extremely low-effort affiliate spam, mostly claiming that they researched something “so I don’t have to”, followed by a bunch of Amazon links and explicitly acknowledged scraped reviews, and finally an obviously uninformative summary. They don’t even pretend to have real content!

What is Google doing?

8-10 years google search was amazing. A well crafted query would hit informational gold most of the time. I've been noticing and commenting on the decline privately for most of the period, but it's only in the oast year this seems to have come to broader awareness. There's an argument that in an information economy, searching for information should be treated like a public utility necessary for the functioning of society. I'm not making that argument but when you experience the long slippery slope of degradation of a service that was near ideal for the technology of the time, it does xome to mind.

That's the thing. It DID work. Really well for a while. But it was always atomic and context-less. We now have the opportunity to make it even better by refining results through dialogue. I hope someone does.. soon.

I find it ironic that there’s a desperate plea for the indie web from a Substack blog.

The shepherd.com link is the second one that shows up on Kagi, after a Quora link: https://kagi.com/search?q=best+books+on+Battle+of+Midway

Long ago I wrote a blog post[1] about intermittent hardware, and not letting it suck you in. This was posted in a Blogger/blogspot blog, which is owned by Google.

This morning Google couldn't find it, neither could Bing.

[1] https://mikewarot.blogspot.com/2007/10/mikes-law-of-intermit...

Simple answer --- stop using Google.

Get your friends and family and other internet users to do the same.

The only thing likely to get their attention is if enough people follow suit.

I woke up this morning.

Then checked my mailchimp subscriptions

and my grow.me subscriptions

And my substack subscriptions

Made sure my Cloudinary was properly configured

And my newrelic analytics

And my sentry analytics

And my rlcdn analytics

And my growplow analytics

And my 33across analytics

And my Scorecard research analytics

And my openxcdn analytics

And my trustarc analytics

And my creativecdn analytics

And my Google Tag Manager

And my Google Analytics

Then I checked my Mediavine ads

And my adsrvr ads

And my adform ads

And my adnxs ads

And my yieldmo ads

And my criteo ads

And my mediavine ads

And my pubmatic ads

And my id5-sync ads

And my rubicon-project ads

And my triplelift ads

And my pghub ads

And my zemanta ads

And my cognitivlabs ads

And my doubleverify ads

And my media.net ads

And my kargo ads

And my Amazon ads

And my Google Adsense

When I finally got to WeWork

I booted up my Macbook

And checked my Google Workspace email

Our Amazon Affiliate account was approved

And I checked our private Github issues for tasks

Then I let Microsoft CoPilot write Stripe integration

I write a new post for my Substack

Finally using my Chrome browser and ycombinator's platform I posted:

'Google is killing the independent web'

1) It has already been proven by research that Google quality of results slips from year to year [1]

2) Google has many incentives to make the search more difficult for you

3) Google has proven that it prefers money over quality of results with allowing "malvertising"

4) It is true that the landscape is more difficult. There are more walled gardens, to which even Google might not have access. There are more scams, casinos. More AI slop. The game was always hard on the other hand, so these are just 'excuses'

5) Why so often I see in Google results leading to major news sites instead of normal links?

6) If I write "Warhammer" I would expect thousands and thousands of pages in results. I think that Google prefers "content" over "quality". "newer" is "better". I would expect thousands of fan pages, which do exists, but are not crawled, or forgotten. Why can't I browse older pages? Why is there a limit to 10 pages?

7) For "Emulation" first page leads to "wikipedia", "cambridge dictionary", "vocabulary", it is so f boring

[1] https://www.404media.co/google-search-really-has-gotten-wors...

Visits from Google to one of my independent sites have been about the same in 2022, 2023, and 2024. Visits to another have roughly doubled from year to year. I don't use Google Search or surveillance ads but I don't have the issue that they report.

AyyEye's observation that their site is loaded with trackers and what look a lot like affiliate marketing links is one reason why Google Search might not like it any more.

Any idea how many Googlers still have the pre-dotcom-boom Internet technologist-citizen mindset?

I know a bunch flocked to Google early on. But the landscape has changed a lot since then, and many people weren't even born until after the Internet and society were very different.

Google recently changed who is running search. Basically, Prabhakar Raghavan was moved from running Search and ads to being Chief Technologist. Nick Fox is the new person running Google Search. This happened on 10/17/2024. For more information, please see https://searchengineland.com/google-shakes-up-leadership-rag... and https://www.theverge.com/2024/10/17/24272786/google-search-p... .

I wonder if Prabhakar Raghavan was moved out of Google Search because of Search's decline in quality?

Only somewhat related, but I was just complaining about search in Gmail. I normally use Thunderbird and didn't have it, so I used the web. Basic searches on the subject were so bad! I even put quotes around it and tried all of my Google-foo.

It felt like I was fighting some AI that was sure it knew what I wanted despite my "exact phrase" search.

This problem is age old and always the same. Optimizing for Google is a fools errand. You can win in the short term, but when their priorities change you will find your website excluded for the very things that boosted it previously.

Google are unreliable and untrustworthy. Their focus is ad revenue for themselves and nothing else. Build sites for humans and let Google do what it wants to.

I've had my lifetime quota of hearing the rants of guys who think their search quality metric is objectively superior, oh and by a total coincidence their preferred ranking gives them a direct financial benefit. Just a complete failure to imagine the larger issues.

These days I usually only search google for an exact phrase or title I got from chatgpt to check if it hallucinated.

Take the same time and go back in and see what the other people have said about how they get a job to make a good night and see how you want it out land on a good work! I'm sure if poor is going on that e and the next year Prime Minister of course he would. And the other one of my own parents were that he had been working with him and he didn't want the money he was. But the next thing was the other people that had to make the other people that had no idea how to get out. But the only thing I know about this would have Saints of the game

Google got so big it swallowed the internet. It now has digested it and what is left is... this.

I used to love crafting websites and cared about SEO. What's the point now, no one is going to find your content. It won't even be on the third page. Google will answer questions by regurgitating whatever it swallowed on your websites and presume no one will click through, it won't even bother marking the authors.

Instead it appears to be prioritising whichever website is going to give it revenue first, e.g. the click farms.

The regular folks don't care, they google for stuff like "am I dying if I have a pimple?" (to which the answer is always yes, apparently). No one does actual meaningful research using Google anymore, if you do, good luck, get your gloves out <picture of dinosaur poop in Jurassic Park>.

The global internet as it stands is close to dead. Discoverability of "cool" things is down to social media, tricked by "influencers", who are tricked by marketing themselves.

We need a hard reset button, it needs to start from the ground up with site rings, and good content. Ah... that last part, "good content", is now stuffed with AI Samey McSamey sounding text. I really don't see a way out of here.

The funny part is we used to think that the internet was going to change the world. We thought all idiots needed was information. Access to information would fix the world! Instead, it only has given the village idiots a global voice: if you can think of some dumb crazy thing, you'll find dumb crazy people agreeing with you, so you must be right!

I've been on the internet since 1997 and I think it's the worst it's ever been.

Switched to Perplexity Pro and haven't looked back

Always use google in incogniti so they dont give personalised results, thats the least we can do before google search completely dies, Or just prevent google any cookie permission, keep it session only

As far as query trackers on google urls, they are necessary

One good case of necessity of google tracking parameters is --> you search for movie or related, but muktiple movies by same name have been made , you dont remember year, you do remember some actor or story description Searxg and google will in show with media carda foe movies you ckick iy and yoi get exact same film with year yoi wantes

Wait ->I am telling above because when that mefia image of film i inspecred itsurl it was just Googke ?search=filmname&teackers The search ket wasnt modified with year same as i searched first yet it gave me the desires year film

may be because of trackers &ved amd all paramers link tags to search teaukts its loke ctoss maching in databases

Google can't please all of the sites all of the time, or all the visitors.

It is too big to evem worry about that 4k a day clicks for one site. It is like us optimizing the expense of 0.01c. It makes a difference when that 0.01c is an API call that you call a million times. But it only surfaces if you do aggregate it.

Therefore this problem can only even be seem by Google if it can be surfaced in aggregate overy say a billion queries.

I wonder how that can be done.

Probably only can be done using data. Which means spying on people in various ways. And making assumptions about length of time on site equals quality.

They probably use machine learning too. There may be no reason for the lost rankings other than a wind change caused by some updated parameters in an OKR chasing model.

Why don't people just stop using Google? And by people, I mean everyone here.

Whenever this point comes up, I see people claim they ONLY see the results they want in Google. How would you know if you don't actually use anything else? Kagi is excellent search. Neeva was pretty great when it was active. DuckDuckGo is passable. Idk how Qwant gets money but it's been around a bit.

Complaining about the same thing forever and expecting a change doesn't make any sense. Y'all are in abusive relationships with Google and refuse to leave. Sure, your job may use Google Suite and you need to make money. What about the rest of your life? Stop hitting yourself.

The author talks about how long users spend on his page, but can Google track that? Has ubiquitous tracking blocking, and Google's failure to adapt, eroded the quality of search?

Checking my Search Console Tools recently, the number of "Crawled - currently not indexed" pages across all my sites has risen sharply recently, especially on sites where the age of the content is more than a few years.

I put a fair bit of time into trying to improve the sites with JSON-LD and breadcrumbs and what not. It seems to have helped just a little bit.

I don't make any money off any of it, but it's still kind of irritating that no one can find it.

The indie web was around before google and it will be around long after google is gone. I would argue that the indie web has incurred a much larger loss from people thinking seo/engagement metrics are something worth optimizing. Many of the best examples of the indie/small web don’t have js tracking and little to no css.

Google didn’t shit the bed, they just did whatever was profitable to them and they’ll continue doing so until they either fail or there’s no more profit to be squeezed. They dropped the “don’t be evil” motto a long tome ago.

As much as I share the frustration, what does Google gain from providing a better service?

Its only serious competitor is Bing, which isn't even a search engine anymore but a billboard for Microsoft to advertise ChatGPT.

>Explore only the books that readers love.

that doesn't sound like "the independent web", that sounds like an airport bookseller.

Maybe we should just start looking at other search engines seriously and without irony.

How’s shepherd.com (for example) doing on bing or duckduckgo?

Maybe it’s time for google to die.

"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclair

Great work as always Danny Sullivan!

There are people inside Google who can put a stop to this, if they merely stand up and say "this is wrong" including their CEO. It simply requires bravery and faith in Google's institutional ability to continue to profit doing different things.

Big companies have pivoted before on the heels of brave executives. This applies as much to search as it does to privacy, AI, their transparency, their support of open source, and their weaponization of their browser.

It's a crying shame to see how enshittified a company that could be changing the world has become.

exa.ai good for finding these sites? but yes, many people will still be googling

thought this was going to be about the 'log in with google' popup

but to google you're just a desperate flea from the indie web.

If you're still barking in the general direction of google and similarly sized search engines, good luck. I'd rather look towards kagi and brave.

you should check out exa.ai/search if you miss indie content!

Hi (company that benefits from hurting me), please stop hurting me.

Company: No.

is there a way to detect and block substack?

Hi stranger, please stop messing with the scrollbar

Big adtech cares not as long as the ads keep flowing

The real problem is that Google solves for the needs of the lowest common denominator of content consumers, which leads inevitably to enshittification

Well they replaced the Yahoo alum accused of tanking search quality with a McKinsey guy....I'm sure things will get better now /s.

https://www.wheresyoured.at/requiem-for-raghavan/