Show HN: I'm making an AI scraper called FetchFox

You have LinkedIn and Twitter examples, where you're very likely violating their TOS as they prohibit any scraping.

I also assume you don't check the robots.txt of websites?

I'm all for automating tedious work, but with all this (mostly AI-related) scraping, things are getting out of hand and creating a lot of headaches for developers maintaining heavily scraped sites.

related:

- "Dear AI Companies, instead of scraping OpenStreetMap, how about a $10k donation?" - https://news.ycombinator.com/item?id=41109926

- "Multiple AI companies bypassing web standard to scrape publisher sites" https://news.ycombinator.com/item?id=40750182

I am really curious how do people actually evaluate scrapers? There are so many options and I am dizzy just trying to read them...

Also wondering how does the OP think about comparing themselves and standing out in the marketplace of seemingly bazillion options

Out of interest - why is it called FetchFox - but it doesn't work on Firefox?

A bit off-topic, but why do people still use the GIF format? The "example-hn.gif" is 8.5MB, for 45 seconds of pretty stuttery video. I converted it to a similar looking VP9 video, and it was only 1.5MB, and with AV1 I got it down to 550KB with basically lossless quality.

Of all the names to give something you built for chrome instead of Firefox…

This is really cool, I'd just go ahead and double check that max spend limit on your OAI key before going to bed :)

This kind of stuff gets expensive fast.

This is a really cool tool. Have been playing with similar scraping capabilities, so appreciate you sharing the source code as well. People who are saying "loads of scraping tools already exist" have likely not suffered through the current state of the art too, as heuristic based approaches absolutely pale in comparison to what an LLM can extract.

Would love something like this that allows users to trivially turn sites like Facebook/Twitter into RSS feeds. I'm sure this kinda thing is a useful stepping stone to doing that.

Maybe instead of selling scraping to end users, invert the problem and sell data to AI companies:

1. Pay users to install a browser extension that scrapes social media content they browse. Or ask them in exchange for a service, e.g. "remember everything I browse and make it searchable", etc.

2. Ship the data you scrape to your servers.

3. Sell training data to companies at a discount.

This gets past the new rate limiters and blocks that Reddit and others have installed.

I've wanted to make something like this myself, so thanks and good job!

How does this work? Does it rely on GPT to extract the data or does it actually generate a bunch of selectors? If it's the former, then the results aren't reliable since it can just hallucinate whole results or even just parts.

Hmmm... assume one can harvest the financial news from some big website, then correlate it with market (historic) movements (when articles stating X Y and Z are posted, then after 24h Gold price dropped)(with a 80%-90% rate)(that could be used to predict and trade 'regularly')

Interesting. How long did it take to figure out how to do this with ChatGPT?

> "By scraping raw text with AI, FetchFox lets you circumvent anti-scraping measures on sites like LinkedIn and Facebook. Even the the complicated HTML structures are possible to parse with FetchFox."

Nice! Any plans for Firefox support?

This is really cool, I don't know when I will need something like this, but I am sure the day will come! I hope the tech and the policies that govern scrapers are in place at that time! Best of luck

Can I recommend you provide some cost estimates next to the examples for using your own key? I tried a few custom extractions and then checked my usage dashboard and it was already over $2.

This is interesting. How much difference is it (in cost, quality) by using this approach compared to taking a image capture of the page and then sending it off to a multi modal LLM?

Useful! Keep up the good work!

Sweet ill take a look!

Gm gm! This is a good tool. Tried it out to scrape for email from various professional sites. Thumbs up

Forgetting about law and copyright and robots.txt because hey most scrapers have to rely on fair use anyway, you even forget about basic consideration for the sites you hammer.

That sounds like a scold, but it's meant as an observation.

Now I will embed some implied scolding in what's to follow, but feel free to ignore that part; I wouldn't expect you to care.

But if you lack even a shred of human decency or morals, perhaps there's one more reason you might consider for spreading out your requests across sites and time, instead of absolutely pushing the abuse to the hilt, and that is that if you change what you are doing, and take a slower, more gentle approach, and I'm appealing to your selfishness here because clearly that is the only viable way into your head, then you will be less likely to cause countermeasures, and more likely to succeed.