There is so much spam, abuse, and aggressive scraping that I've had to block:
- AWS
- Google Cloud
- Azure
- Alibaba Cloud
- Oracle
I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time.
Even Apple has aggressively ramped up its scraping in the past few months, but thankfully they do respect robots.txt.
It's frustrating because some of my real users route through these clouds via their work VPNs. Also, there are a lot of less-than-reputable AI scrapers using global bot-nets via residential IPs to scrape, making it even harder to detect and ban them without affecting real people.
* like leaked .env routes, brute forcing admin panels, and so on
It's too late to comment on your older posts, but I'd just like to say: it's nice to see someone making oldschool web browsers work properly on the modern web.
https://www.sammystraus.com/#using-llms-to-make-the-emacs-we...
:) ive been dreaming of a usable emacs browser for so long
Have you tried dealing with bots using CF Turnstile? Ref: https://www.cloudflare.com/products/turnstile/
I did this for alyph.ai, but was recently flooded with so many browser-using, Turnstile-defeating bots from Taiwan that I had to completely turn off sign ups. It's not fun anymore.
Not even third party login worked (Google, Linkedin, ...) to curb this.
They were logging in, scraping the client API for vulnerabilities/possibilities and then spammed erotic dog poems to Claude Fable until the $5 welcome credit got drained.
Fun New World.
not yet, but I think something like this would be a next step
[flagged]
What confused me most about this is that the entire site is, apparently, a single page (there's no sitemap that I can use to verify this; there IS an RSS file which appears to confirm it) containing all posts. Links at the top (headed "Table of Contents") are all in-page links.
At first, I assumed this was a single blog post, especially with a "Table of Contents" at the top. I guess it's following some old-school blogging conventions, but AFAICR, even those sites retained individual pages for separate posts.
it's org-html-export-to-html babyyy lmao
I think I could improve the format a bit with something like Jekyll, but I haven't had a good chunk of time to sit down and work on it
> one specific user. I won't name names
Who is?!
haha im not naming names
You shouldnt defend bad actors and abusers. Hiding them is tantamount to them attacking others indiscriminately. Cause youre not the first victim, and definitely not the last.
You see something, you say something, or you do something.
> I won't name names, but I looked them up, and they describe themselves as a service that listens for changes to a website, and then summarizes them into an AI slop news letter. And by 'listens to changes' I mean query our website many many times per-minute. Blocked.
That sort of behavior would justify a name and shame imo
Inconsiderate shitty services don’t deserve the politeness of protecting their name
Same with me, every-time I got 1st page I got a lot of robots
Ok, on the one hand to have one bot hitting your website to populate some kind of slop newsletter is pretty annoying.
On the other hand, “multiple times per minute” doesn’t seem that high? Even if it was 60x per min that’s fully within the limits of what a 2015 laptop in the closet can handle.
yeah, it was a wake-up call that I needed to make certain parts of my application faster, and I've been working on it for the past few weeks
Try posting something viral on Mastodon. The logs clearly show that it isn't the most efficient social network.
Heh. I posted a GitHub project Show HN, a couple of days ago, and it never even made it out of new, but it got me some bots.
That's a big reason that I never post any of my apps here.
Welcome to hell, kid...
[flagged]
[flagged]
I don't know HN's rationale, but: if you try to ban self-promotion, then in practice you're only banning honest self-promotion. And if you allow it, but try to ban accounts that only exist for that purpose, then in practice you're encouraging them to spam mediocre comments in other threads to dilute the signal. [edit: I see that the guidelines do say not to use HN primarily for promotion. I'm willing to bet this significantly increases the amount of insincere engagement, but obviously the admins/moderators here are more aware of the tradeoffs than I am, and they have decided they are worth accepting.]
IMO the important thing to try to crack down on is vote manipulation; if you have that under control, then you can probably happily allow self-promotion and trust that only the interesting stuff will rise to the top.
Any thoughts on what the alternative would be?
Perhaps a contribution requirement per post? Though this proposal just invites bots to actually post comments because that becomes a requirement for posting links.
The guidelines already cover this:
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
https://news.ycombinator.com/newsguidelines.html
I assume that users who primarily/only post their own stuff and get flagged a lot reap the consequences, there's plenty of spam filtering on here thank goodness.
[delayed]
make interactions with others' posts/post others' stuff a soft req for posting your own stuff? So not following it could be deemed actionable by the hn mod team?
Edit: nvm thats already in the guidelines
Ostensibly, they aren’t. From the Guidelines (https://news.ycombinator.com/newsguidelines.html):
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
I _also_ posted to HN and made it to #1[0]; the effect this had on my CPU/network traffic on a 5/EUR month VPS over at Hetzner was in the single digit percentages.
As per the wiki page for the C10k problem[1] where it says "_was_ [emphasis mine] the problem of optimizing computer networking stacks to handle a large number of clients at the same time". This was solved over a decade ago...
Now get off my lawn!
(TBF I think the author acknowledges this)
The c10k problem is a different problem. It relates to simultaneous open TCP connections at a server. It would take a very very large number of crawler bots to force open 10k connections, I suspect.
Isn’t scanning for vulnerabilities illegal? Traditionally security researchers had to be very careful about such things
Application-level banning seems very inefficient, these domains should be banned at the network (IP) level. Just drop any incoming packets from these domains/IPs, send absolutely nothing back that even confirms that you exist.
They are banned at the edge in Cloudflare, they never hit my application.