I Do Believe We've Been Invaded

[Alassë Estel]

It seems there is an attack of bots.

[Galadriel55]

Ah, I remember these feoul creatures. They would come in my youth, and raid our forum where the defenses were weak...

ETA: holy cow, they made over a page of spam threads on N&N and Mirth? That's prolific. That's bot talent...

[Alassë Estel]

It seems they've returned in force, and with newfound vigour.

😂

[Galadriel55]

Thankfully it looks like the mighty Mithadan and fellow admins prevailed, and the forum has been scoured of this spam-filth. All hail the great defenders of the Downs!

[Alassë Estel]

Indeed. The devious foes have been vanquished by the valour of our guardians!

[Mithadan]

Much credit belongs to Estelyn and others, who did most of the pruning. Thanks to all.

[Bêthberry]

I would rep you for your speedy and heroic efforts, Mithadan, but I am told that I have to spread the rep around to others before I can do so.

I was able to rep Estelyn and mormegil.

[Snowdog]

It's nice to be a part of a legacy Middle Earth forum that hasn't been abandoned to the AI spambots Well done killing them off.

[ryanrobbie]

I value that you wrote this great post for us. I promise that this will be good for most people mapquest driving directions

[Galadriel55]

Originally Posted by ryanrobbie

I value that you wrote this great post for us. I promise that this will be good for most people mapquest driving directions

This has got to be the most ironic choice of thread for this post.

[Alassë Estel]

Quite so.

[Morthoron]

Originally Posted by Galadriel55

This has got to be the most ironic choice of thread for this post.

That lefthand turn direction into the Dead Marshes was regrettable.

[Huinesoron]

The parade of Socratic bots continues; WCH has attracted two in a row to say "Oh yes, [Insert User Here], you are so wise." My assumption is that they're powered by current-generation 'AI' chatbots.

I've mostly been ignoring the ones that don't drop links, but is it worth reporting them on sight? They seem to mostly make one or two posts, but it's occurred to me they might be waiting to reach a critical mass and then spam the forum massively. (The other option is that they try to spam right away, but the one-post-per-minute block catches them.)

hS

[Galadriel55]

Originally Posted by Huinesoron

I've mostly been ignoring the ones that don't drop links, but is it worth reporting them on sight? They seem to mostly make one or two posts, but it's occurred to me they might be waiting to reach a critical mass and then spam the forum massively. (The other option is that they try to spam right away, but the one-post-per-minute block catches them.)

I've been wondering the same thing. They're kinda harmless bots, they don't appear to advertise or openly subvert anything, but they are bots nonetheless. Should we be reporting these posts?

[Mithadan]

We are aware of them. Those posts that have links embedded get deleted. I find the mindless (no pun intended) posts wearisome but harmless. It is the sheer number of scrapers that bothers me. I used to be amused at the number of “guests” but now recognize that they affect the website’s resources (familiar word)?

[Galadriel55]

Originally Posted by Mithadan

It is the sheer number of scrapers that bothers me. I used to be amused at the number of “guests” but now recognize that they affect the website’s resources (familiar word)?

Ahhh, so that's the reason.

[Elvellon]

There are currently 3959 visitors on the forum: me and 3958 guests. Those must be bots. I believe there's a way to ban these IPs. It makes it incredibly difficult to use the forum because every other time I click a link I get "Resource Limit Reached".

Edit: In fact, I got a "Resource Limit Reached" message while I was trying to post this!

[Mithadan]

They are bots or AI scrapers. The culprit appears to be the Movies forum for some reason. 80% or more of the "visitors" are congregating there. As I understand it, the 508 error, also known as "Resource Limit Reached", results from excessive traffic. The obvious solution would be to take down the Movies forum, but this would be an undesirable option, and there is no guarantee that it would cure the problem.

[Elvellon]

Originally Posted by Mithadan

They are bots or AI scrapers. The culprit appears to be the Movies forum for some reason. 80% or more of the "visitors" are congregating there. As I understand it, the 508 error, also known as "Resource Limit Reached", results from excessive traffic. The obvious solution would be to take down the Movies forum, but this would be an undesirable option, and there is no guarantee that it would cure the problem.

Look at Full-length LotR Parody Movies. It has 17.9 million views for a thread with 650 replies. Other threads on the board with over a million views have replies in the thousands.

There's something very strange going on there that's attracting the bots like moths to a flame. The admins really need to do something about this, it's making the board harder and harder to use.

[Huinesoron]

Originally Posted by Elvellon

Look at Full-length LotR Parody Movies. It has 17.9 million views for a thread with 650 replies.

Yeah, we've viewed it a lot while working on the Zong, but not *that* much.

I'm worried now that I've gone and linked that thread specifically somewhere that's drawing the locusts... I'll try and take a look round, because that's annoying if so.

hS

[Mithadan]

I did not notice that. Huinesoron, check on your end, and I will make some inquiries as well.

[Galadriel55]

Originally Posted by Huinesoron

Yeah, we've viewed it a lot while working on the Zong, but not *that* much.

I'm worried now that I've gone and linked that thread specifically somewhere that's drawing the locusts... I'll try and take a look round, because that's annoying if so.

I did share a link to that thread specifically with a few people, but I haven't posted it anywhere. I'll ask if any of them put it up somewhere else. Indeed, if all those views were real traffic, that would be quite the audience. Is there anything else about the thread that might attract them? Would the sheer number of hyperlinks somehow serve as a magnet?

[Huinesoron]

Originally Posted by Mithadan

I did not notice that. Huinesoron, check on your end, and I will make some inquiries as well.

Okay, so I didn't link to the Downs on any of the videos (which have no views anyway due to being semi-private), or on the Libretto document. I posted one link on another forum back in 2020; I've gone back and removed that, but it doesn't seem a likely source. That was this link to page 9; I don't know if there's any way to check which posts are specifically being viewed?

Originally Posted by Galadriel55

I did share a link to that thread specifically with a few people, but I haven't posted it anywhere. I'll ask if any of them put it up somewhere else. Indeed, if all those views were real traffic, that would be quite the audience. Is there anything else about the thread that might attract them? Would the sheer number of hyperlinks somehow serve as a magnet?

Or the number of pictures (mostly hosted on Imgur), or the number of YouTube links specifically? But I don't know of any actual mechanism for a bot farm to get hold of that information. AI scrapers (like the Wayback machine) would be a possible answer, except I think they work the other way round - they'd cluster somewhere because lots of incoming links lead them there, they wouldn't lurk there because of how much is going out.

I also don't understand why they would. They've sent 4000 bots to our thread today because, say, it links to lots of YouTube videos, and then.. what? What's the profit? What's the point? They're not even posting!

If the Downs tracks visiting IP addresses it might be possible to check them against lists of known scrapers. And if there's a tool to look at what exactly is getting the views (eg, are they coming in through the front page or straight to a post?) we could try fixing that specific thing. All I know how to see is the viewers per forum, not per thread.

hS

[Galadriel55]

Originally Posted by Hui

I also don't understand*why*they would. They've sent 4000 bots to our thread today because, say, it links to lots of YouTube videos, and then.. what? What's the profit? What's the*point? They're not even posting!

I do wonder though if this is a credit to Mithadan and the new security systems. To post you have to make an account first, and it seems like that part of the security screening is really holding back the flow.

[Elvellon]

I can't help but wonder why it seems like nobody is doing anything about the bots? I was hoping that bringing it up and pointing out the target of the bot activity would help spur some kind of change. Are there any admins or mods who can be contacted or who can comment on the situation? I don't know about anyone else, but the bot activity is seriously hampering my ability to use the forum.

[Galadriel55]

Originally Posted by Elvellon

I can't help but wonder why it seems like nobody is doing anything about the bots? I was hoping that bringing it up and pointing out the target of the bot activity would help spur some kind of change. Are there any admins or mods who can be contacted or who can comment on the situation? I don't know about anyone else, but the bot activity is seriously hampering my ability to use the forum.

The admins are working on it, but it's not something that they can fix immediately. Give them a bit of time to work on it.

In the meantime, though I get the 508 errors very frequently too, they don't really limit me all that much beyond being a mild annoyance. Just refresh the page once or twice, and you get through. It's annoying when it happens as you're submitting a post, cause then the site thinks you are trying to post twice and puts you in the "timeout zone" for posting in rapid succession, and I am always scared I will lose my post when this happens - but I've never lost it, and you literally click on any other page to get out of the timeout counter.

[Snowdog]

The bots are getting wily since all they have to talk to on Arwen Undomiel is each other. They have gone looking for more old tech boards to invade.

[Elvellon]

Originally Posted by Galadriel55

The admins are working on it, but it's not something that they can fix immediately. Give them a bit of time to work on it.

I appreciate that. That's the first acknowledgment I've seen that anyone is working on it, hence my comment, but maybe I missed it.

[Mithadan]

If you have recently entered the forums, you have all noticed the frequent 508 error messages (insufficient resources) preventing access to the Downs, I have attempted a fix.

A 508 message, based upon my feeble, tech-deficient research, is the result of a site's servers being overwhelmed by too many users viewing or interacting with the site. Others have commented upon the large number of visitors shown as being on the site, often more than 2500 (the counter shows that the most people on the site at one time was 6300 this past May).

Recently, Huinesoron noticed that one forum, The Movies, was consistently attracting more than 90% of the visitors. He also noticed that one thread, Full-Length LoTR Parody Movies, was showing an absurd amount of "hits," presently more than 22 million (not a typo) with the number increasing by hundreds of thousands every week. It appears that something in this thread has been attracting bots/AI scrapers. This is based upon observation, not technical confirmation. I know of no way to determine whether any particular post in the thread is attracting the attention.

The primary posters on this thread over the last several months have been Huinesoron and Galadriel55 who have been using it to develop their Finrod Rock Opera (a worthy endeavor if you have not noticed it). After discussing this with them, I have moved this thread into a private forum. It is no longer visible in The Movies. They will continue to privately work on the Finrod Opera in the private forum. I am sure that if any of you want to see it or follow their progress, they will accommodate you.

Before I moved the thread, The Movies forum showed roughly 2000 visitors. After I moved it, it showed 135 visitors. I suspect that the "visitors" that we viewing/mining the thread when it was moved will remain there, clogging our server until they leave the page they are viewing. After that, they will not be able to reach the thread, hopefully curing the 508 problem. Or maybe this ham-handed effort will not be a fix. We'll see.
__________________

[Huinesoron]

Thank you, Mithadan. Hopefully now we're in hiding things will settle down. If anyone still wants to see what's going on with Finrod-Zong, you can access the Google Doc of the current state of it through my signature or G55's. If the bots want to try and DDOS Google, they're welcome to!

(Credit where it's due, it was you and Elvellon who put together where the bots were swarming; I just theorised on why.)

At this time there are 2796 "guests" on the Downs, with only about 600 of them viewing specific forums. The rest are hopefully either still viewing the Zong thread until they refresh, have been bounced to the main page, or are hitting the "no such thread" page; in any of those cases I hope we'll soon see a drop-off in numbers.

EDIT: For reference/tracking, 15 minutes of "Viewing" and "User" counts. Times are in GMT+1, date is 2024-08-09; this is about half an hour after the Zong thread was moved.

hS

[Galadriel55]

I have absolutely no clue where you guys can see the visitors to specific fora. I can only find the guests listed on the main page - still around 2.8K, similar to Hui's observation a couple hours back. I still ran into the 508 twice while jumping back and forth between the forums. And views to the Zong are still at 22M - up from 17.9M a month ago as observed by Elvellon. Let's see how these numbers change over the coming days and keep our fingers crossed. Thank you Mithadan!

[Huinesoron]

Originally Posted by Galadriel55

I have absolutely no clue where you guys can see the visitors to specific fora. I can only find the guests listed on the main page - still around 2.8K, similar to Hui's observation a couple hours back. I still ran into the 508 twice while jumping back and forth between the forums. And views to the Zong are still at 22M - up from 17.9M a month ago as observed by Elvellon. Let's see how these numbers change over the coming days and keep our fingers crossed. Thank you Mithadan!

On the front page, I just see the viewer count next to each sub forum. So for instance "The Books (91 viewing)".

Currently we're at 6294 Guests, but only slight increases in each forum. So the bot servers have come online, but they're on the main or error pages for sure. Hopefully once they realise they're not reaching the correct page they'll back off.

hS

[Mithadan]

Interesting that just after the thread was made private, visitors shot up from about 2200 to a new record high of 6589....

[Huinesoron]

8:10 am at GMT+1, we have 5721 guests. Breakdown across forums, top to bottom:

318/109/194
181/33/208
62/5
15
2/6/3/67
153/5

I don't know if it's useful, but I'm trying to log the numbers at various times of day to see how they rise and fall.

EDIT: 9:05am GMT+1, also on 10 Aug: 5464 guests.

40/21/101
41/13/37
32/0
7
4/6/2/60
17/11

EDIT2: 10:46pm GMT+1, 10 Aug:

5292 guests
65/32/71
22/11/37
45/3
7
2/1/2/62
42/7

EDIT3: 11:53pm GMT+1, 10 Aug:

4927 guests
55/22/87
25/11/41
55/1
9
4/5/2/59
19/4

EDIT4: 9:22am GMT+1, 11 Aug

3191 guests
83/11/114
43/6/67
160/4
4
0/0/0/44
13/5

EDITAGAIN: Dropping the full logging until something interesting happens with it, but:

11 Aug, 9:43pm GMT+1: 3217 guests
10 Aug, 00:30am GMT+1: 7328 guests.
11 Aug, 11:07pm GMT+1: 3328 guests
12 Aug 6:49am GMT+1: 3313 guests
12 Aug 9:26am GMT+1: 3023 guests
12 Aug 11:49am GMT+1: 3140 guests
12 Aug 5:48pm GMT+1: 3165 guests
12 Aug 9:46pm GMT+1: 3135 guests
12 Aug 10:35pm GMT+1: 2847 guests
13 Aug 9:27 am GMT+1: 3072 guests
13 Aug 1:11pm GMT+1: 3140 guests (starting to see a theme here...)
13 Aug 9:49pm GMT+1: 3092 guests
13 Aug 10:55pm GMT+1: 3226 guests
14 Aug 12:34pm GMT+1: 2987 guests

hS

[Snowdog]

I find it amusing that a couple of the AI Bots posted to the discussion thread for my old fanfic story.

[Galadriel55]

I started paying more attention to the numbers too, and it's visobly noticeable that the amount of 508s correlates. I hardly get them at 2000, but get them at every turn at 4-5K. And each time I observe, relatively few are sitting in specific fora - less than a thousand total. Most are just hanging out. The sad thing is that they really don't appear to be falling off. The numbers fluctuate, but aren't really decreasing. Would a captcha "capture" these rogues, do you think?

[Elvellon]

Not sure if this is helpful, but dropping this here:

https://forum.vbulletin.com/forum/vb...09#post4482109

[Huinesoron]

Originally Posted by Elvellon

Not sure if this is helpful, but dropping this here:

https://forum.vbulletin.com/forum/vb...09#post4482109

If accurate, this idea would seem like a good solution:

Originally Posted by The link

You can also reduce load for your registered users by increasing the value of the Minutes to Cache Content for Guests option on the same page. Guests don't need up to the second updates on content. So you can set this to 5 or even 10 minutes.

So guests (including real guests) would only get a new snapshot of the site every few minutes, while logged-in users would still get it in real-time. My limited understanding of IT systems says this ought to work - not to get rid of the bots, but to stop the 508 errors almost completely.

hS

[Huinesoron]

I don't want to jinx it, but the numbers I wrote a few posts above were all 3000+; since I stopped recording them, I've only seen one 3000+ guest count, with everything else being in the 2000s. (I think I saw a 2400 at one point!) Maaaaybe the bots are slowly winding down?

16 Aug, 09:48 GMT+1: 2874 guests.
16 Aug, 11:43 GMT+1: 2750 guests
16 Aug, 12:54 GMT+1: 2855 guests
16 Aug, 14:14 GMT+1: 2806 guests
16 Aug, 22:36 GMT+1: 2803 guests
17 Aug 14:32 GMT+1: 2755 guests
17 Aug 21:34 GMT+1: 2747 guests
17 Aug 22:47 GMT+1: 2675 guests
19 Aug 10:25 GMT+1: 2817 guests
19 Aug 13:48 GMT+1: 2949 guests
20 Aug 13:48 GMT+1: 2604 guests. Earlier today I noticed another 26xx and a 27xx.
20 Aug 14:36 GMT+1: 2651 guests

hS

[Huinesoron]

I've kept tracking guest counts occasionally, and over the past 5 days it kind of looks like there's a pattern:



The bottom of the "trough" on the 22nd is at 1pm GMT+1, which means if this pattern is real and not just a data artefact, the bots are cycling from ~2600 at GMT noon to ~3000 at GMT midnight and back.

I'm not sure that qualifies as useful information, unfortunately.

EDIT: I've also just had my daily "this website has no security certificate and is going to eat your computer" warning at 1pm GMT+1. I'm not sure if that's my computer waiting a day between sending the message, or if the Downs does something at GMT noon that could be linked to both issues.

hS