General » Site/Ad Issues
Reminder of the previous message
Site/Ad Issues
Published 11/07/2023 @ 11:59:17, By antp
I see that in the last two days the "visitor" with the highest number of request was an address in Finland using this software:
https://wiki.archiveteam.org/index.php/ArchiveBot
2nd place was Bingbot, with less than half of requests, but normally it runs only in low-traffic hours now.
The first is probably not the bot of the internet wayback machine (archive.org), cf these points:
and later:
So I suppose I could block it...
Via http://archivebot.com/3 we can see it in realtime, and indeed it is fetching pages too quickly. Now page requests appear in red there, with a 503 error code (service unavailable) instead of the usual "200 ok".
I hope it will be less a burden for imcdb...
edit: I found how to contact them, so they reduced the frequency, it should work better then. Please keep notify me if you see an improvement, or if it is still very slow.
Latest Edition: 11/07/2023 @ 12:25:01
https://wiki.archiveteam.org/index.php/ArchiveBot
2nd place was Bingbot, with less than half of requests, but normally it runs only in low-traffic hours now.
The first is probably not the bot of the internet wayback machine (archive.org), cf these points:
We're not Internet Archive. (We do what we want.)
We're not the Wayback Machine. Specifically, we are not ia_archiver or archive.org_bot. (We don't run crawlers on behalf of other crawlers.)
We're not the Wayback Machine. Specifically, we are not ia_archiver or archive.org_bot. (We don't run crawlers on behalf of other crawlers.)
and later:
Also, please remember that we are not the Internet Archive.
So I suppose I could block it...
Via http://archivebot.com/3 we can see it in realtime, and indeed it is fetching pages too quickly. Now page requests appear in red there, with a 503 error code (service unavailable) instead of the usual "200 ok".
I hope it will be less a burden for imcdb...
edit: I found how to contact them, so they reduced the frequency, it should work better then. Please keep notify me if you see an improvement, or if it is still very slow.
Latest Edition: 11/07/2023 @ 12:25:01
Site/Ad Issues
Published 17/07/2023 @ 20:55:16, By night cub
Site is slow today. Taking forever for pages to load.
Site/Ad Issues
Published 22/07/2023 @ 22:28:47, By night cub
Yikes the site is super slow today. Was going to validate, but it's taking too long for the pages to load. Will have to try later.
Site/Ad Issues
Published 26/07/2023 @ 03:50:08, By night cub
Don't know what is happening right now, but the site just slowed down to a crawl. I'm in the middle of replacing 109 pics on Daredevil Drivers and it was working ok, then boom, every click is taking 30s + to load.
Site/Ad Issues
Published 26/07/2023 @ 10:08:49, By antp
The most active visitors on pages (excluding images) in the hours around your post here were the Bing bot, the Google bot... and you
But there are of course a lot of other visitors, I assume the problem is just that a lot of people browse it at the same time
However something interesting: the daily backup was done at 3:44, you posted here at 3:50 (CET), it was probably not a coincidence
The time it is done is not exactly the same each day (changes by a few minutes)
I should try to change the time to use a time slot with less traffic.
Latest Edition: 26/07/2023 @ 10:09:10
But there are of course a lot of other visitors, I assume the problem is just that a lot of people browse it at the same time
However something interesting: the daily backup was done at 3:44, you posted here at 3:50 (CET), it was probably not a coincidence
The time it is done is not exactly the same each day (changes by a few minutes)
I should try to change the time to use a time slot with less traffic.
Latest Edition: 26/07/2023 @ 10:09:10
Site/Ad Issues
Published 26/07/2023 @ 11:54:09, By night cub
Well I could definitely tell when one of those bots kicked in. Like I said, I was in the middle of replacing pics and one moment it was working fine, then it suddenly slowed down to a crawl. It was just frustrating because I was down to the last 20-30 pics to replace. It was several minutes before it returned to normal.
Site/Ad Issues
Published 26/07/2023 @ 12:12:15, By antp
In this particular case the slowdown was maybe rather the backup.
I changed its time so it occurs later (between 6 and 8 AM CET, so a little before or after midnight on your time, depending on which coast you are).
I'm not sure it will work, I'll check tomorrow if the setting is well applied.
About the bots, Bing is less active at these times (theoretically) but Google does not allow any control on that
I changed its time so it occurs later (between 6 and 8 AM CET, so a little before or after midnight on your time, depending on which coast you are).
I'm not sure it will work, I'll check tomorrow if the setting is well applied.
About the bots, Bing is less active at these times (theoretically) but Google does not allow any control on that
Site/Ad Issues
Published 31/07/2023 @ 20:01:20, By night cub
It's 2pm here, and the site is dragging again. Not sure if there are any bots trawling right now, but it's taking forever to do simple edits and validating.
Site/Ad Issues
Published 31/07/2023 @ 20:34:01, By antp
Right now it seems better, maybe a temporary peak in visits? I'll check tomorrow in the logs
Site/Ad Issues
Published 01/08/2023 @ 20:34:50, By night cub
Don't know if you looked at the logs, but having the same issue today at the same time. Started validating around 2pm, and it's a little better than yesterday but still taking a while for pages to load when editing.
Site/Ad Issues
Published 02/08/2023 @ 08:37:52, By antp
I did not had time to check that, I'll try to do that. Strange that it is at the same time... Is it just more visitors at that time? Or someone doing something that cause the issue?
Site/Ad Issues
Published 02/08/2023 @ 11:49:11, By antp
As usual Google bot & Bing bot are in the top of the list.
I see that the Google bot competely ignores the rare set in robots.txt, they fetch pages at a too high rate (much more than a normal user browsing the site). They used to have an option to temporary reduce the rate, but it does not seem to work
Yesterday we also had that one joining the party:
https://dataforseo.com/dataforseo-bot
As well as the Amazon bot
The first IP address of a (probably) real user is at the 10th place in the list...
Latest Edition: 02/08/2023 @ 11:57:47
I see that the Google bot competely ignores the rare set in robots.txt, they fetch pages at a too high rate (much more than a normal user browsing the site). They used to have an option to temporary reduce the rate, but it does not seem to work
Yesterday we also had that one joining the party:
https://dataforseo.com/dataforseo-bot
As well as the Amazon bot
The first IP address of a (probably) real user is at the 10th place in the list...
Latest Edition: 02/08/2023 @ 11:57:47
Site/Ad Issues
Published 02/08/2023 @ 20:00:01, By night cub
Day 3 - same time 2pm EDT-US - just timed how long it took to load the All Comments page, 1:06. Yes, you reading that correctly, over 1 minute to load the page fully. The Trawlers have taken over.
Site/Ad Issues
Published 02/08/2023 @ 20:51:11, By walter.
I've just given up logging in too right now, this is a pain.
Site/Ad Issues
Published 02/08/2023 @ 21:07:56, By 48bux
I had some serious trouble in uploading the show I was working on, too..and opening the site and the all comments page takes very long
Latest Edition: 02/08/2023 @ 21:10:10
Latest Edition: 02/08/2023 @ 21:10:10
Site/Ad Issues
Published 02/08/2023 @ 22:47:35, By dhill_cb7
Site seems crippled right now.
Site/Ad Issues
Published 03/08/2023 @ 00:35:19, By night cub
This is now 4.5 hours of the site being unusable. Don't these trawlers ever end?
Site/Ad Issues
Published 03/08/2023 @ 12:16:09, By antp
Between the time of your message until now, Google is clearly the one who made the most requests. I don't know if it is the culprit, or just a combination of that and the amount of visitors (and also the database getting always bigger, thanks background useless cars )
I've set a setting in Google tools to reduce the frequency, I don't know if it will work as this is a setting from an old part of the old console, the new one does not have it.
Also, it can't be set for the whole domain, so in case of IMCDb there is with or without www, with or without https. I've set it just for the www+https version as it seems that Google rather uses that one.
We'll see if it improves the site speed in the evening.
Latest Edition: 03/08/2023 @ 12:17:18
I've set a setting in Google tools to reduce the frequency, I don't know if it will work as this is a setting from an old part of the old console, the new one does not have it.
Also, it can't be set for the whole domain, so in case of IMCDb there is with or without www, with or without https. I've set it just for the www+https version as it seems that Google rather uses that one.
We'll see if it improves the site speed in the evening.
Latest Edition: 03/08/2023 @ 12:17:18
Add Reply