IMCDb Forum
General » Site/Ad Issues
Category:  
« Previous topic
Reminder of the previous message
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 11/07/2023 @ 00:06:28, By night cub
Site seems to be at a standstill right now


Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 11/07/2023 @ 11:59:17, By antp
I see that in the last two days the "visitor" with the highest number of request was an address in Finland using this software:
https://wiki.archiveteam.org/index.php/ArchiveBot
2nd place was Bingbot, with less than half of requests, but normally it runs only in low-traffic hours now.
The first is probably not the bot of the internet wayback machine (archive.org), cf these points:
We're not Internet Archive. (We do what we want.)
We're not the Wayback Machine. Specifically, we are not ia_archiver or archive.org_bot. (We don't run crawlers on behalf of other crawlers.)

and later:
Also, please remember that we are not the Internet Archive.

So I suppose I could block it...
Via http://archivebot.com/3 we can see it in realtime, and indeed it is fetching pages too quickly. Now page requests appear in red there, with a 503 error code (service unavailable) instead of the usual "200 ok".
I hope it will be less a burden for imcdb...

edit: I found how to contact them, so they reduced the frequency, it should work better then. Please keep notify me if you see an improvement, or if it is still very slow.

Latest Edition: 11/07/2023 @ 12:25:01
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 17/07/2023 @ 20:55:16, By night cub
Site is slow today. Taking forever for pages to load.
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 17/07/2023 @ 21:59:50, By walter.
Same here, it's such a pain.
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 17/07/2023 @ 22:23:32, By Baube
same here
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 22/07/2023 @ 22:28:47, By night cub
Yikes the site is super slow today. Was going to validate, but it's taking too long for the pages to load. Will have to try later.
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 26/07/2023 @ 03:50:08, By night cub
Don't know what is happening right now, but the site just slowed down to a crawl. I'm in the middle of replacing 109 pics on Daredevil Drivers and it was working ok, then boom, every click is taking 30s + to load.
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 26/07/2023 @ 10:08:49, By antp
The most active visitors on pages (excluding images) in the hours around your post here were the Bing bot, the Google bot... and you :grin:
But there are of course a lot of other visitors, I assume the problem is just that a lot of people browse it at the same time :ohwell:

However something interesting: the daily backup was done at 3:44, you posted here at 3:50 (CET), it was probably not a coincidence :figti:
The time it is done is not exactly the same each day (changes by a few minutes)

I should try to change the time to use a time slot with less traffic.

Latest Edition: 26/07/2023 @ 10:09:10
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 26/07/2023 @ 11:54:09, By night cub
Well I could definitely tell when one of those bots kicked in. Like I said, I was in the middle of replacing pics and one moment it was working fine, then it suddenly slowed down to a crawl. It was just frustrating because I was down to the last 20-30 pics to replace. It was several minutes before it returned to normal.
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 26/07/2023 @ 12:12:15, By antp
In this particular case the slowdown was maybe rather the backup.
I changed its time so it occurs later (between 6 and 8 AM CET, so a little before or after midnight on your time, depending on which coast you are).
I'm not sure it will work, I'll check tomorrow if the setting is well applied.

About the bots, Bing is less active at these times (theoretically) but Google does not allow any control on that :sad:
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 31/07/2023 @ 20:01:20, By night cub
It's 2pm here, and the site is dragging again. Not sure if there are any bots trawling right now, but it's taking forever to do simple edits and validating.
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 31/07/2023 @ 20:34:01, By antp
Right now it seems better, maybe a temporary peak in visits? I'll check tomorrow in the logs
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 01/08/2023 @ 20:34:50, By night cub
Don't know if you looked at the logs, but having the same issue today at the same time. Started validating around 2pm, and it's a little better than yesterday but still taking a while for pages to load when editing.
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 02/08/2023 @ 08:37:52, By antp
I did not had time to check that, I'll try to do that. Strange that it is at the same time... Is it just more visitors at that time? Or someone doing something that cause the issue?
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 02/08/2023 @ 11:49:11, By antp
As usual Google bot & Bing bot are in the top of the list.
I see that the Google bot competely ignores the rare set in robots.txt, they fetch pages at a too high rate (much more than a normal user browsing the site). They used to have an option to temporary reduce the rate, but it does not seem to work :ohwell:

Yesterday we also had that one joining the party:
https://dataforseo.com/dataforseo-bot
As well as the Amazon bot

The first IP address of a (probably) real user is at the 10th place in the list...

Latest Edition: 02/08/2023 @ 11:57:47
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 02/08/2023 @ 20:00:01, By night cub
Day 3 - same time 2pm EDT-US - just timed how long it took to load the All Comments page, 1:06. Yes, you reading that correctly, over 1 minute to load the page fully. The Trawlers have taken over.
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 02/08/2023 @ 20:51:11, By walter.
I've just given up logging in too right now, this is a pain.
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 02/08/2023 @ 21:07:56, By 48bux
I had some serious trouble in uploading the show I was working on, too..and opening the site and the all comments page takes very long :ohwell:

Latest Edition: 02/08/2023 @ 21:10:10
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 02/08/2023 @ 22:47:35, By dhill_cb7
Site seems crippled right now.
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 03/08/2023 @ 00:35:19, By night cub
This is now 4.5 hours of the site being unusable. Don't these trawlers ever end?
Direct link to this message Edit  Quote  Add this message as quote for multiple quotes  Delete  Top  Bottom
Site/Ad Issues
Published 03/08/2023 @ 12:16:09, By antp
Between the time of your message until now, Google is clearly the one who made the most requests. I don't know if it is the culprit, or just a combination of that and the amount of visitors (and also the database getting always bigger, thanks background useless cars :whistle: )

I've set a setting in Google tools to reduce the frequency, I don't know if it will work as this is a setting from an old part of the old console, the new one does not have it.
Also, it can't be set for the whole domain, so in case of IMCDb there is with or without www, with or without https. I've set it just for the www+https version as it seems that Google rather uses that one.
We'll see if it improves the site speed in the evening.

Latest Edition: 03/08/2023 @ 12:17:18
Add Reply - Category:  
Add Reply
Bold [b]Text[/b] Italic [i]Italic[/i] Underline [u]Underline[/u] Strike Out [strike]Strike Out[/strike]
Email [email=nobody@nobody.org]Name[/email] Link [url=http://www.website.com]Text[/url] Anchor [anchor]Name[/anchor] Image [img]http://www.website.com/image.jpg[/img]
Align Left [align=left]Text[/align] Centered [align=center]Text[/align] Align Right [align=right]Text[/align] Text Justify [align=justify]Text[/text]
Color [color=#000000]Text[/color] Highlight [highlight=pascal]Text[/highlight] Widgets Smileys :code: [:code] HTML to BBCode converter Word to BBCode converter
Preview Spell Checker






Ada
CSS
Cobol
CPP
HTML
Fortran
Java
JavaScript
Pascal
Perl
PHP
Python
SQL
VB
XML
Anon URL
DailyMotion
eBay
Flickr
FLV
Google Video
Metacafe
MP3
SeeqPod
Veoh
Yahoo Video
YouTube
Sign In :: Sign Up :: Lost your login or your password?
KelCommunity.be :: © 2004-2024 Akretio SPRL :: Powered by Kelare