Site keeps going down

It's been like five or six times in the past 4 hours and it stays down for about five minutes each time :/

Site just went down again twice, right at 3am PST, came back for less then a minute then went down again for about two more minutes.

Same problem happened yesterday, I think it's what happens when they're doing maintenance or changes to the site... but it kept happening repeatedly for like thirty minutes, during the busiest time of comicvine.

That was due to Bing.com crawling us repeatedly; we made some changes and hopefully it won't happen again.

Sites been down three five times in the past 25 minutes.

The sites been going down every five minutes since 3:50am PST.

The site still keeps going down, it's up for a couple minutes then down for a few on repeat now.

@renchamp: that would probably be more fun and interesting then the real reason for the problems :/

It's starting to happen to me now.

@humanrocket: if it's happening to me it is happening to everyone, if it was just me I wouldn't keep posting here.

@renchamp: editing is even worse :/

@renchamp: eh if I could stop doing it I probably would >.<

Its up, its down, its down, its up then its down again. All day. Kinda frustrating.

@strigidae_23: @pikahyper: @renchamp: I think this is still the Bingbot crawling us; it's been doing so for a few days. We've looked into trying to get them to stop but I think we're just going to give the bot an IP ban until we can assure this doesn't happen again. Sorry for the issues.

@rorie: oh ya ban the hell out of that thing, they should be owing the site some compensation for this amount of downtime.

@rorie: Its been stable for a few hours now, thanks for looking into it for us.

Considering the dozen other unfixed bugs in circulation for about a year I think they would lose that negotiation..

The site keeps going down again, don't tell me you guys didn't ban that stupid Bing bot?????

Here we go again...

The 502's and blank SF pages have started again.

If you see the other post in this forum entitled "Can't scrape anymore?" you'll see an approximate cause for this. Sometimes other sites and apps can hit our API so much that it'll cause site stability issues. We're looking into ways to throttle high-volume scrapers; it hasn't always been an issue but it's gotten worse recently apparently. Sorry for the ups and downs.

@rorie: Can they be banned for now? I don't mean to be mean and I don't pretend to understand what scraping is but basically they're taking the entire site off line over and over. This is like a DOS attack or something. It might not be malicious in intent but its harmful in effect.

ya I've got to agree, as soon as these things happen they need to be temp banned even if for just a day, its one line in an htaccess file for god sakes "deny from X.X.X.X" or if you want to get fancy:

RewriteEngine on

Rewrite Cond %{HTTP_USER_AGENT} ^ScraperAgent

or

RewriteEngine on

RewriteCond %{REMOTE_ADDR} ^X\.X\.X\.X

RewriteRule ^ - [F]

It's not time consuming or hard to do.

Every time I think I'm gonna actually get to some editing that doesn't involve the weekly new releases I get screwed by the site...

It happens ALL the time.

This.

I am experiencing the white screen problems (page not loading), and it is happening on all my devices when i get on comicvine for like 5 minutes randomly. Looks like the site momentarily goes down ): Rather annoying

It's getting stupid now, happened for like 2 hours on and off.

Yeah, I apologize for this this morning - it looks like three of our servers went down (which causes stress to all the other servers, obviously) and did not immediately respond to attempts to bring them back up. I'm not sure what caused the servers to go down in the first place, unfortunately.

We are going to move the site to a new colocation facility entirely sometime soon; hopefully that will improve site stability when that happens. Apologies for the disruptions; I know the site's been up and down quite a bit lately.

For the love of god please convince them to do that on a Thursday, Friday or Saturday cause there is no way in hell that's going to go smoothly, that sounds like a week of non-stop instability, bad performance and crashes >.<

just a quick postmortem on the outage this morning, it was once again the scraper (hit us over 800k time in the period of 2-3 hours). I've blocked them for now and am working with the owner of that app.

@pikahyper: I'm pretty sure we're going to try and do it on a Thursday. I'll try to make a post indicating that there might be some weirdness when we get closer to the day.

@rorie: perfect, that way if things go wrong it'll only make it hard to edit for one week instead of two.

Any update on the scraper status? Or will the new colocation facility alleviate that issue? Thanks!

Should comic vine scraper still be working? Because right now it's not for me.

we've put an api limiter in place and haven't seen any spikes since *knock on wood* that have taken the site down.

Site's lagging horribly and failing to load, again.

been working fine for me