API Request Times out for Certain Queries

Avatar image for cbanack
cbanack

124

Forum Posts

199

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#1  Edited By cbanack

Hi guys,

This week I've been getting a lot of reports from the users of my third party app that say they are suddenly unable to use the Comic Vine API to search for a couple of specific volumes. So I investigated, and I found that when you use the Search API to search for those two volumes, the http request will always time out and you'll get either a 500 or a 502 (bad gateway) error.

This is a new behaviour. It might indicate that some kind of infinite loop or something similar was recently introduced on the server side of Comic Vine's public API. What follows is a specific description of the problem. Note: you will have to substitute your own API key into the URL's I've provided.

-------

Say I'm searching for "Superman" volumes. The API query might look like:

https://comicvine.gamespot.com/api/search/?api_key=[YOUR-API-KEY]&format=xml&limit=20&query=superman&resources=volume

But as of this moment, that query always times out. If you change the search term to something else that will return a lot of volumes ('spiderman' or 'batman') it still works fine and returns results quickly.

What about if I'm searching for "Cyborg"? As of this writing, there's only 26 volumes that should be returned by this http request:

https://comicvine.gamespot.com/api/search/?api_key=[YOUR-API-KEY]&format=xml&limit=100&query=cyborg&resources=volume

But this request times out too! Note that limit=100 and query=cyborg.

If I switch to limit results to 20 volumes instead, it works fine:

https://comicvine.gamespot.com/api/search/?api_key=[YOUR-API-KEY]&format=xml&limit=20&query=cyborg&resources=volume

In fact, if I try anything less than 20 issues, it also works fine. But 21 issues or more, and it always times out. Also, for the first query I mentioned (superman), it times out no matter what I set the limit to.

-------

Hopefully this is enough info to help you guys track down the bug. It is probably related to changes that were made in the last week or so, since my app has a lot of users and they all started noticing problems on Wednesday. The problems have all been related to the same queries I've described above ('cyborg' and 'superman', limit=100).

If I can help verify a fix or provide more information, just let me know.

Avatar image for solidus0079
solidus0079

11

Forum Posts

174

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#2  Edited By solidus0079

Here's a link to the Cyborg #18 page, another issue that is causing problems. It can't even be viewed in a browser, I get a "Something's Wrong T_T" error. . https://comicvine.gamespot.com/cyborg-18-singularity-aftermath-the-end/4000-634508/ so this isn't just an API or 3rd party program problem.

Avatar image for pikahyper
pikahyper

19027

Forum Posts

37057995

Wiki Points

0

Followers

Reviews: 5

User Lists: 581

#3  Edited By pikahyper  Moderator

@solidus0079: page is viewable now, the site was having some outage problems through out the week.

Avatar image for solidus0079
solidus0079

11

Forum Posts

174

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Thanks, pkahyper -

But now something else seems to be going on. Scraping any comics at all seems to result in an IP ban. I'm posting via the Tor browser at the moment.

Avatar image for pezh0re
pezh0re

2

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#5  Edited By pezh0re

I'm also seeing an IP ban whenever attempting an API call. Attempting to reach the first url in op (https://comicvine.gamespot.com/api/search/?api_key=[YOUR-API-KEY]&format=xml&limit=20&query=superman&resources=volume) resulted in a page saying site undergoing maintenance... is there an ongoing issue with the API service?

Avatar image for chrisravnos
Crave

1

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

I am also getting IP banned for 30+ minutes for running any API query.

Avatar image for solidus0079
solidus0079

11

Forum Posts

174

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Seems resolved for now at least. :)

Avatar image for pezh0re
pezh0re

2

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

@solidus0079: Yup, looks good on for me too. I wonder what happened....

Avatar image for admjim
admjim

3

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

still having issues with Superman. Although the message I get is different "The Comic Vine online database could not be reached. This may be a temporary technical error with the database, or there may be a problem with your internet connection."

I am able to get info for other comics right after this error. Prior to today I was unable to get info for any book, even though I was able to seemy current api usage showing I was within the limits. BTW this started 3 days ago in the middle of a request for 34 book updates. Let me know if there is any other info you need.

Avatar image for solidus0079
solidus0079

11

Forum Posts

174

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

@admjim I was able to scrape that issue just fine after whatever fix was fixed.

Avatar image for romsnesrom
romsnesrom

11

Forum Posts

1864

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

@solidus0079: page is viewable now, the site was having some outage problems through out the week.

Is the outage problem solved? I am still having trouble scraping. Not only superman and cyborg, but anything past 20 queries give or take.

Avatar image for solidus0079
solidus0079

11

Forum Posts

174

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

@romsnesrom: I just did over 400 scrapes, including the problematic Superman and Cyborg issues. Not one hiccup.

Avatar image for cbanack
cbanack

124

Forum Posts

199

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#13  Edited By cbanack

Just to clarify, the original bug I reported in this thread is still occurring (i.e. can't search for Superman or Cyborg with the API).

People have started discussing a separate, unrelated bug (IP bans) in this thread as well.

Avatar image for admjim
admjim

3

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Just to clarify my issue is still going on I cannot scrape the Superman Rebirth issues 30-34 however the message I get is as stated earlier it times out and then gives "The Comic Vine online database could not be reached. This may be a temporary technical error with the database, or there may be a problem with your internet connection." I attempted to scrape each issue one at a time and as a group with other books, each time it failed on the Superman Book.

I do not get a either a "500 or a 502 (bad gateway) error.

I am able to scrape other Superman issues from other years as well as other books not related to Superman. Allof this is within the last 15 minutes.

I have not tested the Cyborg issue so I cannot comment on that.

Avatar image for cbanack
cbanack

124

Forum Posts

199

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#15  Edited By cbanack

@admjim said:

"The Comic Vine online database could not be reached. This may be a temporary technical error with the database, or there may be a problem with your internet connection."

<snip>

I do not get a either a "500 or a 502 (bad gateway) error.

That error you are getting in the Comic Vine Scraper application (which is not created by nor directly supported by the comicvine.com) is actually being caused by 500 and 502 errors that Comic Vine Scraper encounters when it tries to access the Comic Vine API.

My original bug report is just showing those failing API calls directly (and taking Comic Vine Scraper out of the equation) but we are actually reporting the same problem.

Avatar image for steveeasley
steveeasley

18

Forum Posts

101

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#16 steveeasley  Staff

This should now be resolved, thanks for all the reports. We believe it was being caused by Comic Vine's own network provider misidentifying the CVS app as a threat. They have removed the signature causing the issue.

Let me know if its still a problem.

Avatar image for chrishadley
Chrishadley

5

Forum Posts

276

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

I have been having severe problems using the Comic Vine scraping script for Comicrack for the past few days. I've been getting the error "The Comic Vine online database could not be reached. This may be due to a temporary technical error with the database, or there may be a problem with your internet connection.". This problem has cleared up today, except for scraping comics with very generic names. For example, I have a copy of "Marvels" that I can't scrape for. It acts like it's working, and when it gets to 500 entries it stops, and gives me the "Cannot Access Online Database" error.

Avatar image for romsnesrom
romsnesrom

11

Forum Posts

1864

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

This should now be resolved, thanks for all the reports. We believe it was being caused by Comic Vine's own network provider misidentifying the CVS app as a threat. They have removed the signature causing the issue.

Let me know if its still a problem.

The problem persists for me, Steve. If there is another data I could provide to help diagnose the situation, just ask.

Avatar image for steveeasley
steveeasley

18

Forum Posts

101

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#19 steveeasley  Staff

@romsnesrom What are your specific symptoms? The more detail the better.

Avatar image for steveeasley
steveeasley

18

Forum Posts

101

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#20  Edited By steveeasley  Staff

@chrishadley We found the issue. Its unrelated to the issue we already fixed, but would have the same timeout symptoms. We will get a fix out asap and I will report back. (Thanks for the detail about results after 500 being the issue, that was the magic detail that narrowed it down for us).

@romsnesrom This might be the source of your issue as well.

Avatar image for solidus0079
solidus0079

11

Forum Posts

174

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Thanks for all the help of late, Steve.

Avatar image for robinsonr74
robinsonr74

6

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#22  Edited By robinsonr74

Superman, Cyborg, and 'The Order', here, still not working.

Avatar image for chrishadley
Chrishadley

5

Forum Posts

276

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#23  Edited By Chrishadley

I'm still having the issue with items with very generic names, like 'Marvels', 'Marvel Super-Heroes', 'The Cat', and also, amusingly enough, 'Cyborg' and 'The Order'. It looks like it is hitting a limit of number of responses and locking up. I understand there may be a limit of responses on the API? Is there some way to rate-limit Comic Vine Scraper? I found no way to do that without rewriting it. It will let you pause between comics, but there's no way to limit the number of responses on a single comic. Thank you so much for your help so far!

Edit: Also, it's getting to 600 responses before dying now :)

Avatar image for romsnesrom
romsnesrom

11

Forum Posts

1864

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

@chrishadley We found the issue. Its unrelated to the issue we already fixed, but would have the same timeout symptoms. We will get a fix out asap and I will report back. (Thanks for the detail about results after 500 being the issue, that was the magic detail that narrowed it down for us).

@romsnesrom This might be the source of your issue as well.

I am happy to report that most of the problem is gone. I am still having problem with the series everyone is already complaining: "Cyborg", "Superman", "The Order", "Armies".

Avatar image for admjim
admjim

3

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

@cbanack: or @steveeasley As of right now (Sunday 11/19/17 11:15am CT) I am still having issues with Superman 30-34 (2017) and Cyborg 17 &18 (2017). I have been using the script on and off for the last 3 days and have never been able to get these books to scrape yet I am able to get 100's of others to wrok with now issue's. Please let me know if there is something I can do to help trace the issue. Maybe setup a Date/Time to test. I know what to do at this point to break it ;)

Avatar image for jmducat
jmducat

4

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Hi all - I'm having the same issues as @admjim. I can't scrape Cyborg 18 and Superman 34, 35. Just did a bunch of other scraping. I tried linking to those addresses in post #1 with my API key and got the 502 error.

Avatar image for steveeasley
steveeasley

18

Forum Posts

101

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

#29 steveeasley  Staff

Can you guys try again? We just released the fix I mentioned above related to some timeouts.

Avatar image for chrishadley
Chrishadley

5

Forum Posts

276

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

It looks like all of the ones that scraping was failing on are now working, I've cleared my whole backlog successfully. You rock!

Avatar image for jmducat
jmducat

4

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Same here - everything scraped with no issues. Thank so much!