[ensembl-dev] Duration of 403 error?

[Kieron Taylor]

Hi Melissa,

Normally the X-RateLimit headers are sufficient to avoid our client sin bin. If you ignore the rate limit, you get blacklisted for what I believe is a two hour period, as if you had been running at half the recommended rate. We do this to protect our servers from excessive stress from runaway code, or very greedy use.

The response headers that tell you your "budget" for the next time period can be accessed via Python's HTTPResponse object if you are using the standard HTTP client library.

e.g. 
conn.request(...)
response = conn.getresponse()
countdown = response.getheader("X-RateLimit-Remaining", default="54999");

If you're splitting the time period to calculate how long to wait before sending your next request, beware of rounding the result down. You can also just pre-emptively cap yourself at 15 requests per second directly. Either way, always observe the response code - some of our users can receive thousands of 400 errors due to mistakes in their requests.

Regards,

Kieron



> On 27 Jun 2016, at 15:15, Melissa van Wieringen <m.vanwieringen at omnigen.nl> wrote:
> 
> Hello,
> 
> While testing code for my internship, I may have overloaded your REST API. 
> As a result, I suppose, I have been getting a 403 response. (HTTPError: 403 Client Error: Forbidden, "You are submitting far too many requests and have been temporarily forbidden access to the service. Wait and retry with a maximum of 15 requests per second." -https://github.com/Ensembl/ensembl-rest/wiki/HTTP-Response-Codes)
> 
> While trying to figure out how long my access would be restricted, I stumbled across HTTP Headers, which Ensembl REST supposedly provides. (https://github.com/Ensembl/ensembl-rest/wiki/HTTP-Headers)
> I have tried the following python code, but it doesn't provide any of the 'special' extras, as the waiting time or number of calls left.
> 
> server = "http://rest.ensembl.org"
> site = server + ext.format(rsid=variant, downloadoption=option)
> r = requests.get(site, headers={"Content-Type": "application/json"})
> print(r.headers)
> > CaseInsensitiveDict({'connection': 'close', 'content-type': 'text/html', 'cache-control': 'no-cache'})
> 
> My questions are: How long will my access to the Ensembl REST API be restricted & How can I check X-RateLimit-Limit, X-RateLimit-Reset, X-RateLimit-Period, X-RateLimit-Remaining myself?
> Hoping to hear from you soon.
> 
> With kind regards,
> Melissa van Wieringen,
> Intern at Omnigen.
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/