[ensembl-dev] EnsEMBL REST server gateway timeout

[Kieron Taylor]

Hi Pablo,

I'm happy to inform you that we have already made an implementation of 
POST support for sequences. It should appear in our next release, but 
will also be subjected to limits. Large scale sequence requests should 
still be directed to BioMart or our FTP site.

Thank you for using the service respectfully. Our issues yesterday 
appear to be related to excessive/extensive use of the overlap endpoint 
by another IP domain than yours. Our rate limiter will identify a 
firewalled institution as a single user at this time, which can lead to 
unexpected limits, but you should still receive responses immediately 
that tell you when to try again. The response headers contain a 
suggested waiting time.

Please also bear in mind that your institution gateway can become 
saturated by other users, hence your own traffic can be delayed before 
it ever reaches Ensembl. As ever, we are striving to achieve the best 
compromise between performance, utility, and reliability. Your feedback 
is invaluable.

Regards,

Kieron

-- 
Kieron Taylor PhD.
Ensembl Core team
EBI


On 13/11/2014 14:14, Paolo Cozzi wrote:
>
> Dear Magali,
>
> Thank you for your reply. I've started from yours python client example
> in order to perform 15 request/second as yours directives suggest. The
> VEP requests are handled via POST interface, and the client submits (at
> the moment) up to 50 entries per request. I think that POST request are
> very useful: maybe could be interesting to develop the same protocol
> even for the Sequence endpoint.
>
> I've another question: I see that the time needed to perform a query
> range from 0 to 5 seconds, regardless of the query I do. Could be a
> problem due to my network configuration? If I do the same query outside
> my institution, I don't see this latency. Maybe this behavior is due to
> rate limit? (I saw that my institution is rate limited ) or there is a
> kind of cache in queries?
>
> Thank you for your support,
>
> regards,
>
> Paolo
>
>
>
> Il 13/11/2014 14:07, mag ha scritto:
>> Hi Paolo,
>>
>> As long as you have a system in place to respect the 15
>> requests/second limit, the server should be able to cope with your
>> request.
>>
>> If you are using the VEP endpoint, it also supports POST request.
>> This allows you to submit several (up to 1000) entries in a single
>> request.
>> It reduces the overhead in accessing the server, and means you can
>> technically submit 15,000 entries per second rather than 15.
>>
>> There is documentation on how to set up a POST request here:
>> https://github.com/Ensembl/ensembl-rest/wiki/POST-Requests
>>
>> Please give it a try if you find this useful, we'd like to get
>> feedback on it and expand it to more endpoints if it proves successful.
>>
>>
>> Regards,
>> Magali
>>
>> On 13/11/2014 10:09, Paolo Cozzi wrote:
>>>
>>> Dear Kieron,
>>>
>>> Thank for your support. In my institution we are developing an
>>> application which use REST interface to query VEP endpoint in
>>> particular. I think that in production environment we don't need to
>>> do a lot of requests on your server (we take account of the
>>> rate-limit conditions of the service, but I saw that users do 1 or 2
>>> requests per day), but maybe did I stress your server in this days
>>> during developing? If so, sorry for the inconvenience. Now I think
>>> that things are quite stable, and our requests will be fewer.
>>>
>>> Thanks again and thanks to ensembl team for support,
>>>
>>> Regards,
>>>
>>> Paolo
>>>
>>>
>>> Il 12/11/2014 17:18, Kieron Taylor ha scritto:
>>>> Hi Paolo,
>>>>
>>>> Apologies for the inconsistent service. We are investigating the
>>>> cause of three outages today. There may be intermittent further
>>>> request failures while we restart individual components.
>>>>
>>>> Are you doing a great deal of VEP querying?
>>>>
>>>> Regards,
>>>>
>>>> Kieron Taylor
>>>>
>>>> Ensembl Core
>>>> EMBL-EBI
>>>>
>>>> On 12/11/2014 15:49, Paolo Cozzi wrote:
>>>>>
>>>>> Dear all,
>>>>>
>>>>> When doing a query on EnsEMBL rest server, I receive "gateway
>>>>> timeout",
>>>>> for instance
>>>>>
>>>>> $ curl -H "content-type:application/json" -H "accept:application/json"
>>>>> --data '{ "variants" : ["21 26960070 rs116645811 G A . . .", "21
>>>>> 26965148 rs1135638 G A . . ." ] }'
>>>>> http://rest.ensembl.org/vep/sheep/region
>>>>> <html><body><h1>504 Gateway Time-out</h1>
>>>>> The server didn't respond in time.
>>>>> </body></html>
>>>>>
>>>>> Thank you for your attention,
>>>>>
>>>>> regards,
>>>>>
>>>>> Paolo
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Dev mailing list    Dev at ensembl.org
>>>>> Posting guidelines and subscribe/unsubscribe info:
>>>>> http://lists.ensembl.org/mailman/listinfo/dev
>>>>> Ensembl Blog: http://www.ensembl.info/
>>>>
>>>> _______________________________________________
>>>> Dev mailing list    Dev at ensembl.org
>>>> Posting guidelines and subscribe/unsubscribe info:
>>>> http://lists.ensembl.org/mailman/listinfo/dev
>>>> Ensembl Blog: http://www.ensembl.info/
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list    Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/
>>
>>
>> _______________________________________________
>> Dev mailing list    Dev at ensembl.org
>> Posting guidelines and subscribe/unsubscribe info:
>> http://lists.ensembl.org/mailman/listinfo/dev
>> Ensembl Blog: http://www.ensembl.info/
>
>
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> Posting guidelines and subscribe/unsubscribe info:
> http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/