[ensembl-dev] Dev Digest, Vol 8, Issue 3

[Joseph Steinberger]

How to I get the number of maximum connections allowed? Is there a REST API get URL for this information?

________________________________
From: Dev <dev-bounces at ensembl.org> on behalf of dev-request at ensembl.org <dev-request at ensembl.org>
Sent: Wednesday, 9 October 2019 14:00
To: dev at ensembl.org <dev at ensembl.org>
Subject: Dev Digest, Vol 8, Issue 3

Send Dev mailing list submissions to
        dev at ensembl.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://mail.ensembl.org/mailman/listinfo/dev_ensembl.org
or, via email, send a message with subject or body 'help' to
        dev-request at ensembl.org

You can reach the person managing the list at
        dev-owner at ensembl.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Dev digest..."


Today's Topics:

   1. personal Amazon hosted Ensembl database (Joseph Steinberger)
   2. Re: personal Amazon hosted Ensembl database (Thibaut Hourlier)


----------------------------------------------------------------------

Message: 1
Date: Tue, 8 Oct 2019 13:53:58 +0000
From: Joseph Steinberger <joseph.steinberger at weizmann.ac.il>
To: "dev at ensembl.org" <dev at ensembl.org>
Subject: [ensembl-dev] personal Amazon hosted Ensembl database
Message-ID:
        <VI1P191MB0560016827498519B1EE442DB59A0 at VI1P191MB0560.EURP191.PROD.OUTLOOK.COM>

Content-Type: text/plain; charset="us-ascii"

Dear Development community,

Is there a precedent for having a personal Amazon AWS clone of the entire Ensembl database?
I am making many short requests and getting timed out -
Specifically, I want the genetree id for each of a few million genes, and I cannot do it in parallel or I get a "too many requests" error.

Thank you for your time.

Sincerely,
Joseph Steinberger

laboratory of Naama Barkai, Weizmann Institute of Science
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20191008/fc3eccc7/attachment-0001.html>

------------------------------

Message: 2
Date: Tue, 8 Oct 2019 15:31:51 +0100
From: Thibaut Hourlier <thibaut at ebi.ac.uk>
To: Ensembl developers list <dev at ensembl.org>
Subject: Re: [ensembl-dev] personal Amazon hosted Ensembl database
Message-ID: <C58403FD-70EC-4E84-9B6C-927AE4777ABC at ebi.ac.uk>
Content-Type: text/plain; charset="us-ascii"

Hi Joseph,
One thing you can try is to know how many connections are allowed to the server, then you should have less jobs running in parallel than the maximum number of connections allowed.
As you have many short request you should also make sure that you do not disconnect each time. Each time you disconnect, the server will use a different port, so you may also quickly run out of port to connect to. If you use the Perl API, you can set -disconnect_when_inactive => 0 when creating the connection to the database.

Hope this helps
Thibaut

> On 8 Oct 2019, at 14:53, Joseph Steinberger <joseph.steinberger at weizmann.ac.il> wrote:
>
> Dear Development community,
>
> Is there a precedent for having a personal Amazon AWS clone of the entire Ensembl database?
> I am making many short requests and getting timed out -
> Specifically, I want the genetree id for each of a few million genes, and I cannot do it in parallel or I get a "too many requests" error.
>
> Thank you for your time.
>
> Sincerely,
> Joseph Steinberger
>
> laboratory of Naama Barkai, Weizmann Institute of Science
> _______________________________________________
> Dev mailing list    Dev at ensembl.org <mailto:Dev at ensembl.org>
> Posting guidelines and subscribe/unsubscribe info: https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org <https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org>
> Ensembl Blog: http://www.ensembl.info/ <http://www.ensembl.info/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20191008/c47a9158/attachment-0001.html>

------------------------------

Subject: Digest Footer

_______________________________________________
Dev mailing list    Dev at ensembl.org
Posting guidelines and subscribe/unsubscribe info: https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org
Ensembl Blog: http://www.ensembl.info/


------------------------------

End of Dev Digest, Vol 8, Issue 3
*********************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20191104/fbcbd2d5/attachment.html>