[ensembl-dev] VEP scaling / MySQL load balancing, anyone?

[Will McLaren]

Hi Jan,

I assume you've tried using the cache?

http://www.ensembl.org/info/docs/variation/vep/vep_script.html#db

I'm no database expert so I can't offer any advice on that aspect of your query.

Cheers

Will McLaren
Ensembl Variation

On 1 August 2012 05:46, Jan Vogel <jan.vogel at gmail.com> wrote:
>
> Hello Ensembl,
>
> I have a question about mysql load balancing and the scaling of the variant
> effect predictor script - I'm running an in-house mirror of the ensembl
> mysql server and I'm using the variant effect predictor VEP script.
>
> Currently, the mySQL server gets pretty high load when I'm running larger
> number of jobs in parallel. Experiments with tcmalloc improved the
> performance slightly; sold-state disks did not improve as much as I hoped;
>
> I currently see 2 main options to improve the performance/scalability:
>
> A) The nicest setup is a dynamic scaling of the mysql servers;  (maybe some
> virtual machines) and some monitoring software, which automatically brings
> more servers up if needed; of course all servers all listen to the same
> virtual IP and port; downside is that this needs quite a bit of
> configuration etc;
>
> B) The simplest solution I think is to change the code in the ENSEMBL
> REGISTRY so I can add more servers for the same cores and have the code
> randomly pick a server;
>
> Is EnsEMBL using any database load balancing / automatic scaling for it's
> MySQL databases for read-only data other than the mirrors ? Are there any
> suggestions on what solutions to focus on or have you tried anything which
> did not work ?
>
> Are other people out there running into the same problems ? (and have
> solutions ready?)
>
> Cheers,
>
>   Jan Vogel
>
>
>
>
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> List admin (including subscribe/unsubscribe):
> http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/
>