[ensembl-dev] VEP scaling / MySQL load balancing, anyone?

[Jan Vogel]

Hey Will, 

yes - we're using the cache….. 

Jan 

On Aug 1, 2012, at 1:34 AM, Will McLaren wrote:

> Hi Jan,
> 
> I assume you've tried using the cache?
> 
> http://www.ensembl.org/info/docs/variation/vep/vep_script.html#db
> 
> I'm no database expert so I can't offer any advice on that aspect of your query.
> 
> Cheers
> 
> Will McLaren
> Ensembl Variation
> 
> On 1 August 2012 05:46, Jan Vogel <jan.vogel at gmail.com> wrote:
>> 
>> Hello Ensembl,
>> 
>> I have a question about mysql load balancing and the scaling of the variant
>> effect predictor script - I'm running an in-house mirror of the ensembl
>> mysql server and I'm using the variant effect predictor VEP script.
>> 
>> Currently, the mySQL server gets pretty high load when I'm running larger
>> number of jobs in parallel. Experiments with tcmalloc improved the
>> performance slightly; sold-state disks did not improve as much as I hoped;
>> 
>> I currently see 2 main options to improve the performance/scalability:
>> 
>> A) The nicest setup is a dynamic scaling of the mysql servers;  (maybe some
>> virtual machines) and some monitoring software, which automatically brings
>> more servers up if needed; of course all servers all listen to the same
>> virtual IP and port; downside is that this needs quite a bit of
>> configuration etc;
>> 
>> B) The simplest solution I think is to change the code in the ENSEMBL
>> REGISTRY so I can add more servers for the same cores and have the code
>> randomly pick a server;
>> 
>> Is EnsEMBL using any database load balancing / automatic scaling for it's
>> MySQL databases for read-only data other than the mirrors ? Are there any
>> suggestions on what solutions to focus on or have you tried anything which
>> did not work ?
>> 
>> Are other people out there running into the same problems ? (and have
>> solutions ready?)
>> 
>> Cheers,
>> 
>>  Jan Vogel
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Dev mailing list    Dev at ensembl.org
>> List admin (including subscribe/unsubscribe):
>> http://lists.ensembl.org/mailman/listinfo/dev
>> Ensembl Blog: http://www.ensembl.info/
>> 
> 
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> List admin (including subscribe/unsubscribe): http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/