[ensembl-dev] VEP scaling / MySQL load balancing, anyone?

[Jan Vogel]

Hello Ensembl, 

I have a question about mysql load balancing and the scaling of the variant effect predictor script - I'm running an in-house mirror of the ensembl mysql server and I'm using the variant effect predictor VEP script.

Currently, the mySQL server gets pretty high load when I'm running larger number of jobs in parallel. Experiments with tcmalloc improved the performance slightly; sold-state disks did not improve as much as I hoped; 

I currently see 2 main options to improve the performance/scalability:

A) The nicest setup is a dynamic scaling of the mysql servers;  (maybe some virtual machines) and some monitoring software, which automatically brings more servers up if needed; of course all servers all listen to the same virtual IP and port; downside is that this needs quite a bit of configuration etc; 

B) The simplest solution I think is to change the code in the ENSEMBL  REGISTRY so I can add more servers for the same cores and have the code randomly pick a server; 

Is EnsEMBL using any database load balancing / automatic scaling for it's MySQL databases for read-only data other than the mirrors ? Are there any suggestions on what solutions to focus on or have you tried anything which did not work ? 

Are other people out there running into the same problems ? (and have solutions ready?) 

Cheers, 

  Jan Vogel 



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20120731/e777e51f/attachment.html>