[ensembl-dev] VEP upper limit on number of variations?

Taylor, Sean Sean.Taylor at seattlechildrens.org
Tue Jun 7 15:00:11 BST 2016


Hello,

I have a library of about 93 million variants that I want to run VEP on. I have downloaded VEP version 84 onto a local linux machine running CentOS Linux release 7.1.1503. The machine has 125G and 40 cores @ 2.3GHz.

My variant library is stored in an impala table in hdfs. To generate my input into VEP, I run a simple impala query as follows:

impala -B -o ~/vep/knownvariants.tsv  --output_delimiter="\\t" -q "select contig, pos, pos+length, concat(ref,'\/',alt), '+' from ingest.central_variant_store where variant_type != 'COMPLEX'"

This writes the variants to a tab delimited file in ensembl format:
10      57676885        57676886        C/G     +
7       34456697        34456698        A/G     +
8       62679252        62679253        G/A     +
7       9184853 9184854 C/A     +
3       29205854        29205855        C/T     +
10      42815272        42815273        C/T     +
8       117963405       117963406       C/T     +
12      53054550        53054551        C/T     +
6       105515195       105515196       T/C     +
20      665650  665651  G/C     +


The entire library is about 93 million. I filtered out the complex variants (i.e. CNVs) as my experience showed that these just threw warnings in VEP and remained unannotated. My idea was to break this output into smaller chunks for processing, ideally around 25M variants each.

I then feed each chunk into VEP using the following arguments:
perlbrew switch 5.16.3

perlbrew exec perl ./variant_effect_predictor.pl --cache --offline --everything --json --buffer_size 25000000 --force_overwrite --verbose -i knownvariants.tsv -o knownvariants.json 2>&1 | tee knownvariants.log

VEP will crunch on this for a while. It will usually clear the following steps:
-load variants into buffer
-check for existing variations
-read transcript data from cache
-read regulatory data from cache
-begin analyzing chromosomes

Somewhere in the last step, VEP will crash with the following error message:

Command [perl ./variant_effect_predictor.pl --cache --offline --everything --json --buffer_size 25000000 --force_overwrite --verbose -i knownvariants.tsv -o knownvariants.json] terminated with exit code 0 ($? = 9) under the following perl environment:
Command terminated with non-zero status.
Current perl:
  Name: perl-5.16.3
 Path: /opt/perl5/perls/perl-5.16.3/bin/perl
  Config: -de -Dprefix=/opt/perl5/perls/perl-5.16.3 -Aeval:scriptdir=/opt/perl5/perls/perl-5.16.3/bin
  Compiled at: Apr 15 2016 06:06:22

perlbrew:
  version: 0.75
  ENV:
    PERLBREW_ROOT: /opt/perl5
    PERLBREW_HOME:
    PERLBREW_PATH: /opt/perl5/bin:/opt/perl5/perls/perl-5.16.3/bin
    PERLBREW_MANPATH: /opt/perl5/perls/perl-5.16.3/man


This exit code is not very specific so it is hard to know what is going on. Is it running out of memory? I lean away from that because it seems that I have seen specific error messages related to memory usage when that happens. I have tried inputting smaller numbers of variants such as 12.5M, 10M, 5M, 2.5M, 1M, 100K, and 10K. So far, I can generally execute just fine on up to 2.5M variants. Anything bigger and I get this same error message. I also got this once on a 2.5M run, only this time the program crashed after analysis of all the chromosomes. It was actively writing to the json output when it died.

This leads me to ask if there is a known upper limit to how many variants one can practically push through at a time? Or perhaps a timeout limit? Processing 93M in 2.5M chunks is a bit tedious. Any thoughts on how to improve or optimize this would be appreciated. I have attached the log files from several runs for reference.

Thanks,
Sean Taylor
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20160607/3dacf7b6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: knownvariants12Msimple.log
Type: application/octet-stream
Size: 268080 bytes
Desc: knownvariants12Msimple.log
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20160607/3dacf7b6/attachment.obj>


More information about the Dev mailing list