[ensembl-dev] VEP error: Forked process failed.
Will McLaren
wm2 at ebi.ac.uk
Wed May 15 12:35:16 BST 2013
This is now fixed on branch and head too.
Will
On 14 May 2013 17:30, Duarte Molha <duartemolha at gmail.com> wrote:
> Sorry Will... today you have loads of people bugging you :-)
>
> I just updated to the latest VEP you pointed me to ... and now the fail in
> on line 1307
>
> Use of uninitialized value $line in join or string at
> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1307.
>
> Cheers
>
> Duarte
>
>
> =========================
> Duarte Miguel Paulo Molha
> http://about.me/duarte
> =========================
>
>
> On Tue, May 14, 2013 at 4:26 PM, Will McLaren <wm2 at ebi.ac.uk> wrote:
>
>> Hi Duarte,
>>
>> You are correct, it was the --individual flag causing the problem.
>>
>> I have committed a fix to the 71 branch (does not include forking fixes)
>> and to head (does include forking fixes, download from
>> http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils/VEP.pm?revision=1.94&root=ensembl
>> ).
>>
>> Regards
>>
>> Will
>>
>>
>> On 14 May 2013 15:17, Duarte Molha <duartemolha at gmail.com> wrote:
>>
>>> However I believe it must be something related to the --individual
>>> [all|ind list] flag
>>> And does not seem to be related to the --forks argument since it also
>>> thoughs that error when running single threaded.
>>>
>>> However in single thread mode it simply displays the error and keeps
>>> going and with multiple forks the process dies entirely.
>>>
>>> My vcf contains 3 samples genotypes and I have included the --individual
>>> all option.
>>>
>>> Best regards
>>> Duarte
>>>
>>>
>>> =========================
>>> Duarte Miguel Paulo Molha
>>> http://about.me/duarte
>>> =========================
>>>
>>>
>>> On Tue, May 14, 2013 at 2:00 PM, Duarte Molha <duartemolha at gmail.com>wrote:
>>>
>>>> Hi Will
>>>>
>>>> Unfortunately the script does not report in which line it fails and I
>>>> cannot provide you with the entire file since it is private data.
>>>> Is there a way of reporting the line >
>>>>
>>>> Thanks
>>>>
>>>> Duarte
>>>>
>>>>
>>>> =========================
>>>> Duarte Miguel Paulo Molha
>>>> http://about.me/duarte
>>>> =========================
>>>>
>>>>
>>>> On Tue, May 14, 2013 at 1:05 PM, Will McLaren <wm2 at ebi.ac.uk> wrote:
>>>>
>>>>> Hi Duarte,
>>>>>
>>>>> Do you have some input that causes this error?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Will
>>>>>
>>>>>
>>>>> On 14 May 2013 12:57, Duarte Molha <duartemolha at gmail.com> wrote:
>>>>>
>>>>>> Another bug using the updated version... now using the
>>>>>> --check_alleles and --check_existing options the script dies at line 4759
>>>>>>
>>>>>> Use of uninitialized value in string ne at
>>>>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 4759.
>>>>>>
>>>>>>
>>>>>> Best regards
>>>>>>
>>>>>> Duarte
>>>>>>
>>>>>> =========================
>>>>>> Duarte Miguel Paulo Molha
>>>>>> http://about.me/duarte
>>>>>> =========================
>>>>>>
>>>>>>
>>>>>> On Tue, May 14, 2013 at 11:22 AM, Will McLaren <wm2 at ebi.ac.uk> wrote:
>>>>>>
>>>>>>> Thanks - try
>>>>>>> http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils/VEP.pm?revision=1.93&root=ensembl
>>>>>>>
>>>>>>> Will
>>>>>>>
>>>>>>>
>>>>>>> On 14 May 2013 11:09, Duarte Molha <duartemolha at gmail.com> wrote:
>>>>>>>
>>>>>>>> It seems the problems are still there :
>>>>>>>>
>>>>>>>> Here is my output:
>>>>>>>>
>>>>>>>> perl variant_effect_predictor.pl --config vep_human.ini -i
>>>>>>>> INPUT.vcf --fork 16
>>>>>>>>
>>>>>>>> 2013-05-14 10:33:38 - Read configuration from vep_human.ini
>>>>>>>> #----------------------------------#
>>>>>>>> # ENSEMBL VARIANT EFFECT PREDICTOR #
>>>>>>>> #----------------------------------#
>>>>>>>>
>>>>>>>> version 71
>>>>>>>>
>>>>>>>> By Will McLaren (wm2 at ebi.ac.uk)
>>>>>>>>
>>>>>>>> Configuration options:
>>>>>>>>
>>>>>>>> ###
>>>>>>>> allow_non_variant 1
>>>>>>>> buffer_size 500000
>>>>>>>> cache 1
>>>>>>>> canonical 1
>>>>>>>> ccds 1
>>>>>>>> check_alleles 1
>>>>>>>> check_existing 1
>>>>>>>> config vep_human.ini
>>>>>>>> core_type core
>>>>>>>> custom
>>>>>>>> /ReferenceData/vep_additional_annotations/Somatic_variation_phenotypes.bed.gz,Somatic,bed,exact
>>>>>>>>
>>>>>>>>
>>>>>>>> /ReferenceData/vep_additional_annotations/dbsnp135_ensembl_variation_phenotype.bed.gz,dbsnp135,bed,exact
>>>>>>>> dir /ReferenceData/vep_cache/
>>>>>>>> domains 1
>>>>>>>> force_overwrite 1
>>>>>>>> fork 16
>>>>>>>> gmaf 1
>>>>>>>> hgnc 1
>>>>>>>> host ensembldb.ensembl.org
>>>>>>>> individual all
>>>>>>>> input_file INPUT.vcf
>>>>>>>> numbers 1
>>>>>>>> plugin Blosum62
>>>>>>>> Condel,/ReferenceData/vep_cache/Plugins/config/Condel/config,b Carol
>>>>>>>> polyphen b
>>>>>>>> port 5306
>>>>>>>> protein 1
>>>>>>>> regulatory 1
>>>>>>>> sift b
>>>>>>>> species homo_sapiens
>>>>>>>> stats HASH(0x35a8000)
>>>>>>>> terms SO
>>>>>>>> toplevel_dir /ReferenceData/vep_cache/
>>>>>>>> verbose 1
>>>>>>>> xref_refseq 1
>>>>>>>>
>>>>>>>> --------------------
>>>>>>>>
>>>>>>>> Will only load v71 databases
>>>>>>>> Species 'homo_sapiens' loaded from database
>>>>>>>> 'homo_sapiens_core_71_37'
>>>>>>>> Species 'homo_sapiens' loaded from database
>>>>>>>> 'homo_sapiens_cdna_71_37'
>>>>>>>> Species 'homo_sapiens' loaded from database
>>>>>>>> 'homo_sapiens_vega_71_37'
>>>>>>>> Species 'homo_sapiens' loaded from database
>>>>>>>> 'homo_sapiens_otherfeatures_71_37'
>>>>>>>> Species 'homo_sapiens' loaded from database
>>>>>>>> 'homo_sapiens_rnaseq_71_37'
>>>>>>>> homo_sapiens_variation_71_37 loaded
>>>>>>>> homo_sapiens_funcgen_71_37 loaded
>>>>>>>> Bio::EnsEMBL::Compara::DBSQL::DBAdaptor not found so the following
>>>>>>>> compara databases will be ignored: ensembl_compara_71
>>>>>>>> ensembl_ancestral_71 loaded
>>>>>>>> ensembl_ontology_71 loaded
>>>>>>>> 2013-05-14 10:33:39 - Connected to core version 71 database and
>>>>>>>> variation version 71 database
>>>>>>>> 2013-05-14 10:33:39 - Read existing cache info
>>>>>>>> 2013-05-14 10:33:39 - Loaded plugin: Blosum62
>>>>>>>> 2013-05-14 10:33:39 - Loaded plugin: Condel
>>>>>>>> 2013-05-14 10:33:39 - Loaded plugin: Carol
>>>>>>>> 2013-05-14 10:33:40 - Starting...
>>>>>>>> 2013-05-14 10:33:40 - Detected format of input file as vcf
>>>>>>>> 2013-05-14 10:33:46 - Read 195789 variants into buffer
>>>>>>>> 2013-05-14 10:33:46 - Skipping 67552 non-variant loci
>>>>>>>> 2013-05-14 10:33:46 - Reading transcript data from cache and/or
>>>>>>>> database
>>>>>>>> [======================================================================================================]
>>>>>>>> [ 100% ]
>>>>>>>> 2013-05-14 10:40:19 - Retrieved 189344 transcripts (0 mem, 202901
>>>>>>>> cached, 0 DB, 13557 duplicates)
>>>>>>>> 2013-05-14 10:40:19 - Reading regulatory data from cache and/or
>>>>>>>> database
>>>>>>>> [======================================================================================================]
>>>>>>>> [ 100% ]
>>>>>>>> 2013-05-14 10:50:09 - Retrieved 872092 regulatory features (0 mem,
>>>>>>>> 872351 cached, 0 DB, 259 duplicates)
>>>>>>>> 2013-05-14 10:50:12 - Calculating consequences
>>>>>>>> Use of uninitialized value $_ in pattern match (m//) at
>>>>>>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1022.
>>>>>>>> Use of uninitialized value $_ in pattern match (m//) at
>>>>>>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1030.
>>>>>>>> Use of uninitialized value $_ in pattern match (m//) at
>>>>>>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1037.
>>>>>>>> Use of uninitialized value $_ in pattern match (m//) at
>>>>>>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1089.
>>>>>>>> Use of uninitialized value $_ in concatenation (.) or string at
>>>>>>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1097.
>>>>>>>>
>>>>>>>> ERROR: Forked process failed
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> =========================
>>>>>>>> Duarte Miguel Paulo Molha
>>>>>>>> http://about.me/duarte
>>>>>>>> =========================
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 14, 2013 at 10:42 AM, Duarte Molha <
>>>>>>>> duartemolha at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Running a annotation using 16 forks... lets see how it handles :)
>>>>>>>>> I'll report back any issues.
>>>>>>>>>
>>>>>>>>> Thanks for the update
>>>>>>>>>
>>>>>>>>> Duarte
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> =========================
>>>>>>>>> Duarte Miguel Paulo Molha
>>>>>>>>> http://about.me/duarte
>>>>>>>>> =========================
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 14, 2013 at 10:16 AM, Will McLaren <wm2 at ebi.ac.uk>wrote:
>>>>>>>>>
>>>>>>>>>> Stuart, Guillermo, Duarte,
>>>>>>>>>>
>>>>>>>>>> I'm currently working on some code as I stated above to improve
>>>>>>>>>> stability and performance under forking.
>>>>>>>>>>
>>>>>>>>>> I've committed some code to the HEAD of our CVS tree which should
>>>>>>>>>> help the problems you are encountering. You'd all be welcome to test this
>>>>>>>>>> out, with the obvious proviso that this is development code and may contain
>>>>>>>>>> bugs!
>>>>>>>>>>
>>>>>>>>>> To use this, you should download the copy of VEP.pm from:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils/VEP.pm?revision=1.92&root=ensembl
>>>>>>>>>>
>>>>>>>>>> and replace the VEP.pm under
>>>>>>>>>> ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils (or just
>>>>>>>>>> Bio/EnsEMBL/Variation/Utils if you use INSTALL.pl) with this one.
>>>>>>>>>>
>>>>>>>>>> This code will appear in production in the next proper release of
>>>>>>>>>> Ensembl.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>>
>>>>>>>>>> Will
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 14 May 2013 09:55, Stuart Meacham <sm766 at cam.ac.uk> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I certainly don't want to hijack this thread but it seemed daft
>>>>>>>>>>> to start another. I am also getting forking errors. I don't use any custom
>>>>>>>>>>> plugins and am using a validated VCF as input (with about 600,000
>>>>>>>>>>> variants). Trying to fork more than 4 threads is unstable even on my
>>>>>>>>>>> machine which has 64 cores and half a TB of RAM.
>>>>>>>>>>>
>>>>>>>>>>> I haven't found anything reproducible, however if I do I'll
>>>>>>>>>>> report back to the list.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> Stuart
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 14/05/2013 09:42, Will McLaren wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> Your aa_grantham_distance plugin is somewhat inefficient - it
>>>>>>>>>>> retrieves the peptide alleles from the HGVS annotation, which itself
>>>>>>>>>>> requires some database fetching and processing to produce. This is why it
>>>>>>>>>>> is slow.
>>>>>>>>>>>
>>>>>>>>>>> You can get the peptides from the transcript variation object:
>>>>>>>>>>>
>>>>>>>>>>> my @peps = split "/",
>>>>>>>>>>> $tva->transcript_variation->pep_allele_string();
>>>>>>>>>>>
>>>>>>>>>>> This will give you single-letter AA codes, but you could
>>>>>>>>>>> either modify your hash or use BioPerl to convert:
>>>>>>>>>>>
>>>>>>>>>>> $seqobj = Bio::PrimarySeq->new ( -seq => $single_letter_aa);
>>>>>>>>>>> $three_letter_aa = Bio::SeqUtils->seq3($seqobj);
>>>>>>>>>>>
>>>>>>>>>>> You should also declare your distances hash in the new() sub
>>>>>>>>>>> and store it on $self; this will also marginally speed up your plugin.
>>>>>>>>>>>
>>>>>>>>>>> Regarding the forking issues, we are working on improving
>>>>>>>>>>> stability under forking.
>>>>>>>>>>>
>>>>>>>>>>> Thanks for your patience
>>>>>>>>>>>
>>>>>>>>>>> Will
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 14 May 2013 07:37, Guillermo Marco Puche <
>>>>>>>>>>> guillermo.marco at sistemasgenomicos.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not really sure which one of those plugins is causing the
>>>>>>>>>>>> fork error. I cannot recreate it now running each one of them separately.
>>>>>>>>>>>>
>>>>>>>>>>>> Here are both:
>>>>>>>>>>>>
>>>>>>>>>>>> https://github.com/guillermomarco/vep_plugins_71
>>>>>>>>>>>>
>>>>>>>>>>>> They also slow the calculating consequences process a lot.
>>>>>>>>>>>> aa_grantham_distance.pm is just a hardcoded plugin from one of
>>>>>>>>>>>> the biologists in my work. It was just a pure copy paste and adaptation to
>>>>>>>>>>>> make it work as a VEP plugin. Maybe the problem is in the matrix definition
>>>>>>>>>>>> every time the sub routine is called. I'm not running out of memory nor
>>>>>>>>>>>> CPU. I'm currently using it with 2 threads and buffersize of 500 for a 5000
>>>>>>>>>>>> variant vcf file.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm my honest opinion, I think one (or even both) of those
>>>>>>>>>>>> plugins are slowing so much the calculating process that sometimes the fork
>>>>>>>>>>>> just dies. Like when you have a timeout during to heavy network traffic. So
>>>>>>>>>>>> when you use them together with lot of other plugins like Condel,
>>>>>>>>>>>> Consequence, etc.. they may be causing the process to handle and die.
>>>>>>>>>>>>
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Guillermo.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 05/13/2013 03:55 PM, Duarte Molha wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I also get this error... it is so prevalent and so difficult to
>>>>>>>>>>>> pinpoint what is causing it that I have given up on forking my annotation
>>>>>>>>>>>> process.
>>>>>>>>>>>>
>>>>>>>>>>>> I do think it is related to the number of forks. It seems to
>>>>>>>>>>>> crash less often if you use a low number of forks... anything above 5
>>>>>>>>>>>> will undoubtedly crash the script at least in my experience.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>>
>>>>>>>>>>>> Duarte
>>>>>>>>>>>>
>>>>>>>>>>>> =========================
>>>>>>>>>>>> Duarte Miguel Paulo Molha
>>>>>>>>>>>> http://about.me/duarte
>>>>>>>>>>>> =========================
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, May 13, 2013 at 2:50 PM, Will McLaren <wm2 at ebi.ac.uk>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Guillermo,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Test each plugin individually until you find the one that
>>>>>>>>>>>>> causes the error. It is highly unlikely that a particular combination of
>>>>>>>>>>>>> plugins is causing the crash.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Check that there are no "print" (to STDOUT or STDERR)
>>>>>>>>>>>>> statements in your plugin - forking assumes that code remains silent
>>>>>>>>>>>>> otherwise it will throw errors like this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, check what, if anything, is cached between runs of
>>>>>>>>>>>>> your plugin. If you are caching things (for example to avoid re-querying a
>>>>>>>>>>>>> database), you may need to write storable hooks to ensure the data is
>>>>>>>>>>>>> getting cached between forks - see
>>>>>>>>>>>>> https://github.com/ensembl-variation/VEP_plugins/blob/master/ProteinSeqs.pmfor an example.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you still have no luck, send me the code and an input
>>>>>>>>>>>>> file that recreates the problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>
>>>>>>>>>>>>> Will
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 13 May 2013 13:18, Guillermo Marco Puche <
>>>>>>>>>>>>> guillermo.marco at sistemasgenomicos.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've started to recently having problems with VEP script
>>>>>>>>>>>>>> while using different plugins (most of them own plugins).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013-05-13 13:59:44 - Connected to core version 71 database and variation version 71 database
>>>>>>>>>>>>>> 2013-05-13 13:59:44 - Loaded plugin: vcf_input
>>>>>>>>>>>>>> 2013-05-13 13:59:44 - Loaded plugin: biobase
>>>>>>>>>>>>>> 2013-05-13 13:59:44 - Loaded plugin: aa_grantham_distance
>>>>>>>>>>>>>> 2013-05-13 13:59:44 - Loaded plugin: flanking_sequence
>>>>>>>>>>>>>> 2013-05-13 13:59:44 - Loaded plugin: Condel
>>>>>>>>>>>>>> 2013-05-13 13:59:44 - Output fields redefined (37 defined)
>>>>>>>>>>>>>> 2013-05-13 13:59:44 - Starting...
>>>>>>>>>>>>>> 2013-05-13 13:59:45 - Read 3888 variants into buffer
>>>>>>>>>>>>>> 2013-05-13 13:59:54 - Reading transcript data from cache and/or database
>>>>>>>>>>>>>> [===============================================] [ 100% ]
>>>>>>>>>>>>>> 2013-05-13 14:02:38 - Retrieved 6463 transcripts (0 mem, 0 cached, 13743 DB, 7280 duplicates)
>>>>>>>>>>>>>> 2013-05-13 14:02:38 - Calculating consequences
>>>>>>>>>>>>>> [===================================> ] [ 78% ]
>>>>>>>>>>>>>> ERROR: Forked process failed
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not getting any other error message. So I cannot debug
>>>>>>>>>>>>>> properly. I thought my plugins were OK but it's seems they don't. I think
>>>>>>>>>>>>>> the problem occurs when I use "aa_grantham_distance plugin" together with
>>>>>>>>>>>>>> "flanking_sequence". I've no idea what could be causing this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm running VEP on verbose mode but I can't get any usefull
>>>>>>>>>>>>>> information. How could I debug that?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>> Guillermo.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Dev mailing list Dev at ensembl.org
>>>>>>>>>>>>>> Posting guidelines and subscribe/unsubscribe info:
>>>>>>>>>>>>>> http://lists.ensembl.org/mailman/listinfo/dev
>>>>>>>>>>>>>> Ensembl Blog: http://www.ensembl.info/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Dev mailing list Dev at ensembl.org
>>>>>>>>>>>> Posting guidelines and subscribe/unsubscribe info:
>>>>>>>>>>>> http://lists.ensembl.org/mailman/listinfo/dev
>>>>>>>>>>>> Ensembl Blog: http://www.ensembl.info/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Dev mailing list Dev at ensembl.org
>>>>>>>>>>> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev
>>>>>>>>>>> Ensembl Blog: http://www.ensembl.info/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Dev mailing list Dev at ensembl.org
>>>>>>>>>>> Posting guidelines and subscribe/unsubscribe info:
>>>>>>>>>>> http://lists.ensembl.org/mailman/listinfo/dev
>>>>>>>>>>> Ensembl Blog: http://www.ensembl.info/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Dev mailing list Dev at ensembl.org
>>>>>>>>>> Posting guidelines and subscribe/unsubscribe info:
>>>>>>>>>> http://lists.ensembl.org/mailman/listinfo/dev
>>>>>>>>>> Ensembl Blog: http://www.ensembl.info/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Dev mailing list Dev at ensembl.org
>>>>>>>> Posting guidelines and subscribe/unsubscribe info:
>>>>>>>> http://lists.ensembl.org/mailman/listinfo/dev
>>>>>>>> Ensembl Blog: http://www.ensembl.info/
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dev mailing list Dev at ensembl.org
>>>>>>> Posting guidelines and subscribe/unsubscribe info:
>>>>>>> http://lists.ensembl.org/mailman/listinfo/dev
>>>>>>> Ensembl Blog: http://www.ensembl.info/
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Dev mailing list Dev at ensembl.org
>>>>>> Posting guidelines and subscribe/unsubscribe info:
>>>>>> http://lists.ensembl.org/mailman/listinfo/dev
>>>>>> Ensembl Blog: http://www.ensembl.info/
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Dev mailing list Dev at ensembl.org
>>>>> Posting guidelines and subscribe/unsubscribe info:
>>>>> http://lists.ensembl.org/mailman/listinfo/dev
>>>>> Ensembl Blog: http://www.ensembl.info/
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/
>>>
>>>
>>
>> _______________________________________________
>> Dev mailing list Dev at ensembl.org
>> Posting guidelines and subscribe/unsubscribe info:
>> http://lists.ensembl.org/mailman/listinfo/dev
>> Ensembl Blog: http://www.ensembl.info/
>>
>>
>
> _______________________________________________
> Dev mailing list Dev at ensembl.org
> Posting guidelines and subscribe/unsubscribe info:
> http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20130515/5e82512b/attachment.html>
More information about the Dev
mailing list