[ensembl-dev] VEP error: Forked process failed.
Will McLaren
wm2 at ebi.ac.uk
Fri May 17 16:51:11 BST 2013
Fixed
On 17 May 2013 16:41, Duarte Molha <duartemolha at gmail.com> wrote:
> Something is definitely something wrong with the later changes ... both
> using the branch version and the head version of VEP throughs thousands of
> errors of the type
>
> Use of uninitialized value $_ in scalar chomp at
> /services/.array/OGTIP/duartem/workspace/VEP_dev/Bio/EnsEMBL/Variation/Utils/VEP.pm
> line 1081, <$child> line XXXXXXX.
>
> Best regards
>
> Duarte
>
>
>
> =========================
> Duarte Miguel Paulo Molha
> http://about.me/duarte
> =========================
>
>
> On Fri, May 17, 2013 at 4:13 PM, Will McLaren <wm2 at ebi.ac.uk> wrote:
>
>> I pushed the fork changes to branch as well today as I felt they were
>> stable enough.
>>
>> So, you can use branch and it should be fine.
>>
>> There will be more coming but they can wait for the next main release.
>>
>> Will
>>
>>
>> On 17 May 2013 16:06, Duarte Molha <Duarte.Molha at ogt.com> wrote:
>>
>>> I believe you have to use the version in HEAD from the CSV repository.**
>>> **
>>>
>>> ** **
>>>
>>> Please correct me if I am wrong Will.****
>>>
>>> ** **
>>>
>>> Best regards****
>>>
>>> ** **
>>>
>>> Duarte****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> *From:* dev-bounces at ensembl.org [mailto:dev-bounces at ensembl.org] *On
>>> Behalf Of *Guillermo Marco Puche
>>> *Sent:* 16 May 2013 12:30
>>> *To:* Ensembl developers list
>>> *Subject:* Re: [ensembl-dev] VEP error: Forked process failed.****
>>>
>>> ** **
>>>
>>> Hello,
>>>
>>> I got lost in this topic.
>>> So if I understand well.. to fix the VEP forking issue I must can
>>> re-download the VEP script from ensembl website?
>>>
>>> Thank you !
>>>
>>> Best regards,
>>> Guillermo.
>>>
>>> On 05/15/2013 01:35 PM, Will McLaren wrote:****
>>>
>>> This is now fixed on branch and head too. ****
>>>
>>> ** **
>>>
>>> Will****
>>>
>>> ** **
>>>
>>> On 14 May 2013 17:30, Duarte Molha <duartemolha at gmail.com> wrote:****
>>>
>>> Sorry Will... today you have loads of people bugging you :-) ****
>>>
>>> ** **
>>>
>>> I just updated to the latest VEP you pointed me to ... and now the fail
>>> in on line 1307****
>>>
>>> ** **
>>>
>>> Use of uninitialized value $line in join or string at
>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1307.***
>>> *
>>>
>>> ** **
>>>
>>> Cheers****
>>>
>>>
>>> Duarte****
>>>
>>> ** **
>>>
>>>
>>> ****
>>>
>>> =========================
>>> Duarte Miguel Paulo Molha ****
>>>
>>> http://about.me/duarte
>>> =========================****
>>>
>>> ** **
>>>
>>> On Tue, May 14, 2013 at 4:26 PM, Will McLaren <wm2 at ebi.ac.uk> wrote:****
>>>
>>> Hi Duarte, ****
>>>
>>> ** **
>>>
>>> You are correct, it was the --individual flag causing the problem.****
>>>
>>> ** **
>>>
>>> I have committed a fix to the 71 branch (does not include forking fixes)
>>> and to head (does include forking fixes, download from
>>> http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils/VEP.pm?revision=1.94&root=ensembl
>>> ).****
>>>
>>> ** **
>>>
>>> Regards****
>>>
>>> ** **
>>>
>>> Will****
>>>
>>> ** **
>>>
>>> On 14 May 2013 15:17, Duarte Molha <duartemolha at gmail.com> wrote:****
>>>
>>> However I believe it must be something related to the --individual
>>> [all|ind list] flag ****
>>>
>>> And does not seem to be related to the --forks argument since it also
>>> thoughs that error when running single threaded. ****
>>>
>>> ** **
>>>
>>> However in single thread mode it simply displays the error and keeps
>>> going and with multiple forks the process dies entirely.****
>>>
>>> ****
>>>
>>> My vcf contains 3 samples genotypes and I have included the --individual
>>> all option. ****
>>>
>>> ** **
>>>
>>> Best regards****
>>>
>>> Duarte****
>>>
>>> ** **
>>>
>>>
>>> ****
>>>
>>> =========================
>>> Duarte Miguel Paulo Molha ****
>>>
>>> http://about.me/duarte
>>> =========================****
>>>
>>> ** **
>>>
>>> On Tue, May 14, 2013 at 2:00 PM, Duarte Molha <duartemolha at gmail.com>
>>> wrote:****
>>>
>>> Hi Will ****
>>>
>>> ** **
>>>
>>> Unfortunately the script does not report in which line it fails and I
>>> cannot provide you with the entire file since it is private data.****
>>>
>>> Is there a way of reporting the line >****
>>>
>>> ** **
>>>
>>> Thanks****
>>>
>>>
>>> Duarte****
>>>
>>> ** **
>>>
>>>
>>> ****
>>>
>>> =========================
>>> Duarte Miguel Paulo Molha ****
>>>
>>> http://about.me/duarte
>>> =========================****
>>>
>>> ** **
>>>
>>> On Tue, May 14, 2013 at 1:05 PM, Will McLaren <wm2 at ebi.ac.uk> wrote:****
>>>
>>> Hi Duarte, ****
>>>
>>>
>>> Do you have some input that causes this error?****
>>>
>>> ** **
>>>
>>> Thanks****
>>>
>>> ** **
>>>
>>> Will****
>>>
>>> ** **
>>>
>>> On 14 May 2013 12:57, Duarte Molha <duartemolha at gmail.com> wrote:****
>>>
>>> Another bug using the updated version... now using the --check_alleles
>>> and --check_existing options the script dies at line 4759 ****
>>>
>>> ** **
>>>
>>> Use of uninitialized value in string ne at
>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 4759.***
>>> *
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> Best regards****
>>>
>>> ** **
>>>
>>> Duarte****
>>>
>>>
>>> ****
>>>
>>> =========================
>>> Duarte Miguel Paulo Molha ****
>>>
>>> http://about.me/duarte
>>> =========================****
>>>
>>> ** **
>>>
>>> On Tue, May 14, 2013 at 11:22 AM, Will McLaren <wm2 at ebi.ac.uk> wrote:***
>>> *
>>>
>>> Thanks - try
>>> http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils/VEP.pm?revision=1.93&root=ensembl
>>> ****
>>>
>>> ** **
>>>
>>> Will****
>>>
>>> ** **
>>>
>>> On 14 May 2013 11:09, Duarte Molha <duartemolha at gmail.com> wrote:****
>>>
>>> It seems the problems are still there : ****
>>>
>>> ** **
>>>
>>> Here is my output:****
>>>
>>> ** **
>>>
>>> perl variant_effect_predictor.pl --config vep_human.ini -i INPUT.vcf
>>> --fork 16****
>>>
>>> ** **
>>>
>>> 2013-05-14 10:33:38 - Read configuration from vep_human.ini****
>>>
>>> #----------------------------------#****
>>>
>>> # ENSEMBL VARIANT EFFECT PREDICTOR #****
>>>
>>> #----------------------------------#****
>>>
>>> ** **
>>>
>>> version 71****
>>>
>>> ** **
>>>
>>> By Will McLaren (wm2 at ebi.ac.uk)****
>>>
>>> ** **
>>>
>>> Configuration options:****
>>>
>>> ** **
>>>
>>> ###****
>>>
>>> allow_non_variant 1****
>>>
>>> buffer_size 500000****
>>>
>>> cache 1****
>>>
>>> canonical 1****
>>>
>>> ccds 1****
>>>
>>> check_alleles 1****
>>>
>>> check_existing 1****
>>>
>>> config vep_human.ini****
>>>
>>> core_type core****
>>>
>>> custom
>>> /ReferenceData/vep_additional_annotations/Somatic_variation_phenotypes.bed.gz,Somatic,bed,exact
>>> ****
>>>
>>>
>>> /ReferenceData/vep_additional_annotations/dbsnp135_ensembl_variation_phenotype.bed.gz,dbsnp135,bed,exact
>>> ****
>>>
>>> dir /ReferenceData/vep_cache/****
>>>
>>> domains 1****
>>>
>>> force_overwrite 1****
>>>
>>> fork 16****
>>>
>>> gmaf 1****
>>>
>>> hgnc 1****
>>>
>>> host ensembldb.ensembl.org****
>>>
>>> individual all****
>>>
>>> input_file INPUT.vcf****
>>>
>>> numbers 1****
>>>
>>> plugin Blosum62
>>> Condel,/ReferenceData/vep_cache/Plugins/config/Condel/config,b Carol***
>>> *
>>>
>>> polyphen b****
>>>
>>> port 5306****
>>>
>>> protein 1****
>>>
>>> regulatory 1****
>>>
>>> sift b****
>>>
>>> species homo_sapiens****
>>>
>>> stats HASH(0x35a8000)****
>>>
>>> terms SO****
>>>
>>> toplevel_dir /ReferenceData/vep_cache/****
>>>
>>> verbose 1****
>>>
>>> xref_refseq 1****
>>>
>>> ** **
>>>
>>> --------------------****
>>>
>>> ** **
>>>
>>> Will only load v71 databases****
>>>
>>> Species 'homo_sapiens' loaded from database 'homo_sapiens_core_71_37'***
>>> *
>>>
>>> Species 'homo_sapiens' loaded from database 'homo_sapiens_cdna_71_37'***
>>> *
>>>
>>> Species 'homo_sapiens' loaded from database 'homo_sapiens_vega_71_37'***
>>> *
>>>
>>> Species 'homo_sapiens' loaded from database
>>> 'homo_sapiens_otherfeatures_71_37'****
>>>
>>> Species 'homo_sapiens' loaded from database 'homo_sapiens_rnaseq_71_37'*
>>> ***
>>>
>>> homo_sapiens_variation_71_37 loaded****
>>>
>>> homo_sapiens_funcgen_71_37 loaded****
>>>
>>> Bio::EnsEMBL::Compara::DBSQL::DBAdaptor not found so the following
>>> compara databases will be ignored: ensembl_compara_71****
>>>
>>> ensembl_ancestral_71 loaded****
>>>
>>> ensembl_ontology_71 loaded****
>>>
>>> 2013-05-14 10:33:39 - Connected to core version 71 database and
>>> variation version 71 database****
>>>
>>> 2013-05-14 10:33:39 - Read existing cache info****
>>>
>>> 2013-05-14 10:33:39 - Loaded plugin: Blosum62****
>>>
>>> 2013-05-14 10:33:39 - Loaded plugin: Condel****
>>>
>>> 2013-05-14 10:33:39 - Loaded plugin: Carol****
>>>
>>> 2013-05-14 10:33:40 - Starting...****
>>>
>>> 2013-05-14 10:33:40 - Detected format of input file as vcf****
>>>
>>> 2013-05-14 10:33:46 - Read 195789 variants into buffer****
>>>
>>> 2013-05-14 10:33:46 - Skipping 67552 non-variant loci****
>>>
>>> 2013-05-14 10:33:46 - Reading transcript data from cache and/or database
>>> ****
>>>
>>> [======================================================================================================]
>>> [ 100% ]****
>>>
>>> 2013-05-14 10:40:19 - Retrieved 189344 transcripts (0 mem, 202901
>>> cached, 0 DB, 13557 duplicates)****
>>>
>>> 2013-05-14 10:40:19 - Reading regulatory data from cache and/or database
>>> ****
>>>
>>> [======================================================================================================]
>>> [ 100% ]****
>>>
>>> 2013-05-14 10:50:09 - Retrieved 872092 regulatory features (0 mem,
>>> 872351 cached, 0 DB, 259 duplicates)****
>>>
>>> 2013-05-14 10:50:12 - Calculating consequences****
>>>
>>> Use of uninitialized value $_ in pattern match (m//) at
>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1022.***
>>> *
>>>
>>> Use of uninitialized value $_ in pattern match (m//) at
>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1030.***
>>> *
>>>
>>> Use of uninitialized value $_ in pattern match (m//) at
>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1037.***
>>> *
>>>
>>> Use of uninitialized value $_ in pattern match (m//) at
>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1089.***
>>> *
>>>
>>> Use of uninitialized value $_ in concatenation (.) or string at
>>> /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1097.***
>>> *
>>>
>>> ** **
>>>
>>> ERROR: Forked process failed****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> ** **
>>>
>>>
>>> ****
>>>
>>> =========================
>>> Duarte Miguel Paulo Molha ****
>>>
>>> http://about.me/duarte
>>> =========================****
>>>
>>> ** **
>>>
>>> On Tue, May 14, 2013 at 10:42 AM, Duarte Molha <duartemolha at gmail.com>
>>> wrote:****
>>>
>>> Thanks ****
>>>
>>> ** **
>>>
>>> Running a annotation using 16 forks... lets see how it handles :)****
>>>
>>> I'll report back any issues.****
>>>
>>> ** **
>>>
>>> Thanks for the update****
>>>
>>> ** **
>>>
>>> Duarte****
>>>
>>> ** **
>>>
>>>
>>> ****
>>>
>>> =========================
>>> Duarte Miguel Paulo Molha ****
>>>
>>> http://about.me/duarte
>>> =========================****
>>>
>>> ** **
>>>
>>> On Tue, May 14, 2013 at 10:16 AM, Will McLaren <wm2 at ebi.ac.uk> wrote:***
>>> *
>>>
>>> Stuart, Guillermo, Duarte, ****
>>>
>>> ** **
>>>
>>> I'm currently working on some code as I stated above to improve
>>> stability and performance under forking.****
>>>
>>> ** **
>>>
>>> I've committed some code to the HEAD of our CVS tree which should help
>>> the problems you are encountering. You'd all be welcome to test this out,
>>> with the obvious proviso that this is development code and may contain bugs!
>>> ****
>>>
>>> ** **
>>>
>>> To use this, you should download the copy of VEP.pm from:****
>>>
>>> ** **
>>>
>>>
>>> http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils/VEP.pm?revision=1.92&root=ensembl
>>> ****
>>>
>>> ** **
>>>
>>> and replace the VEP.pm under
>>> ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils (or just
>>> Bio/EnsEMBL/Variation/Utils if you use INSTALL.pl) with this one.****
>>>
>>> ** **
>>>
>>> This code will appear in production in the next proper release of
>>> Ensembl.****
>>>
>>> ** **
>>>
>>> Regards****
>>>
>>> ** **
>>>
>>> Will****
>>>
>>> ** **
>>>
>>> On 14 May 2013 09:55, Stuart Meacham <sm766 at cam.ac.uk> wrote:****
>>>
>>> Hi,
>>>
>>> I certainly don't want to hijack this thread but it seemed daft to start
>>> another. I am also getting forking errors. I don't use any custom plugins
>>> and am using a validated VCF as input (with about 600,000 variants). Trying
>>> to fork more than 4 threads is unstable even on my machine which has 64
>>> cores and half a TB of RAM.
>>>
>>> I haven't found anything reproducible, however if I do I'll report back
>>> to the list.
>>>
>>> Thanks
>>>
>>> Stuart ****
>>>
>>>
>>>
>>> On 14/05/2013 09:42, Will McLaren wrote:****
>>>
>>> Hello, ****
>>>
>>> ** **
>>>
>>> Your aa_grantham_distance plugin is somewhat inefficient - it retrieves
>>> the peptide alleles from the HGVS annotation, which itself requires some
>>> database fetching and processing to produce. This is why it is slow.****
>>>
>>> ** **
>>>
>>> You can get the peptides from the transcript variation object:****
>>>
>>> ** **
>>>
>>> my @peps = split "/", $tva->transcript_variation->pep_allele_string();**
>>> **
>>>
>>> ** **
>>>
>>> This will give you single-letter AA codes, but you could either modify
>>> your hash or use BioPerl to convert:****
>>>
>>> ** **
>>>
>>> $seqobj = Bio::PrimarySeq->new ( -seq => $single_letter_aa);
>>> $three_letter_aa = Bio::SeqUtils->seq3($seqobj); ****
>>>
>>> ** **
>>>
>>> You should also declare your distances hash in the new() sub and store
>>> it on $self; this will also marginally speed up your plugin.****
>>>
>>> ** **
>>>
>>> Regarding the forking issues, we are working on improving stability
>>> under forking.****
>>>
>>> ** **
>>>
>>> Thanks for your patience****
>>>
>>> ** **
>>>
>>> Will****
>>>
>>> ** **
>>>
>>> On 14 May 2013 07:37, Guillermo Marco Puche <
>>> guillermo.marco at sistemasgenomicos.com> wrote:****
>>>
>>> Hello,
>>>
>>> I'm not really sure which one of those plugins is causing the fork
>>> error. I cannot recreate it now running each one of them separately.
>>>
>>> Here are both:
>>>
>>> https://github.com/guillermomarco/vep_plugins_71
>>>
>>> They also slow the calculating consequences process a lot.
>>> aa_grantham_distance.pm is just a hardcoded plugin from one of the
>>> biologists in my work. It was just a pure copy paste and adaptation to make
>>> it work as a VEP plugin. Maybe the problem is in the matrix definition
>>> every time the sub routine is called. I'm not running out of memory nor
>>> CPU. I'm currently using it with 2 threads and buffersize of 500 for a 5000
>>> variant vcf file.
>>>
>>> I'm my honest opinion, I think one (or even both) of those plugins are
>>> slowing so much the calculating process that sometimes the fork just dies.
>>> Like when you have a timeout during to heavy network traffic. So when you
>>> use them together with lot of other plugins like Condel, Consequence, etc..
>>> they may be causing the process to handle and die.
>>>
>>> Best regards,
>>> Guillermo. ****
>>>
>>>
>>>
>>> On 05/13/2013 03:55 PM, Duarte Molha wrote:****
>>>
>>> I also get this error... it is so prevalent and so difficult to pinpoint
>>> what is causing it that I have given up on forking my annotation process.
>>> ****
>>>
>>> ** **
>>>
>>> I do think it is related to the number of forks. It seems to crash less
>>> often if you use a low number of forks... anything above 5
>>> will undoubtedly crash the script at least in my experience.****
>>>
>>> ** **
>>>
>>> Cheers****
>>>
>>>
>>> Duarte****
>>>
>>>
>>> ****
>>>
>>> =========================
>>> Duarte Miguel Paulo Molha ****
>>>
>>> http://about.me/duarte
>>> =========================****
>>>
>>> ** **
>>>
>>> On Mon, May 13, 2013 at 2:50 PM, Will McLaren <wm2 at ebi.ac.uk> wrote:****
>>>
>>> Hi Guillermo, ****
>>>
>>> ** **
>>>
>>> Test each plugin individually until you find the one that causes the
>>> error. It is highly unlikely that a particular combination of plugins is
>>> causing the crash.****
>>>
>>> ** **
>>>
>>> Check that there are no "print" (to STDOUT or STDERR) statements in your
>>> plugin - forking assumes that code remains silent otherwise it will throw
>>> errors like this.****
>>>
>>> ** **
>>>
>>> Also, check what, if anything, is cached between runs of your plugin. If
>>> you are caching things (for example to avoid re-querying a database), you
>>> may need to write storable hooks to ensure the data is getting cached
>>> between forks - see
>>> https://github.com/ensembl-variation/VEP_plugins/blob/master/ProteinSeqs.pmfor an example.
>>> ****
>>>
>>> ** **
>>>
>>> If you still have no luck, send me the code and an input file that
>>> recreates the problem.****
>>>
>>> ** **
>>>
>>> Regards****
>>>
>>> ** **
>>>
>>> Will****
>>>
>>> ** **
>>>
>>> On 13 May 2013 13:18, Guillermo Marco Puche <
>>> guillermo.marco at sistemasgenomicos.com> wrote:****
>>>
>>> Hello,
>>>
>>> I've started to recently having problems with VEP script while using
>>> different plugins (most of them own plugins).****
>>>
>>> 2013-05-13 13:59:44 - Connected to core version 71 database and variation version 71 database****
>>>
>>> 2013-05-13 13:59:44 - Loaded plugin: vcf_input****
>>>
>>> 2013-05-13 13:59:44 - Loaded plugin: biobase****
>>>
>>> 2013-05-13 13:59:44 - Loaded plugin: aa_grantham_distance****
>>>
>>> 2013-05-13 13:59:44 - Loaded plugin: flanking_sequence****
>>>
>>> 2013-05-13 13:59:44 - Loaded plugin: Condel****
>>>
>>> 2013-05-13 13:59:44 - Output fields redefined (37 defined)****
>>>
>>> 2013-05-13 13:59:44 - Starting...****
>>>
>>> 2013-05-13 13:59:45 - Read 3888 variants into buffer****
>>>
>>>
>>> 2013-05-13 13:59:54 - Reading transcript data from cache and/or database****
>>>
>>> [===============================================] [ 100% ]****
>>>
>>> 2013-05-13 14:02:38 - Retrieved 6463 transcripts (0 mem, 0 cached, 13743 DB, 7280 duplicates)****
>>>
>>> 2013-05-13 14:02:38 - Calculating consequences****
>>>
>>> [===================================> ] [ 78% ]****
>>>
>>> ERROR: Forked process failed****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> I'm not getting any other error message. So I cannot debug properly. I
>>> thought my plugins were OK but it's seems they don't. I think the problem
>>> occurs when I use "aa_grantham_distance plugin" together with
>>> "flanking_sequence". I've no idea what could be causing this.
>>>
>>> I'm running VEP on verbose mode but I can't get any usefull information.
>>> How could I debug that?
>>>
>>> Best regards,
>>> Guillermo.****
>>>
>>> ** **
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ** **
>>>
>>>
>>>
>>> ****
>>>
>>> _______________________________________________****
>>>
>>> Dev mailing list Dev at ensembl.org****
>>>
>>> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev****
>>>
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ** **
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ** **
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ** **
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ** **
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ** **
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ** **
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ** **
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ** **
>>>
>>>
>>>
>>>
>>> ****
>>>
>>> _______________________________________________****
>>>
>>> Dev mailing list Dev at ensembl.org****
>>>
>>> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev****
>>>
>>> Ensembl Blog: http://www.ensembl.info/****
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Dev mailing list Dev at ensembl.org
>>> Posting guidelines and subscribe/unsubscribe info:
>>> http://lists.ensembl.org/mailman/listinfo/dev
>>> Ensembl Blog: http://www.ensembl.info/
>>>
>>>
>>
>> _______________________________________________
>> Dev mailing list Dev at ensembl.org
>> Posting guidelines and subscribe/unsubscribe info:
>> http://lists.ensembl.org/mailman/listinfo/dev
>> Ensembl Blog: http://www.ensembl.info/
>>
>>
>
> _______________________________________________
> Dev mailing list Dev at ensembl.org
> Posting guidelines and subscribe/unsubscribe info:
> http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20130517/fe138153/attachment.html>
More information about the Dev
mailing list