[ensembl-dev] VEP script using Fork

[Will McLaren]

Garbled output probably won't help, but thanks for the offer anyway!
It might be more useful to know what compute setup you are using
(Mac/UNIX/Windows, local machine/farm, RAM/CPU etc)

Speed can vary wildly depending on the type of data you have - that
example I gave is on a test set covering only a small chromosomal
region, so the script doesn't spend very long loading transcript data
from the cache.

Will

On 3 August 2012 12:23, Duarte Molha <Duarte.Molha at ogt.co.uk> wrote:
> I get a bunch of garbled output, probably a memory bump of some kind.
> I can send you a file with the output if you wish.
> However, there was another thing I notice on your email... 750 vars/sec !!!
>
> I am glad if I reach 75 vars/sec using cache.
>
> I am not sure if there is something wrong with my setup but that value is completely unreachable in my system :S
>
> Cheers
>
> Duarte
>
>
> -----Original Message-----
> From: dev-bounces at ensembl.org [mailto:dev-bounces at ensembl.org] On Behalf Of Will McLaren
> Sent: 03 August 2012 11:06
> To: Ensembl developers list
> Subject: Re: [ensembl-dev] VEP script using Fork
>
> Hi Duarte,
>
> I've tested every fork number up to 30 without it crashing - what is the error you're seeing when it crashes?
>
> I'd be surprised if you got any significant benefit using any more than 4, though, since there are overheads associated with creating each fork.
>
> You can get a feel for the performance by looking at the status output that gives you the rate, something like:
>
> 2012-08-02 15:23:01 - Processed 16037 total variants (750 vars/sec,
> 927 vars/sec total)
>
> Cheers
>
> Will
>
> On 3 August 2012 10:54, Duarte Molha <Duarte.Molha at ogt.co.uk> wrote:
>> Hi
>>
>>
>>
>> I am trying out the new version and the fork does improve the performance.
>>
>>
>>
>> However the script crashes if I use more than 5 as the argument...
>> since my server has many more cores than that I would like to try and
>> see my much better it could run say forking at 16 (1 fork per core). Is this possible?
>>
>>
>>
>> Best regards,
>>
>>
>>
>> Duarte Molha
>>
>>
>>
>>
>> _______________________________________________
>> Dev mailing list    Dev at ensembl.org
>> List admin (including subscribe/unsubscribe):
>> http://lists.ensembl.org/mailman/listinfo/dev
>> Ensembl Blog: http://www.ensembl.info/
>>
>
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> List admin (including subscribe/unsubscribe): http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/
>
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> List admin (including subscribe/unsubscribe): http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/