[ensembl-dev] run pipeline at localhost

[Susan Fairley]

Hi,

Managing the number of jobs that run concurrently is outwith the scope 
of the Ensembl pipeline. You need some sort of queue management system 
in place on your system to achieve that. The pipeline is designed to 
work alongside such a system by managing the pipeline jobs and managing 
the batches in which those jobs are passed to the submission/queue 
management system along with any requirements they have, such as the 
amount of memory they may need.

The DEFAULT_BATCH_SIZE option allows you to specify how many pipeline 
jobs you wish to place in a batch that will be submitted as a single job 
on your system.

The DEFAULT_BATCH_QUEUE option allows you to specify which queue, among 
those available on your submission system, you wish to use. If the 
queues available to you have different properties (i.e. priority over 
other jobs, limit on runtime, etc.) this can allow you to choose the 
most appropriate queue.

I run the pipeline using LSF but possibly Grid Engine may be an option 
for you. Running with Local, your jobs will, I believe, run serially.

I hope this is of some help.

Best regards,
Susan.

Zhang Di wrote:
> Hi,
>     How can I set the pipeline to run at localhost to use all of its cpus?
>     I known that for ensembl-compara, beekeeper.pl <http://beekeeper.pl> 
> has -local and -local_cpus options, while for ensembl-pipeline, 
> rulemanager.pl <http://rulemanager.pl> doesn't has such options.
>     I set QUEUE_MANAGER = 'Local' and DEFAUL_BATCH_QUEUE_SIZE = 8 in 
> BatchQueue.pm, but my RepeatMask analysis only run 1 process. 
>     
> Best Reguards
> 
> -- 
> Zhang Di
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> List admin (including subscribe/unsubscribe): http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/