[ensembl-dev] Pipeline: Cluster architecture /requirements

Thibaut Hourlier th3 at sanger.ac.uk
Fri Feb 8 11:04:13 GMT 2013


Hi Marc,

On 08/02/13 08:55, Marc Hoeppner wrote:
> Hello again,
> thanks for the helpful comments on how to get a genome into a blank DB!
> I made some decent progress in figuring out how to configure certain 
> things, but have a more general question regarding the system 
> architecture that the gene build pipeline expects to find.
> When trying to run repeat masker, it complains about two things.
> 1) Not being able to create a folder /RepeatMasker
> I can't seem to figure out where that is defined, since I don't really 
> want anything to be created in the root file system
Did you have a look at this documentation? 
http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-doc/pipeline_docs/the_ensembl_pipeline_infrastructure.txt?root=ensembl&view=log
All the setting for the pipeline are in 
ensembl-pipeline/modules/Bio/EnsEMBL/Pipeline/Config/BatchQueue.pm (you 
will have to do a copy from the .example file).

You can either change the DEFAULT_OUTPUT_DIR, the log files will be in 
<DEFAULT_OUTPUT_DIR>/<logic_name>
Or you can add in the analysis hash: output_dir => "/my/path/to/logs",
> 2) No bsub in my system, which I think means that LFS is required for 
> the different jobs to run?
> I guess that makes sense - just want to make sure that LFS is indeed 
> necessary before I set out to install it (seems a bit painful...).
LSF is not necessary, you have 2 (3) choices:
  - run the pipeline locally, in the BatchQueue file set QUEUE_MANAGER 
=> 'Local',
       It will be long on big assemblies but you can make it run on your 
computer as it doesn't parallelize the jobs.
  - if you have a cluster, write the submission module for the system 
used. You can have a look in 
ensembl-pipeline/modules/Bio/EnsEMBL/Pipeline/BatchSubmission/LSF.pm to 
see which methods need to be implemented. There is already a module for 
GridEngine.
  (- install LSF...)

Cheers
Thibaut
> Cheers,
> Marc
>
>
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20130208/6efb40b2/attachment.html>


More information about the Dev mailing list