[ensembl-dev] Pipeline: Cluster architecture /requirements
Thibaut Hourlier
th3 at sanger.ac.uk
Fri Feb 8 11:04:13 GMT 2013
Hi Marc,
On 08/02/13 08:55, Marc Hoeppner wrote:
> Hello again,
> thanks for the helpful comments on how to get a genome into a blank DB!
> I made some decent progress in figuring out how to configure certain
> things, but have a more general question regarding the system
> architecture that the gene build pipeline expects to find.
> When trying to run repeat masker, it complains about two things.
> 1) Not being able to create a folder /RepeatMasker
> I can't seem to figure out where that is defined, since I don't really
> want anything to be created in the root file system
Did you have a look at this documentation?
http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-doc/pipeline_docs/the_ensembl_pipeline_infrastructure.txt?root=ensembl&view=log
All the setting for the pipeline are in
ensembl-pipeline/modules/Bio/EnsEMBL/Pipeline/Config/BatchQueue.pm (you
will have to do a copy from the .example file).
You can either change the DEFAULT_OUTPUT_DIR, the log files will be in
<DEFAULT_OUTPUT_DIR>/<logic_name>
Or you can add in the analysis hash: output_dir => "/my/path/to/logs",
> 2) No bsub in my system, which I think means that LFS is required for
> the different jobs to run?
> I guess that makes sense - just want to make sure that LFS is indeed
> necessary before I set out to install it (seems a bit painful...).
LSF is not necessary, you have 2 (3) choices:
- run the pipeline locally, in the BatchQueue file set QUEUE_MANAGER
=> 'Local',
It will be long on big assemblies but you can make it run on your
computer as it doesn't parallelize the jobs.
- if you have a cluster, write the submission module for the system
used. You can have a look in
ensembl-pipeline/modules/Bio/EnsEMBL/Pipeline/BatchSubmission/LSF.pm to
see which methods need to be implemented. There is already a module for
GridEngine.
(- install LSF...)
Cheers
Thibaut
> Cheers,
> Marc
>
>
> _______________________________________________
> Dev mailing list Dev at ensembl.org
> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20130208/6efb40b2/attachment.html>
More information about the Dev
mailing list