[ensembl-dev] skip lock tables for ensembl-compara

Mateus Patricio mateus at ebi.ac.uk
Wed Nov 15 10:49:12 GMT 2017


Hi Francesco,

There has been some changes in that piece of code recently, which may have fixed this issue.

Could you try updating to e91?

Cheers,

Mateus.

> On 15 Nov 2017, at 10:23, Francesco Lamanna <francesco.lamanna at gmail.com> wrote:
> 
> Hi Mateus,
> 
> I am using release/90 of the Compara pipeline.
> 
> Cheers,
> Francesco.
> 
> 2017-11-15 11:07 GMT+01:00 Mateus Patricio <mateus at ebi.ac.uk <mailto:mateus at ebi.ac.uk>>:
> Hi Francesco,
> 
> Could you please confirm which checkout version of the compara code you have?
> 
> Cheers,
> 
> Mateus.
> 
>> On 13 Nov 2017, at 10:50, Francesco Lamanna <francesco.lamanna at gmail.com <mailto:francesco.lamanna at gmail.com>> wrote:
>> 
>> Hi Mateus and Brandon,
>> 
>> many thanks for your patch, it solved the issue.
>> 
>> Unfortunately, I got stuck at the successive step with the following error message:
>> 
>> mysql> SELECT * FROM msg;
>> +-------------+----------------------------+----------------+--------+---------+-----------+---------------------+-------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+
>> | analysis_id | logic_name                 | log_message_id | job_id | role_id | worker_id | when_logged         | retry | status       | msg                                                                                                                                                                      | is_error |
>> +-------------+----------------------------+----------------+--------+---------+-----------+---------------------+-------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+
>> |           9 | copy_ncbi_table            |              1 |      5 |       3 |         3 | 2017-11-13 11:16:31 |     0 | WRITE_OUTPUT | Successfully copied 1609518 'ncbi_taxa_node' rows                                                                                                                        |        0 |
>> |           9 | copy_ncbi_table            |              2 |      6 |       4 |         4 | 2017-11-13 11:20:15 |     0 | WRITE_OUTPUT | Successfully copied 2438889 'ncbi_taxa_name' rows                                                                                                                        |        0 |
>> |          18 | make_treebest_species_tree |              3 |     10 |       9 |         9 | 2017-11-13 11:25:29 |     0 | FETCH_INPUT  | Can't call method "get_all_subnodes" on an undefined value at /home/hd/hd_hd/hd_cc141/EnsEMBL/ensembl-compara/modules/Bio/EnsEMBL/Compara/Utils/SpeciesTree.pm line 150. |        1 |
>> |          18 | make_treebest_species_tree |              4 |     10 |      11 |        11 | 2017-11-13 11:26:29 |     1 | FETCH_INPUT  | Can't call method "get_all_subnodes" on an undefined value at /home/hd/hd_hd/hd_cc141/EnsEMBL/ensembl-compara/modules/Bio/EnsEMBL/Compara/Utils/SpeciesTree.pm line 150. |        1 |
>> |          18 | make_treebest_species_tree |              5 |     10 |      13 |        13 | 2017-11-13 11:27:31 |     2 | FETCH_INPUT  | Can't call method "get_all_subnodes" on an undefined value at /home/hd/hd_hd/hd_cc141/EnsEMBL/ensembl-compara/modules/Bio/EnsEMBL/Compara/Utils/SpeciesTree.pm line 150. |        1 |
>> |          18 | make_treebest_species_tree |              6 |     10 |      16 |        16 | 2017-11-13 11:28:32 |     3 | FETCH_INPUT  | Can't call method "get_all_subnodes" on an undefined value at /home/hd/hd_hd/hd_cc141/EnsEMBL/ensembl-compara/modules/Bio/EnsEMBL/Compara/Utils/SpeciesTree.pm line 150. |        1 |
>> +-------------+----------------------------+----------------+--------+---------+-----------+---------------------+-------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+
>> 6 rows in set (0.00 sec)
>> 
>> Do you have any suggestion?
>> 
>> Thanks,
>> Francesco
>> 
>> 2017-11-09 17:25 GMT+01:00 Brandon Walts <bwalts at ebi.ac.uk <mailto:bwalts at ebi.ac.uk>>:
>> Hi Francesco
>> 
>> We have just pushed a fix to the MySQLTransfer.pm runnable, so that it no longer locks tables by default. If you go to your eHive checkout and perform a git pull, you should get this fix. 
>> Users who wish to retain the table locking behavior can switch it on by passing a parameter 'lock_tables=1' to MySQLTransfer.pm
>> 
>> Best
>> -Brandon
>> 
>> 
>> On 09/11/2017 16:20, Mateus Patricio wrote:
>>> Hi Francesco,
>>> 
>>> So what’s happening here is that you are trying to lock a table in one of our servers.
>>> 
>>> That can be fixed by telling mysqldump to avoid locking the table. (Like you initially suspected)
>>> 
>>> You could do that directly on the eHive runnable "ehive/master/modules/Bio/EnsEMBL/Hive/RunnableDB/MySQLTransfer.pm <http://fer.pm/>”
>>> 
>>> The dump command is constructed at:
>>> https://github.com/Ensembl/ensembl-hive/blob/6c4877d4eda1bf0469c29bdbe58932671848cc20/modules/Bio/EnsEMBL/Hive/RunnableDB/MySQLTransfer.pm#L117 <https://github.com/Ensembl/ensembl-hive/blob/6c4877d4eda1bf0469c29bdbe58932671848cc20/modules/Bio/EnsEMBL/Hive/RunnableDB/MySQLTransfer.pm#L117>
>>> 
>>> I would suggest you to create a new entry to “mode_options”:
>>> https://github.com/Ensembl/ensembl-hive/blob/6c4877d4eda1bf0469c29bdbe58932671848cc20/modules/Bio/EnsEMBL/Hive/RunnableDB/MySQLTransfer.pm#L112 <https://github.com/Ensembl/ensembl-hive/blob/6c4877d4eda1bf0469c29bdbe58932671848cc20/modules/Bio/EnsEMBL/Hive/RunnableDB/MySQLTransfer.pm#L112>
>>> 
>>> So that you could access directly from your pipeline config file, by setting something like:
>>> 
>>>>         {   -logic_name    => 'copy_ncbi_table',
>>>>             -module        => 'Bio::EnsEMBL::Hive::RunnableDB::MySQLTransfer',
>>>>             -parameters    => {
>>>>                 'src_db_conn'   => '#ncbi_db#',
>>>>                 'mode'          => ’SKIP_LOCK',
>>>>                 'filter_cmd'    => 'sed "s/ENGINE=MyISAM/ENGINE=InnoDB/"',
>>>>             },
>>>>         },
>>> 
>>> Please let me know if that doesn’t work.
>>> 
>>> Cheers,
>>> 
>>> Mateus.
>>> 
>>> 
>>> 
>>> 
>>>> On 9 Nov 2017, at 15:44, Francesco Lamanna <francesco.lamanna at gmail.com <mailto:francesco.lamanna at gmail.com>> wrote:
>>>> 
>>>> Hi Mateus,
>>>> 
>>>> thank you for your answer.
>>>> 
>>>> I am using 'ncbi_db'   => 'mysql://anonymous@ensembldb.ensembl.org/ensembl_compara_90 <http://anonymous@ensembldb.ensembl.org/ensembl_compara_90>' to conncect to ncbi_db (I used the same value as in the NoMasterProteinTrees_conf.pm example).
>>>> 
>>>> The failing analysis is copy_ncbi_table(9).
>>>> 
>>>> Cheers,
>>>> Francesco.
>>>> 
>>>> 2017-11-09 16:29 GMT+01:00 Mateus Patricio <mateus at ebi.ac.uk <mailto:mateus at ebi.ac.uk>>:
>>>> Hi Francesco,
>>>> 
>>>> Looks like your pipeline is using the user anonymous to connect to the database.
>>>> 
>>>> I would suggest that you add the skip option within your runnable.
>>>> 
>>>> Which analysis is actually failing?
>>>> 
>>>> Cheers,
>>>> 
>>>> Mateus.
>>>> 
>>>> 
>>>>> On 8 Nov 2017, at 17:36, Francesco Lamanna <francesco.lamanna at gmail.com <mailto:francesco.lamanna at gmail.com>> wrote:
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> I am trying to apply the ProteinTrees pipeline to a set of Ensembl and non-Ensembl genomes. 
>>>>> 
>>>>> When running the pipeline without a master database (via beekeeper.pl <http://beekeeper.pl/>) I get the following error:
>>>>> 
>>>>> ensembl_compara_90 ncbi_taxa_node  | sed "s/ENGINE=MyISAM/ENGINE=InnoDB/" |  mysql -h127.0.0.1 -P3306 -uroot -p$EHIVE_TMP_PASSWORD_0 hd_cc141_test_protein_trees_no_master_90' resulted in an error code=2
>>>>> stderr is: mysql: [Warning] Using a password on the command line interface can be insecure.
>>>>> mysqldump: Got error: 1044: Access denied for user 'anonymous'@'%' to database 'ensembl_compara_90' when doing LOCK TABLES 
>>>>> 
>>>>> 
>>>>> Is there a way to pass --skip-lock-tables to the pipeline (e.g. in the conf file) in order to be able to dump the database?
>>>>> 
>>>>> Thanks,
>>>>> Francesco.
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Dev mailing list    Dev at ensembl.org <mailto:Dev at ensembl.org>
>>>>> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev <http://lists.ensembl.org/mailman/listinfo/dev>
>>>>> Ensembl Blog: http://www.ensembl.info/ <http://www.ensembl.info/>
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Dev mailing list    Dev at ensembl.org <mailto:Dev at ensembl.org>
>>>> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev <http://lists.ensembl.org/mailman/listinfo/dev>
>>>> Ensembl Blog: http://www.ensembl.info/ <http://www.ensembl.info/>
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Dev mailing list    Dev at ensembl.org <mailto:Dev at ensembl.org>
>>>> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev <http://lists.ensembl.org/mailman/listinfo/dev>
>>>> Ensembl Blog: http://www.ensembl.info/ <http://www.ensembl.info/>
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Dev mailing list    Dev at ensembl.org <mailto:Dev at ensembl.org>
>>> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev <http://lists.ensembl.org/mailman/listinfo/dev>
>>> Ensembl Blog: http://www.ensembl.info/ <http://www.ensembl.info/>
>> 
>> 
>> _______________________________________________
>> Dev mailing list    Dev at ensembl.org <mailto:Dev at ensembl.org>
>> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev <http://lists.ensembl.org/mailman/listinfo/dev>
>> Ensembl Blog: http://www.ensembl.info/ <http://www.ensembl.info/>
>> 
>> 
>> _______________________________________________
>> Dev mailing list    Dev at ensembl.org <mailto:Dev at ensembl.org>
>> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev <http://lists.ensembl.org/mailman/listinfo/dev>
>> Ensembl Blog: http://www.ensembl.info/ <http://www.ensembl.info/>
> 
> 
> _______________________________________________
> Dev mailing list    Dev at ensembl.org <mailto:Dev at ensembl.org>
> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev <http://lists.ensembl.org/mailman/listinfo/dev>
> Ensembl Blog: http://www.ensembl.info/ <http://www.ensembl.info/>
> 
> 
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20171115/98b14ebb/attachment.html>


More information about the Dev mailing list