[ensembl-dev] Ensembl FTP server seems to be down more often than not recently

Ben Moore bmoore at ebi.ac.uk
Fri Feb 1 09:12:53 GMT 2019 

Hi Kurt,

No problem- I’m glad to see that you have been able to resolve the issue downloading files from the FTP site from your end. Please do get back in touch if you continue to experience any problems.

Best wishes

Ben

> On 31 Jan 2019, at 20:00, Kurt Wheeler <kurt.wheeler91 at gmail.com> wrote:
> 
> Sorry!!!
> 
> So I didn't realize that when you use the `--spider` flag, wget will return a status code of 8 for some reason. This means that my script was backwards and I was seeing success ~66% of the time. However, I think the failures may have been related to a networking configuration we had set up. We corrected this and now it seems like we can reliable pull this file.
> 
> Sorry again for any confusion or distraction this caused. If it ends up being a problem again with the network configuration changes we made, I'll follow back up.
> 
> Thanks,
> 
> - Kurt
> 
> On Thu, Jan 31, 2019 at 2:24 PM Kurt Wheeler <kurt.wheeler91 at gmail.com <mailto:kurt.wheeler91 at gmail.com>> wrote:
> Hello again,
> 
> I wrote a small script to make sure that I'm not just getting unlucky with when I try to hit it. This is the script I used:
> 
> 
> #!/bin/bash
> 
> while true; do
>     if $(wget --spider --timeout=30 --tries=1 "ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz <ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz>"); then
>         status="up"
>     else
>         status="down"
>     fi
>     echo "$(date), $status" >> ensembl_statuses.csv
>     sleep 300
> done
> 
> 
> And here is the output for almost 3 hours:
> 
> Thu Jan 31 10:26:05 EST 2019, up
> Thu Jan 31 10:47:00 EST 2019, up
> Thu Jan 31 10:54:26 EST 2019, down
> Thu Jan 31 10:59:28 EST 2019, down
> Thu Jan 31 11:04:29 EST 2019, down
> Thu Jan 31 11:22:58 EST 2019, down
> Thu Jan 31 11:25:03 EST 2019, down
> Thu Jan 31 11:30:05 EST 2019, down
> Thu Jan 31 11:35:06 EST 2019, down
> Thu Jan 31 11:40:37 EST 2019, down
> Thu Jan 31 11:46:08 EST 2019, up
> Thu Jan 31 11:51:10 EST 2019, down
> Thu Jan 31 11:56:41 EST 2019, down
> Thu Jan 31 12:02:12 EST 2019, down
> Thu Jan 31 12:07:43 EST 2019, down
> Thu Jan 31 12:13:14 EST 2019, down
> Thu Jan 31 12:18:45 EST 2019, down
> Thu Jan 31 12:24:16 EST 2019, up
> Thu Jan 31 12:29:48 EST 2019, up
> Thu Jan 31 12:35:19 EST 2019, down
> Thu Jan 31 12:40:49 EST 2019, down
> Thu Jan 31 12:46:20 EST 2019, down
> Thu Jan 31 12:51:52 EST 2019, down
> Thu Jan 31 12:57:23 EST 2019, up
> Thu Jan 31 13:02:54 EST 2019, up
> Thu Jan 31 13:08:25 EST 2019, down
> Thu Jan 31 13:13:56 EST 2019, up
> Thu Jan 31 13:19:27 EST 2019, down
> Thu Jan 31 13:24:29 EST 2019, down
> Thu Jan 31 13:29:30 EST 2019, down
> Thu Jan 31 13:35:01 EST 2019, down
> Thu Jan 31 13:40:32 EST 2019, up
> Thu Jan 31 13:46:03 EST 2019, down
> Thu Jan 31 13:51:34 EST 2019, down
> Thu Jan 31 13:57:05 EST 2019, down
> Thu Jan 31 14:02:36 EST 2019, down
> Thu Jan 31 14:08:07 EST 2019, up
> Thu Jan 31 14:13:09 EST 2019, down
> Thu Jan 31 14:18:40 EST 2019, down
> 
> 
> As you can see, only about 34% of the requests I've made to it have been successful.
> 
> Hope this helps,
> 
> - Kurt
> 
> On Thu, Jan 31, 2019 at 10:21 AM Kurt Wheeler <kurt.wheeler91 at gmail.com <mailto:kurt.wheeler91 at gmail.com>> wrote:
> Hi Ben,
> 
> It came back this morning:
> 
> 2019-01-31 15:08:44,914 local [volume: 0] data_refinery_workers.downloaders.transcriptome_index DEBUG [downloader_job: 1]: Downloading file from ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz <ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz> to /home/user/data_store/Caenorhabditis_elegans/Caenorhabditis_elegans.fa.gz.
> 2019-01-31 15:10:55,982 local [volume: 0] data_refinery_workers.downloaders.transcriptome_index ERROR [downloader_job: 1]: Exception caught while downloading file from: ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz <ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz>
> Traceback (most recent call last):
>   File "/usr/lib/python3.5/urllib/request.py", line 1478, in ftp_open
>     fp, retrlen = fw.retrfile(file, type)
>   File "/usr/lib/python3.5/urllib/request.py", line 2347, in retrfile
>     conn, retrlen = self.ftp.ntransfercmd(cmd)
>   File "/usr/lib/python3.5/ftplib.py", line 360, in ntransfercmd
>     source_address=self.source_address)
>   File "/usr/lib/python3.5/socket.py", line 711, in create_connection
>     raise err
>   File "/usr/lib/python3.5/socket.py", line 702, in create_connection
>     sock.connect(sa)
> TimeoutError: [Errno 110] Connection timed out
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/home/user/data_refinery_workers/downloaders/transcriptome_index.py", line 42, in _download_file
>     with closing(urllib.request.urlopen(download_url)) as request:
>   File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
>     return opener.open(url, data, timeout)
>   File "/usr/lib/python3.5/urllib/request.py", line 466, in open
>     response = self._open(req, data)
>   File "/usr/lib/python3.5/urllib/request.py", line 484, in _open
>     '_open', req)
>   File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
>     result = func(*args)
>   File "/usr/lib/python3.5/urllib/request.py", line 1489, in ftp_open
>     raise exc.with_traceback(sys.exc_info()[2])
>   File "/usr/lib/python3.5/urllib/request.py", line 1478, in ftp_open
>     fp, retrlen = fw.retrfile(file, type)
>   File "/usr/lib/python3.5/urllib/request.py", line 2347, in retrfile
>     conn, retrlen = self.ftp.ntransfercmd(cmd)
>   File "/usr/lib/python3.5/ftplib.py", line 360, in ntransfercmd
>     source_address=self.source_address)
>   File "/usr/lib/python3.5/socket.py", line 711, in create_connection
>     raise err
>   File "/usr/lib/python3.5/socket.py", line 702, in create_connection
>     sock.connect(sa)
> urllib.error.URLError: <urlopen error ftp error: TimeoutError(110, 'Connection timed out')>
> 2019-01-31 15:10:56,025 local [volume: 0] data_refinery_workers.downloaders.utils INFO [downloader_job: 1] [failure_reason: Exception caught while downloading file from: ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz <ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz>] [downloader_task: TRANSCRIPTOME_INDEX]: Downloader job failed!
> 
> 
> It's somewhat unfortunate that it was working when you tried it. We've been seeing it work occasionally, but recently we have been finding it be down more than it has been up. Do the servers have any monitoring set up?
> 
> Thanks,
> 
> - Kurt
> 
> On Thu, Jan 31, 2019 at 6:20 AM Ben Moore <bmoore at ebi.ac.uk <mailto:bmoore at ebi.ac.uk>> wrote:
> Hi Kurt,
> 
> I’ve tried to replicate the error, but have been able to download the C. elegans GTF file from the FTP server. It is possible that you have encountered intermittent errors, but we believe that there are no issues currently affecting the FTP server. Please try again, and get back in touch if you continue to experience the same issues.
> 
> We apologise for any inconvenience you’ve experienced.
> 
> Best wishes
> 
> Ben
> 
>> On 30 Jan 2019, at 19:06, Kurt Wheeler <kurt.wheeler91 at gmail.com <mailto:kurt.wheeler91 at gmail.com>> wrote:
>> 
>> Hello,
>> 
>> My application has been having trouble downloading files from the Ensembl FTP server. One specific file I have had trouble downloading is:
>> 
>>   ftp://ftp.ensembl.org/pub/release-95/gtf/caenorhabditis_elegans/Caenorhabditis_elegans.WBcel235.95.gtf.gz <ftp://ftp.ensembl.org/pub/release-95/gtf/caenorhabditis_elegans/Caenorhabditis_elegans.WBcel235.95.gtf.gz>
>> 
>> However, I also am usually unable to even view a directory such as:
>> 
>>   ftp://ftp.ensembl.org/pub/release-95/gtf/ <ftp://ftp.ensembl.org/pub/release-95/gtf/>
>> 
>> This makes me think the FTP server has not been running for large amounts of time. While I was writing this email it seems to have come back up, but a colleague of mine and I have observed it coming up for small periods of time over the last week or so while remaining generally unreachable.
>> 
>> Is anyone able to look into this?
>> 
>> Thanks,
>> 
>> - Kurt
>> _______________________________________________
>> Dev mailing list    Dev at ensembl.org <mailto:Dev at ensembl.org>
>> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev <http://lists.ensembl.org/mailman/listinfo/dev>
>> Ensembl Blog: http://www.ensembl.info/ <http://www.ensembl.info/>
> 
> Ben Moore
> Ensembl Outreach Officer
> 
> European Bioinformatics Institute (EMBL-EBI)
> European Molecular Biology Laboratory
> Wellcome Trust Genome Campus
> Hinxton
> Cambridge
> CB10 1SD
> UK
> 
> bmoore at ebi.ac.uk <mailto:bmoore at ebi.ac.uk>
> +44 (0)1223 494265
> 
> _______________________________________________
> Dev mailing list    Dev at ensembl.org <mailto:Dev at ensembl.org>
> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev <http://lists.ensembl.org/mailman/listinfo/dev>
> Ensembl Blog: http://www.ensembl.info/ <http://www.ensembl.info/>
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> Posting guidelines and subscribe/unsubscribe info: http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/

Ben Moore
Ensembl Outreach Officer

European Bioinformatics Institute (EMBL-EBI)
European Molecular Biology Laboratory
Wellcome Trust Genome Campus
Hinxton
Cambridge
CB10 1SD
UK

bmoore at ebi.ac.uk
+44 (0)1223 494265

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20190201/c8d6daa9/attachment.html>

More information about the Dev mailing list