[ensembl-dev] Ensembl FTP server seems to be down more often than not recently
Kurt Wheeler
kurt.wheeler91 at gmail.com
Thu Jan 31 19:24:34 GMT 2019
Hello again,
I wrote a small script to make sure that I'm not just getting unlucky with
when I try to hit it. This is the script I used:
#!/bin/bash
while true; do
if $(wget --spider --timeout=30 --tries=1 "
ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz");
then
status="up"
else
status="down"
fi
echo "$(date), $status" >> ensembl_statuses.csv
sleep 300
done
And here is the output for almost 3 hours:
Thu Jan 31 10:26:05 EST 2019, up
Thu Jan 31 10:47:00 EST 2019, up
Thu Jan 31 10:54:26 EST 2019, down
Thu Jan 31 10:59:28 EST 2019, down
Thu Jan 31 11:04:29 EST 2019, down
Thu Jan 31 11:22:58 EST 2019, down
Thu Jan 31 11:25:03 EST 2019, down
Thu Jan 31 11:30:05 EST 2019, down
Thu Jan 31 11:35:06 EST 2019, down
Thu Jan 31 11:40:37 EST 2019, down
Thu Jan 31 11:46:08 EST 2019, up
Thu Jan 31 11:51:10 EST 2019, down
Thu Jan 31 11:56:41 EST 2019, down
Thu Jan 31 12:02:12 EST 2019, down
Thu Jan 31 12:07:43 EST 2019, down
Thu Jan 31 12:13:14 EST 2019, down
Thu Jan 31 12:18:45 EST 2019, down
Thu Jan 31 12:24:16 EST 2019, up
Thu Jan 31 12:29:48 EST 2019, up
Thu Jan 31 12:35:19 EST 2019, down
Thu Jan 31 12:40:49 EST 2019, down
Thu Jan 31 12:46:20 EST 2019, down
Thu Jan 31 12:51:52 EST 2019, down
Thu Jan 31 12:57:23 EST 2019, up
Thu Jan 31 13:02:54 EST 2019, up
Thu Jan 31 13:08:25 EST 2019, down
Thu Jan 31 13:13:56 EST 2019, up
Thu Jan 31 13:19:27 EST 2019, down
Thu Jan 31 13:24:29 EST 2019, down
Thu Jan 31 13:29:30 EST 2019, down
Thu Jan 31 13:35:01 EST 2019, down
Thu Jan 31 13:40:32 EST 2019, up
Thu Jan 31 13:46:03 EST 2019, down
Thu Jan 31 13:51:34 EST 2019, down
Thu Jan 31 13:57:05 EST 2019, down
Thu Jan 31 14:02:36 EST 2019, down
Thu Jan 31 14:08:07 EST 2019, up
Thu Jan 31 14:13:09 EST 2019, down
Thu Jan 31 14:18:40 EST 2019, down
As you can see, only about 34% of the requests I've made to it have been
successful.
Hope this helps,
- Kurt
On Thu, Jan 31, 2019 at 10:21 AM Kurt Wheeler <kurt.wheeler91 at gmail.com>
wrote:
> Hi Ben,
>
> It came back this morning:
>
> 2019-01-31 15:08:44,914 local [volume: 0]
> data_refinery_workers.downloaders.transcriptome_index DEBUG
> [downloader_job: 1]: Downloading file from
> ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz
> to
> /home/user/data_store/Caenorhabditis_elegans/Caenorhabditis_elegans.fa.gz.
> 2019-01-31 15:10:55,982 local [volume: 0]
> data_refinery_workers.downloaders.transcriptome_index ERROR
> [downloader_job: 1]: Exception caught while downloading file from:
> ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz
> Traceback (most recent call last):
> File "/usr/lib/python3.5/urllib/request.py", line 1478, in ftp_open
> fp, retrlen = fw.retrfile(file, type)
> File "/usr/lib/python3.5/urllib/request.py", line 2347, in retrfile
> conn, retrlen = self.ftp.ntransfercmd(cmd)
> File "/usr/lib/python3.5/ftplib.py", line 360, in ntransfercmd
> source_address=self.source_address)
> File "/usr/lib/python3.5/socket.py", line 711, in create_connection
> raise err
> File "/usr/lib/python3.5/socket.py", line 702, in create_connection
> sock.connect(sa)
> TimeoutError: [Errno 110] Connection timed out
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File
> "/home/user/data_refinery_workers/downloaders/transcriptome_index.py", line
> 42, in _download_file
> with closing(urllib.request.urlopen(download_url)) as request:
> File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
> return opener.open(url, data, timeout)
> File "/usr/lib/python3.5/urllib/request.py", line 466, in open
> response = self._open(req, data)
> File "/usr/lib/python3.5/urllib/request.py", line 484, in _open
> '_open', req)
> File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
> result = func(*args)
> File "/usr/lib/python3.5/urllib/request.py", line 1489, in ftp_open
> raise exc.with_traceback(sys.exc_info()[2])
> File "/usr/lib/python3.5/urllib/request.py", line 1478, in ftp_open
> fp, retrlen = fw.retrfile(file, type)
> File "/usr/lib/python3.5/urllib/request.py", line 2347, in retrfile
> conn, retrlen = self.ftp.ntransfercmd(cmd)
> File "/usr/lib/python3.5/ftplib.py", line 360, in ntransfercmd
> source_address=self.source_address)
> File "/usr/lib/python3.5/socket.py", line 711, in create_connection
> raise err
> File "/usr/lib/python3.5/socket.py", line 702, in create_connection
> sock.connect(sa)
> urllib.error.URLError: <urlopen error ftp error: TimeoutError(110,
> 'Connection timed out')>
> 2019-01-31 15:10:56,025 local [volume: 0]
> data_refinery_workers.downloaders.utils INFO [downloader_job: 1]
> [failure_reason: Exception caught while downloading file from:
> ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz]
> [downloader_task: TRANSCRIPTOME_INDEX]: Downloader job failed!
>
>
> It's somewhat unfortunate that it was working when you tried it. We've
> been seeing it work occasionally, but recently we have been finding it be
> down more than it has been up. Do the servers have any monitoring set up?
>
> Thanks,
>
> - Kurt
>
> On Thu, Jan 31, 2019 at 6:20 AM Ben Moore <bmoore at ebi.ac.uk> wrote:
>
>> Hi Kurt,
>>
>> I’ve tried to replicate the error, but have been able to download the *C.
>> elegans* GTF file from the FTP server. It is possible that you have
>> encountered intermittent errors, but we believe that there are no issues
>> currently affecting the FTP server. Please try again, and get back in touch
>> if you continue to experience the same issues.
>>
>> We apologise for any inconvenience you’ve experienced.
>>
>> Best wishes
>>
>> Ben
>>
>> On 30 Jan 2019, at 19:06, Kurt Wheeler <kurt.wheeler91 at gmail.com> wrote:
>>
>> Hello,
>>
>> My application has been having trouble downloading files from the Ensembl
>> FTP server. One specific file I have had trouble downloading is:
>>
>>
>> ftp://ftp.ensembl.org/pub/release-95/gtf/caenorhabditis_elegans/Caenorhabditis_elegans.WBcel235.95.gtf.gz
>>
>> However, I also am usually unable to even view a directory such as:
>>
>> ftp://ftp.ensembl.org/pub/release-95/gtf/
>>
>> This makes me think the FTP server has not been running for large amounts
>> of time. While I was writing this email it seems to have come back up, but
>> a colleague of mine and I have observed it coming up for small periods of
>> time over the last week or so while remaining generally unreachable.
>>
>> Is anyone able to look into this?
>>
>> Thanks,
>>
>> - Kurt
>> _______________________________________________
>> Dev mailing list Dev at ensembl.org
>> Posting guidelines and subscribe/unsubscribe info:
>> http://lists.ensembl.org/mailman/listinfo/dev
>> Ensembl Blog: http://www.ensembl.info/
>>
>>
>> Ben Moore
>> Ensembl Outreach Officer
>>
>> European Bioinformatics Institute (EMBL-EBI)
>> European Molecular Biology Laboratory
>> Wellcome Trust Genome Campus
>> Hinxton
>> Cambridge
>> CB10 1SD
>> UK
>>
>> bmoore at ebi.ac.uk
>> +44 (0)1223 494265
>>
>> _______________________________________________
>> Dev mailing list Dev at ensembl.org
>> Posting guidelines and subscribe/unsubscribe info:
>> http://lists.ensembl.org/mailman/listinfo/dev
>> Ensembl Blog: http://www.ensembl.info/
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20190131/a5bbe1e5/attachment.html>
More information about the Dev
mailing list