[ensembl-dev] Ensembl FTP server seems to be down more often than not recently

[Kurt Wheeler]

Hi Ben,

It came back this morning:

2019-01-31 15:08:44,914 local [volume: 0]
data_refinery_workers.downloaders.transcriptome_index DEBUG
[downloader_job: 1]: Downloading file from
ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz
to
/home/user/data_store/Caenorhabditis_elegans/Caenorhabditis_elegans.fa.gz.
2019-01-31 15:10:55,982 local [volume: 0]
data_refinery_workers.downloaders.transcriptome_index ERROR
[downloader_job: 1]: Exception caught while downloading file from:
ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz
Traceback (most recent call last):
  File "/usr/lib/python3.5/urllib/request.py", line 1478, in ftp_open
    fp, retrlen = fw.retrfile(file, type)
  File "/usr/lib/python3.5/urllib/request.py", line 2347, in retrfile
    conn, retrlen = self.ftp.ntransfercmd(cmd)
  File "/usr/lib/python3.5/ftplib.py", line 360, in ntransfercmd
    source_address=self.source_address)
  File "/usr/lib/python3.5/socket.py", line 711, in create_connection
    raise err
  File "/usr/lib/python3.5/socket.py", line 702, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/home/user/data_refinery_workers/downloaders/transcriptome_index.py", line
42, in _download_file
    with closing(urllib.request.urlopen(download_url)) as request:
  File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/usr/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 1489, in ftp_open
    raise exc.with_traceback(sys.exc_info()[2])
  File "/usr/lib/python3.5/urllib/request.py", line 1478, in ftp_open
    fp, retrlen = fw.retrfile(file, type)
  File "/usr/lib/python3.5/urllib/request.py", line 2347, in retrfile
    conn, retrlen = self.ftp.ntransfercmd(cmd)
  File "/usr/lib/python3.5/ftplib.py", line 360, in ntransfercmd
    source_address=self.source_address)
  File "/usr/lib/python3.5/socket.py", line 711, in create_connection
    raise err
  File "/usr/lib/python3.5/socket.py", line 702, in create_connection
    sock.connect(sa)
urllib.error.URLError: <urlopen error ftp error: TimeoutError(110,
'Connection timed out')>
2019-01-31 15:10:56,025 local [volume: 0]
data_refinery_workers.downloaders.utils INFO [downloader_job: 1]
[failure_reason: Exception caught while downloading file from:
ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz]
[downloader_task: TRANSCRIPTOME_INDEX]: Downloader job failed!


It's somewhat unfortunate that it was working when you tried it. We've been
seeing it work occasionally, but recently we have been finding it be down
more than it has been up. Do the servers have any monitoring set up?

Thanks,

- Kurt

On Thu, Jan 31, 2019 at 6:20 AM Ben Moore <bmoore at ebi.ac.uk> wrote:

> Hi Kurt,
>
> I’ve tried to replicate the error, but have been able to download the *C.
> elegans* GTF file from the FTP server. It is possible that you have
> encountered intermittent errors, but we believe that there are no issues
> currently affecting the FTP server. Please try again, and get back in touch
> if you continue to experience the same issues.
>
> We apologise for any inconvenience you’ve experienced.
>
> Best wishes
>
> Ben
>
> On 30 Jan 2019, at 19:06, Kurt Wheeler <kurt.wheeler91 at gmail.com> wrote:
>
> Hello,
>
> My application has been having trouble downloading files from the Ensembl
> FTP server. One specific file I have had trouble downloading is:
>
>
> ftp://ftp.ensembl.org/pub/release-95/gtf/caenorhabditis_elegans/Caenorhabditis_elegans.WBcel235.95.gtf.gz
>
> However, I also am usually unable to even view a directory such as:
>
>   ftp://ftp.ensembl.org/pub/release-95/gtf/
>
> This makes me think the FTP server has not been running for large amounts
> of time. While I was writing this email it seems to have come back up, but
> a colleague of mine and I have observed it coming up for small periods of
> time over the last week or so while remaining generally unreachable.
>
> Is anyone able to look into this?
>
> Thanks,
>
> - Kurt
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> Posting guidelines and subscribe/unsubscribe info:
> http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/
>
>
> Ben Moore
> Ensembl Outreach Officer
>
> European Bioinformatics Institute (EMBL-EBI)
> European Molecular Biology Laboratory
> Wellcome Trust Genome Campus
> Hinxton
> Cambridge
> CB10 1SD
> UK
>
> bmoore at ebi.ac.uk
> +44 (0)1223 494265
>
> _______________________________________________
> Dev mailing list    Dev at ensembl.org
> Posting guidelines and subscribe/unsubscribe info:
> http://lists.ensembl.org/mailman/listinfo/dev
> Ensembl Blog: http://www.ensembl.info/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ensembl.org/pipermail/dev_ensembl.org/attachments/20190131/37fcdce4/attachment.html>