[ensembl-dev] Downloading lots of files from ensembl

[Stefano Giorgetti]

Hello again Matthew,

I've got an additional bit of advice, which may possibly help you.

It's just occurred to me that your `wget` command is using FTP protocol.
This should work fine, but it is adding un-needed overhead for creating 
the connection.
If this is needed, you might want to establish the FTP connection and 
download multiple files with it ... but you should use an FTP client for it.

Or - using `wget` - you might want to switch to HTTP(S), by just 
replacing "ftp://" with "http://" (or "https://") in your URIs
You should also experience better performance.

Hope it helps

Kind regards,
S

On 08/11/2022 13:37, Stefano Giorgetti wrote:
> Dear Matthew,
>
> Thanks for your email and apologies for this not-so-prompt response.
>
> We have had some intermittent issues on FTP areas, mainly due to 
> maintenance and upgrade activities, which might have affected long 
> downloads.
>
> If you want to reduce requests, you might want to try the 
> "--wait=seconds", "--waitretry=seconds" or "--random-wait" options.
> There is also the "--limit-rate=amount", but I am not aware of such a 
> need on our side. To be confirmed.
>
> In essence, you should be good with your `wget` command.
>
> I am double checking a few details, just in case, but please feel free 
> to retry.
>
> Should you get significant amount of "connection refused" errors 
> again, please do not hesitate to reach out to us.
>
> Kind regards
>
> Stefano
>
> On 26/10/2022 15:03, Matthew Gerring wrote:
>> Should I be reducing requests? (How?
>
-- 
—
Stefano Giorgetti
Ensembl Infrastructure Team Leader, EMBL-EBI
sgiorgetti at ebi.ac.uk