[ensembl-dev] [EXTERNAL]Re: Downloading lots of files from ensembl

[Matthew Gerring]

Thanks for your help, I will give the http one a try, I have no idea that http is faster than ftp with wget.

Bye!

Matt

On 11/8/22, 11:30 AM, "Stefano Giorgetti" <sgiorgetti at ebi.ac.uk> wrote:

    Hello again Matthew,

    I've got an additional bit of advice, which may possibly help you.

    It's just occurred to me that your `wget` command is using FTP protocol.
    This should work fine, but it is adding un-needed overhead for creating 
    the connection.
    If this is needed, you might want to establish the FTP connection and 
    download multiple files with it ... but you should use an FTP client for it.

    Or - using `wget` - you might want to switch to HTTP(S), by just 
    replacing "ftp://" with "http://" (or "https://") in your URIs
    You should also experience better performance.

    Hope it helps

    Kind regards,
    S

    On 08/11/2022 13:37, Stefano Giorgetti wrote:
    > Dear Matthew,
    >
    > Thanks for your email and apologies for this not-so-prompt response.
    >
    > We have had some intermittent issues on FTP areas, mainly due to 
    > maintenance and upgrade activities, which might have affected long 
    > downloads.
    >
    > If you want to reduce requests, you might want to try the 
    > "--wait=seconds", "--waitretry=seconds" or "--random-wait" options.
    > There is also the "--limit-rate=amount", but I am not aware of such a 
    > need on our side. To be confirmed.
    >
    > In essence, you should be good with your `wget` command.
    >
    > I am double checking a few details, just in case, but please feel free 
    > to retry.
    >
    > Should you get significant amount of "connection refused" errors 
    > again, please do not hesitate to reach out to us.
    >
    > Kind regards
    >
    > Stefano
    >
    > On 26/10/2022 15:03, Matthew Gerring wrote:
    >> Should I be reducing requests? (How?
    >
    -- 
    —
    Stefano Giorgetti
    Ensembl Infrastructure Team Leader, EMBL-EBI
    sgiorgetti at ebi.ac.uk


---

The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.