Icgc-get download fails with docker container enabled


#1

Hi, I’m trying to download a sample bam file from EGA with icgc-get with docker container enabled.

The command line I’m using is:

./icgc-get -d True download FI9995

The download process fails. In spite of the fact that it is reported an error with a missing directory at the end, seems like the problem starts with a time out occurred during downloading. Multiple executions of the previous command end up with the same result. Below I copy part of the content of the log file:

| Output: Ticket: 4f57ed6a-d4d2-4c1b-82fe-572b3ffd03af
| URL: http://ega.ebi.ac.uk/ega/rest/ds/v2/downloads/4f57ed6a-d4d2-4c1b-82fe-572b3ffd03af
| Establishing Data Stream for: 4f57ed6a-d4d2-4c1b-82fe-572b3ffd03af
| Data Stream for 4f57ed6a-d4d2-4c1b-82fe-572b3ffd03af is established
| Path: /icgc/mnt/_EGAZ00001225589_efcf90ee-53ae-4f9f-b29a-e0a83ca70272_f5c9381090a53c54358feb2ba5b7a3d7.bam.cip
| Starting transfer loop for 4f57ed6a-d4d2-4c1b-82fe-572b3ffd03af (true)
|
| Download Error occurred at Stage 4
| Error: Read timed out
| Completed download: /EGAZ00001225589/efcf90ee-53ae-4f9f-b29a-e0a83ca70272
/f5c9381090a53c54358feb2ba5b7a3d7.bam.cip: false (0/1) | Download Failed: /EGAZ00001225589/efcf90ee-53ae-4f9f-b29a-e0a83ca70272/f5c9381090a53c54358feb2ba5b7a3d7.bam.cip 4f57ed6a-d4d2-4c1b-82fe-572b3ffd03af
| Iteration Done! (0 / 1)
| Download Attempt Completed. 1 of 1 tickets downloaded successfully.
| Post-Download: No outstanding Tickets
| Download Iteration Completed. Retry loop for 0 unsuccessful downloads.
|
|
| Logout!
|
| Exit
Traceback (most recent call last):
File "icgcget/cli.py", line 397, in
File "icgcget/cli.py", line 393, in main
File "site-packages/click/core.py", line 722, in __call__
File "site-packages/click/core.py", line 697, in main
File "site-packages/click/core.py", line 1066, in invoke
File "site-packages/click/core.py", line 895, in invoke
File "site-packages/click/core.py", line 535, in invoke
File "site-packages/click/decorators.py", line 17, in new_func
File "icgcget/cli.py", line 261, in download
File "icgcget/commands/download.py", line 130, in download
File "icgcget/commands/download.py", line 201, in client_download
File "icgcget/clients/ega/ega_client.py", line 99, in download
OSError: [Errno 2] No such file or directory: '/icgc/mnt'
Failed to execute script cli

Could you please help me to troubleshoot the problem?
Thanks!

j


#2

Hi,

For you information, I am able to use icgc-get to download the data file from Collaboratory by enable docker mode with command:
icgc-get -d True download -r collaboratory FI9995.
Could you give it a try to see if it works for you?

I will also try out downloading from EGA after my EGA account issue is sorted out. And I will update you if I find something which is related to your downloading problem.

-Linda


#3

HI Linda,

I am also able to download the data from Collaboratory! :grinning:

In any case if you find the solution to the EGA problem I will be very grateful if you can report me with a suitable solution.

Thanks a lot!!!

j


#4

Hi Jose,

Just want to let you know that I have try out using “icgc-get” to download data from EGA and experienced the same failures as yours.

As “icgc-get” now works as a wrapper of Java-based EGA download client when downloading data from EGA, we suspect the download failures have something to do with the EGA download client. We contacted EGA helpdesk and they suggest us to switch to use the new Python-base EGA download client: https://ega-archive.org/download/downloader-quickguide-v3. I just tried the new client to download directly from EGA and it works.

So for your case, currently if the data you are interested only exist in EGA, maybe you need to use the new Python-base EGA download client to download directly from EGA. If the data also has copies in Colloaboratory or AWS, you can always use “icgc-get” to get data from other places.

We will keep you updated once the “icgc-get” switch to use the new EGA download client.

Hope this information help with your questions.

Regards,

-Linda


#5

Thanks a lot Linda for the information! :grinning: I will proceed so.

Regards,

Jose