Check content-length and md5 for Release 23 data


#1

I have downloaded data for Release 23, but i don’t know how to check it.

file list: data.open.tar / release.tar

Could you please provide content-length( in byte) and md5 information for each file?

Thanks!


#2

Hi @felix. We currently don’t have support for md5s, but we’ve created a ticket to address. Thanks for pointing this out.


#3

Thanks for your reply.

But i still don’t know how to check data for Release 23.

Can your help us download Release 23 data, because wget a big file is Unreliable.


#4

Hi Felix,

I apologize, but we need to implement this feature first. In the meantime, I can run an md5 on the downloaded tarballs locally and will paste here.


#5

For ICGC23 exports on https://download.icgc.org/exports

data.open.tar
Size = 76209510400
MD5 = 190402d81a2d3f8507c6fd6935cf90aa

release.tar
Size = 43776686080
MD5 = 3644515bb99730e63a1189b0322722a5

repository.tar.gz
Size = 71913045
MD5 = caf8c5efdd9611b9007df3b90f571504


#6

Hi btiernay,

We have tried to download data.open.tar and release.tar using links on https://download.icgc.org/exports from ShenZhen and HongKong, but unfortunately none of them matchs Size and MD5 you provided

md5sum results:

/ifshk1/pub/database/download.icgc.org/exports/release.tar [HongKong]
c272c9a82e854b34e622a7bcfbd704da

/ifs1/pub/database/download.icgc.org/exports/release.tar [ShenZhen]
6c0d9b4b730b0d561cabaafb9537525a 

I was wondering if it is possible to provide a file-link-list file to download data.open.tar and release.tar
eg:

# ICGC_release21_files.txt
https://dcc.icgc.org/api/v1/download?fn=/release_21/README.txt
https://dcc.icgc.org/api/v1/download?fn=/release_21/Summary/README.txt
https://dcc.icgc.org/api/v1/download?fn=/release_21/Summary/donor.all_projects.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Summary/donor_exposure.all_projects.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Summary/donor_family.all_projects.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Summary/donor_therapy.all_projects.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Summary/sample.all_projects.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Summary/simple_somatic_mutation.aggregated.vcf.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Summary/specimen.all_projects.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/ALL-US/donor.ALL-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/ALL-US/exp_array.ALL-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/ALL-US/sample.ALL-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/ALL-US/simple_somatic_mutation.open.ALL-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/ALL-US/specimen.ALL-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/AML-US/donor.AML-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/AML-US/sample.AML-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/AML-US/simple_somatic_mutation.open.AML-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/AML-US/specimen.AML-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-CN/donor.BLCA-CN.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-CN/sample.BLCA-CN.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-CN/simple_somatic_mutation.open.BLCA-CN.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-CN/specimen.BLCA-CN.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-US/copy_number_somatic_mutation.BLCA-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-US/donor.BLCA-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-US/exp_seq.BLCA-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-US/meth_array.BLCA-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-US/mirna_seq.BLCA-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-US/protein_expression.BLCA-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-US/sample.BLCA-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-US/simple_somatic_mutation.open.BLCA-US.tsv.gz
https://dcc.icgc.org/api/v1/download?fn=/release_21/Projects/BLCA-US/specimen.BLCA-US.tsv.gz
...

Thanks!


#7

Hi Felix,

I was afraid of that. The files are dynamically generated and there is a chance that there are non-semantic changes in the file contents. As a result, the md5 may not match. I have some thoughts about streaming the md5 at the end of the tar. Failing that, we can materialize the files to disk after every release.

Would you be so kind as to report the file sizes you have?

Thanks


#8

Hi btiernay,

Here is detail information:

ShenZhen Server

[yangfan2@login-20-3 release_23]$ ls -l
total 146806977
-rw-rw-r-- 1 database database 76209172480 Dec 17 01:38 data.open.tar
-rw-rw-r-- 1 database database    16548094 Dec 17 22:50 data.open.tar.files.md5
-rw-rw-r-- 1 database database          48 Dec 17 10:09 data.open.tar.md5
-rw-rw-r-- 1 database database 43776686080 Dec 16 07:21 release.tar
-rw-rw-r-- 1 database database        1043 Dec 17 19:08 release.tar.files.md5
-rw-rw-r-- 1 database database          46 Dec 16 23:45 release.tar.md5

# md5 info
[yangfan2@login-20-3 release_23]$ cat data.open.tar.md5 
62f1f68646a484b1cfebd9c7e06b248a  data.open.tar
[yangfan2@login-20-3 release_23]$ cat release.tar.md5 
6c0d9b4b730b0d561cabaafb9537525a  release.tar

# file md5 info
[yangfan2@login-20-3 release_23]$ wc -l data.open.tar.files.md5
183389 data.open.tar.files.md5
[yangfan2@login-20-3 release_23]$ head -10 data.open.tar.files.md5 
00001767cee47ec040fd8c06d24ac9dd  ./release_23/data/MELA-AU/DO222514/ssm_open/part-00013.gz
00007afd569e72db82c48b0f1b9f5c01  ./release_23/data/PACA-AU/DO33888/meth_array/part-00026.gz
0000e5125c1a8ccb22c73efb81fa24e5  ./release_23/data/PACA-AU/DO49117/exp_array/part-00001.gz
000114505caab1328fe578708a24b949  ./release_23/data/KIRC-US/DO19470/cnsm/part-00000.gz
00033b9361dc5f48f1acc3cd6bb75094  ./release_23/data/UCEC-US/DO52400/specimen/part-00001.gz
0003e33495cf4d4cb1cd71962cef5143  ./release_23/data/BRCA-EU/DO225384/ssm_open/part-00002.gz
000410d86b2f29fe1129f79999697344  ./release_23/data/KIRC-US/DO19553/pexp/part-00001.gz
00051ecf7806d0b45d3338b7fee0992c  ./release_23/data/PRAD-CA/DO51106/donor_family/part-00000.gz
00056cfc9eaa5d0407f7e1b199498978  ./release_23/data/BLCA-US/DO44063/meth_array/part-00006.gz
0005856a1c8de2afdbb6b60b87192a2b  ./release_23/data/THCA-US/DO39252/cnsm/part-00001.gz
[yangfan2@login-20-3 release_23]$ cat release.tar.files.md5 
ecf6e08020056136bead09807daf1d7b  icgc23-9_diagram.tar.gz
679e46afd0f3018389123b1238b122cd  icgc23-9_donor-centric.tar.gz
20b8cdf608211448039a1ccce157eee8  icgc23-9_donor.tar.gz
437edbac13cfd73c8d7ff7f17491f1d9  icgc23-9_donor-text.tar.gz
4e8792573578597e3844620978d91702  icgc23-9_drug-centric.tar.gz
7f37c0ee7e1dc3626345403dfda299ed  icgc23-9_drug-text.tar.gz
4136ef34dfe9995d6deb91723f1c3039  icgc23-9_gene-centric.tar.gz
3c340e8b2f82dc83a7ee43c12caa2d04  icgc23-9_gene-set.tar.gz
3a607efd00a6e8176253974621a71671  icgc23-9_gene-set-text.tar.gz
483387b7795a1186b1d00c54f4386248  icgc23-9_gene.tar.gz
7135eb14bdad7b61e209509f54c6111b  icgc23-9_gene-text.tar.gz
2b9b88d9571366ab37ecc6486203ea93  icgc23-9_mutation-centric.tar.gz
bbe1bb5530bde6fe3c3b751722664365  icgc23-9_mutation-text.tar.gz
aed17d35ac46ff5fc4a05f74e2bf741a  icgc23-9_observation-centric.tar.gz
d83d72c4387582f7ef312d9c78882723  icgc23-9_project.tar.gz
189dbf2bb1f311e50286df28edcc1f62  icgc23-9_project-text.tar.gz
8c4138800d86acd839ee1f41855c78c1  icgc23-9_release.tar.gz

HongKong Server

[yangfan2@login-0-4 release_23]$ ls -l
total 146832177
-rw-rw-r-- 1 database database 76209172480 Dec 16 02:34 data.open.tar
-rw-rw-r-- 1 database database    16548094 Dec 17 22:47 data.open.tar.files.md5
-rw-rw-r-- 1 database database          48 Dec 16 23:44 data.open.tar.md5
-rw-rw-r-- 1 database database 43776686080 Dec 15 23:44 release.tar
-rw-rw-r-- 1 database database        1043 Dec 17 19:06 release.tar.files.md5
-rw-rw-r-- 1 database database          46 Dec 16 23:43 release.tar.md5

# md5 info
[yangfan2@login-0-4 release_23]$ cat data.open.tar.md5
639abe1870e5f3ad460198cf149ecfeb  data.open.tar
[yangfan2@login-0-4 release_23]$ cat release.tar.md5
c272c9a82e854b34e622a7bcfbd704da  release.tar

# file md5 info
[yangfan2@login-0-4 release_23]$ wc -l data.open.tar.files.md5 
183389 data.open.tar.files.md5
[yangfan2@login-0-4 release_23]$ head -10 data.open.tar.files.md5 
00001767cee47ec040fd8c06d24ac9dd  ./release_23/data/MELA-AU/DO222514/ssm_open/part-00013.gz
00007afd569e72db82c48b0f1b9f5c01  ./release_23/data/PACA-AU/DO33888/meth_array/part-00026.gz
0000e5125c1a8ccb22c73efb81fa24e5  ./release_23/data/PACA-AU/DO49117/exp_array/part-00001.gz
000114505caab1328fe578708a24b949  ./release_23/data/KIRC-US/DO19470/cnsm/part-00000.gz
00033b9361dc5f48f1acc3cd6bb75094  ./release_23/data/UCEC-US/DO52400/specimen/part-00001.gz
0003e33495cf4d4cb1cd71962cef5143  ./release_23/data/BRCA-EU/DO225384/ssm_open/part-00002.gz
000410d86b2f29fe1129f79999697344  ./release_23/data/KIRC-US/DO19553/pexp/part-00001.gz
00051ecf7806d0b45d3338b7fee0992c  ./release_23/data/PRAD-CA/DO51106/donor_family/part-00000.gz
00056cfc9eaa5d0407f7e1b199498978  ./release_23/data/BLCA-US/DO44063/meth_array/part-00006.gz
0005856a1c8de2afdbb6b60b87192a2b  ./release_23/data/THCA-US/DO39252/cnsm/part-00001.gz
[yangfan2@login-0-4 release_23]$ cat release.tar.files.md5 
ecf6e08020056136bead09807daf1d7b  icgc23-9_diagram.tar.gz
679e46afd0f3018389123b1238b122cd  icgc23-9_donor-centric.tar.gz
20b8cdf608211448039a1ccce157eee8  icgc23-9_donor.tar.gz
437edbac13cfd73c8d7ff7f17491f1d9  icgc23-9_donor-text.tar.gz
4e8792573578597e3844620978d91702  icgc23-9_drug-centric.tar.gz
7f37c0ee7e1dc3626345403dfda299ed  icgc23-9_drug-text.tar.gz
4136ef34dfe9995d6deb91723f1c3039  icgc23-9_gene-centric.tar.gz
3c340e8b2f82dc83a7ee43c12caa2d04  icgc23-9_gene-set.tar.gz
3a607efd00a6e8176253974621a71671  icgc23-9_gene-set-text.tar.gz
483387b7795a1186b1d00c54f4386248  icgc23-9_gene.tar.gz
7135eb14bdad7b61e209509f54c6111b  icgc23-9_gene-text.tar.gz
2b9b88d9571366ab37ecc6486203ea93  icgc23-9_mutation-centric.tar.gz
bbe1bb5530bde6fe3c3b751722664365  icgc23-9_mutation-text.tar.gz
aed17d35ac46ff5fc4a05f74e2bf741a  icgc23-9_observation-centric.tar.gz
d83d72c4387582f7ef312d9c78882723  icgc23-9_project.tar.gz
189dbf2bb1f311e50286df28edcc1f62  icgc23-9_project-text.tar.gz
8c4138800d86acd839ee1f41855c78c1  icgc23-9_release.tar.gz

It seems that size for release.tar matchs what you provided.


#9

Hi @btiernay,

Tried to import data files on our Shenzhen server.

ES cluster works fine, i can see release23 info on front end.

But after importing data to HDFS (hdfs@hdfs-01:/dfs/tmp$ hdfs dfs -copyFromLocal release_23 /icgc/input), download service can’t start any more.

Files in HDFS:

root@hdfs-01:~# su - hdfs
hdfs@hdfs-01:~$ hadoop fs -ls /icgc/input/release_23
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Found 5 items
-rw-r--r--   3 hdfs hadoop        354 2017-02-08 16:23 /icgc/input/release_23/README.txt
drwxr-xr-x   - hdfs hadoop          0 2017-02-08 17:52 /icgc/input/release_23/data
drwxr-xr-x   - hdfs hadoop          0 2017-02-08 17:53 /icgc/input/release_23/headers
drwxr-xr-x   - hdfs hadoop          0 2017-02-08 17:53 /icgc/input/release_23/projects_files
drwxr-xr-x   - hdfs hadoop          0 2017-02-08 17:53 /icgc/input/release_23/summary_files

A little experiment:
Delete Release23 data in HDFS, and import Release22 data, download service can start.

Maybe it’s because of the incompletion download of data.open.tar ?

Error info in /srv/dcc-download-server/logs/dcc-download.log

2017-02-08 17:57:31,576 [background-preinit] INFO  o.h.v.i.util.Version - HV000001: Hibernate Validator 5.2.4.Final
2017-02-08 17:57:31,662 [WrapperSimpleAppMain] INFO  o.i.d.d.s.ServerMain - Starting ServerMain on download_server with PID 1418 (/srv/dcc-download-server-4.2.18/lib/dcc-download-server.jar started by root in /srv/dcc-download-server-4.2.18/bin)
2017-02-08 17:57:31,662 [WrapperSimpleAppMain] INFO  o.i.d.d.s.ServerMain - The following profiles are active: production
2017-02-08 17:57:31,726 [WrapperSimpleAppMain] INFO  o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext - Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@4fc4c80e: startup date [Wed Feb 08 17:57:31 CST 2017]; root of context hierarchy
2017-02-08 17:57:33,130 [WrapperSimpleAppMain] INFO  o.s.b.f.s.DefaultListableBeanFactory - Overriding bean definition for bean 'beanNameViewResolver' with a different definition: replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration; factoryMethodName=beanNameViewResolver; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter; factoryMethodName=beanNameViewResolver; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter.class]]
2017-02-08 17:57:34,127 [WrapperSimpleAppMain] INFO  o.s.b.c.e.t.TomcatEmbeddedServletContainer - Tomcat initialized with port(s): 443 (https)
2017-02-08 17:57:34,148 [WrapperSimpleAppMain] INFO  o.a.c.c.StandardService - Starting service Tomcat
2017-02-08 17:57:34,149 [WrapperSimpleAppMain] INFO  o.a.c.c.StandardEngine - Starting Servlet Engine: Apache Tomcat/8.0.32
2017-02-08 17:57:35,938 [localhost-startStop-1] INFO  o.a.c.c.C.[.[.[/] - Initializing Spring embedded WebApplicationContext
2017-02-08 17:57:35,939 [localhost-startStop-1] INFO  o.s.w.c.ContextLoader - Root WebApplicationContext: initialization completed in 4212 ms
2017-02-08 17:57:36,438 [localhost-startStop-1] INFO  o.s.b.c.e.ServletRegistrationBean - Mapping servlet: 'dispatcherServlet' to [/]
2017-02-08 17:57:36,443 [localhost-startStop-1] INFO  o.s.b.c.e.FilterRegistrationBean - Mapping filter: 'metricFilter' to: [/*]
2017-02-08 17:57:36,445 [localhost-startStop-1] INFO  o.s.b.c.e.FilterRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*]
2017-02-08 17:57:36,445 [localhost-startStop-1] INFO  o.s.b.c.e.FilterRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2017-02-08 17:57:36,445 [localhost-startStop-1] INFO  o.s.b.c.e.FilterRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*]
2017-02-08 17:57:36,445 [localhost-startStop-1] INFO  o.s.b.c.e.FilterRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*]
2017-02-08 17:57:36,445 [localhost-startStop-1] INFO  o.s.b.c.e.FilterRegistrationBean - Mapping filter: 'webRequestLoggingFilter' to: [/*]
2017-02-08 17:57:36,445 [localhost-startStop-1] INFO  o.s.b.c.e.FilterRegistrationBean - Mapping filter: 'applicationContextIdFilter' to: [/*]
2017-02-08 17:57:36,936 [WrapperSimpleAppMain] WARN  o.a.h.u.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-02-08 17:57:37,742 [WrapperSimpleAppMain] INFO  o.i.d.d.s.f.PathResolver - Configuration: rootPath - /icgc/input
2017-02-08 17:57:37,758 [WrapperSimpleAppMain] INFO  o.i.d.d.s.f.DownloadFilesReader - Creating donor - download data type - data type file tables...
2017-02-08 17:57:37,941 [WrapperSimpleAppMain] INFO  o.i.d.d.s.f.DownloadFilesReader - Release paths to process: [hdfs://hdfs-01:8020/icgc/input/release_23]
2017-02-08 17:57:37,969 [WrapperSimpleAppMain] INFO  o.i.d.d.s.f.DownloadFilesReader - Creating donor - download data type - data type file table for release 'release_23'
2017-02-08 17:57:37,974 [WrapperSimpleAppMain] INFO  o.i.d.d.s.f.DownloadFilesReader - Recursively reading files for release 'release_23'
2017-02-08 17:57:40,610 [WrapperSimpleAppMain] WARN  o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'exportsController' defined in URL [jar:file:/srv/dcc-download-server-4.2.18/lib/dcc-download-server.jar!/org/icgc/dcc/download/server/endpoint/ExportsController.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.icgc.dcc.download.server.service.ExportsService]: Error creating bean with name 'exportsService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.icgc.dcc.download.server.service.FileSystemService]: Error creating bean with name 'fileSystemService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'fileSystemService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'exportsService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.icgc.dcc.download.server.service.FileSystemService]: Error creating bean with name 'fileSystemService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'fileSystemService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER
2017-02-08 17:57:40,720 [WrapperSimpleAppMain] INFO  o.a.c.c.StandardService - Stopping service Tomcat
2017-02-08 17:57:40,745 [WrapperSimpleAppMain] ERROR o.s.b.SpringApplication - Application startup failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'exportsController' defined in URL [jar:file:/srv/dcc-download-server-4.2.18/lib/dcc-download-server.jar!/org/icgc/dcc/download/server/endpoint/ExportsController.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.icgc.dcc.download.server.service.ExportsService]: Error creating bean with name 'exportsService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.icgc.dcc.download.server.service.FileSystemService]: Error creating bean with name 'fileSystemService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'fileSystemService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'exportsService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.icgc.dcc.download.server.service.FileSystemService]: Error creating bean with name 'fileSystemService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'fileSystemService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:185)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1143)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
        at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:766)
        at org.springframework.boot.SpringApplication.createAndRefreshContext(SpringApplication.java:361)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1191)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1180)
        at org.icgc.dcc.download.server.ServerMain.main(ServerMain.java:31)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:54)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'exportsService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.icgc.dcc.download.server.service.FileSystemService]: Error creating bean with name 'fileSystemService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'fileSystemService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:464)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1192)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1014)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:813)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
        ... 25 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'fileSystemService' defined in class path resource [org/icgc/dcc/download/server/config/ServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1192)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1014)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:813)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
        ... 39 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.download.server.service.FileSystemService]: Factory method 'fileSystemService' threw exception; nested exception is java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
        ... 52 common frames omitted
Caused by: java.lang.IllegalArgumentException: No enum constant org.icgc.dcc.common.core.model.DownloadDataType.BIOMARKER
        at java.lang.Enum.valueOf(Enum.java:238)
        at org.icgc.dcc.common.core.model.DownloadDataType.valueOf(DownloadDataType.java:59)
        at org.icgc.dcc.download.server.fs.DownloadFilesReader.getDataType(DownloadFilesReader.java:283)
        at org.icgc.dcc.download.server.fs.DownloadFilesReader.addFile(DownloadFilesReader.java:227)
        at org.icgc.dcc.download.server.fs.DownloadFilesReader.processPath(DownloadFilesReader.java:206)
        at org.icgc.dcc.download.server.fs.DownloadFilesReader.processPath(DownloadFilesReader.java:204)
        at org.icgc.dcc.download.server.fs.DownloadFilesReader.processPath(DownloadFilesReader.java:204)
        at org.icgc.dcc.download.server.fs.DownloadFilesReader.processPath(DownloadFilesReader.java:204)
        at org.icgc.dcc.download.server.fs.DownloadFilesReader.createReleaseCache(DownloadFilesReader.java:153)
        at org.icgc.dcc.download.server.fs.DownloadFilesReader.createReleaseFileTypes(DownloadFilesReader.java:180)
        at org.icgc.dcc.download.server.fs.DownloadFilesReader.getReleaseDonorFileTypes(DownloadFilesReader.java:79)
        at org.icgc.dcc.download.server.service.FileSystemService.<init>(FileSystemService.java:60)
        at org.icgc.dcc.download.server.config.ServiceConfig.fileSystemService(ServiceConfig.java:92)
        at org.icgc.dcc.download.server.config.ServiceConfig$$EnhancerBySpringCGLIB$$f77a8ff.CGLIB$fileSystemService$1(<generated>)
        at org.icgc.dcc.download.server.config.ServiceConfig$$EnhancerBySpringCGLIB$$f77a8ff$$FastClassBySpringCGLIB$$414ab62c.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:355)
        at org.icgc.dcc.download.server.config.ServiceConfig$$EnhancerBySpringCGLIB$$f77a8ff.fileSystemService(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
        ... 53 common frames omitted

#10

Another question is no release23 info on release history page.

link: https://icgcportal.genomics.cn/projects/history

Do i need to update portal server, current portal server version is 4.3.18 ?


#11

Updated portal-server to version 4.4.4 and download-server to version 4.3.9,all services returned to normal.

Thanks.


#12

Hi Felix,

Just as an extra note, every time a new data release occurs, we release a new version of the portal that has an updated chart for the project history page.

So when release 24 happens and you import that data, you will want to update the portal as well.

Cheers.


#13

Hi @andricDu,

Thanks for your notes.

Could please tell me how can i update portal to version 5.*, tried to update by command bin/install -r latest, but can’t start portal service ?

# all dcc-portal-server-*.jar
root@portal-server-01:/srv/dcc-portal-server/lib# ls -lh
total 360M
-rw-r--r-- 1 root root  69M Oct  9 16:28 dcc-portal-server-4.3.17.jar
-rw-r--r-- 1 root root  73M Oct 12 11:00 dcc-portal-server-4.3.18.jar
-rw-r--r-- 1 root root  71M Feb  9 13:31 dcc-portal-server-4.3.28.jar
-rw-r--r-- 1 root root  71M Feb  9 13:25 dcc-portal-server-4.4.4.jar (Currently used)
-rw-r--r-- 1 root root  78M Feb  9 11:46 dcc-portal-server-5.0.4.jar
lrwxrwxrwx 1 root root   69 Feb  9 13:26 dcc-portal-server.jar -> /srv/dcc-portal-server-4.3.6-SNAPSHOT/lib/dcc-portal-server-5.0.4.jar
-rwxrwxr-x 1 root root  41K Aug 17 23:10 libwrapper-linux-x86-64.so
-rwxrwxr-x 1 root root  83K Aug 17 23:10 libwrapper-macosx-universal-64.jnilib
-rw-rw-r-- 1 root root 118K Aug 17 23:10 wrapper.jar

Do i need to update SNAPSHOT version?

Felix


#14

Hi @felix

DCC Portal versions 5.x.x and higher are now running against Elasticsearch version 5.1.2

We are in the final stages of upgrading all of our projects to be compatible with Elasticsearch 5.
You should avoid using any newer versions of our software until your cluster has been upgraded.

We publish release notes here: http://docs.icgc.org/software/releases/portal/
Though for you, the change logs would be more informative: https://github.com/icgc-dcc/dcc-portal/blob/develop/CHANGES.md

Be aware, we try to follow the principles of semantic versioning, so if you upgrade to a major new version (A.B.C where you’ve changed the value of A) you should expect things to break without any changes on your end.


#15

Hi @andricDu,

Thanks for your information.

I will update our Elasticsearch cluster later.

:blush: