I am new to discussion board, not sure I am using the right topics name.
I am having problem start portal servers after I imported data. Below is what I did:
- I imported index into elastic search with command:
java -Xmx10g -jar dcc-download-import.jar -i release.tar -es es://lxv-icgc-elastic01:9300 -p LAML-CN
- untared and copied data.open.tar to hdfs with command on one of hdfs nodes:
hdfs dfs -copyFromLocal /mnt/download01/data.open-untared/release_21 /icgc/input
- After these, I restarted portal servers with command:
./bin/dcc-portal-server restart
I got the following error in portal logs. It appears to be config issue, but it is hard to pinpoint to the config issue based on the error messages.
In wrapper.log:
INFO | jvm 1 | 2016/09/01 05:19:07 | org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
INFO | jvm 1 | 2016/09/01 05:19:07 | at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:133)
INFO | jvm 1 | 2016/09/01 05:19:07 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532)
INFO | jvm 1 | 2016/09/01 05:19:07 | at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
INFO | jvm 1 | 2016/09/01 05:19:07 | at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:760)
INFO | jvm 1 | 2016/09/01 05:19:07 | at org.springframework.boot.SpringApplication.createAndRefreshContext(SpringApplication.java:360)
INFO | jvm 1 | 2016/09/01 05:19:07 | at org.springframework.boot.SpringApplication.run(SpringApplication.java:306)
INFO | jvm 1 | 2016/09/01 05:19:07 | at org.springframework.boot.SpringApplication.run(SpringApplication.java:1185)
INFO | jvm 1 | 2016/09/01 05:19:07 | at org.springframework.boot.SpringApplication.run(SpringApplication.java:1174)
INFO | jvm 1 | 2016/09/01 05:19:07 | at org.icgc.dcc.portal.server.ServerMain.main(ServerMain.java:31)
INFO | jvm 1 | 2016/09/01 05:19:07 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2016/09/01 05:19:07 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO | jvm 1 | 2016/09/01 05:19:07 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 1 | 2016/09/01 05:19:07 | at java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 1 | 2016/09/01 05:19:07 | at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:54)
INFO | jvm 1 | 2016/09/01 05:19:07 | at org.springframework.boot.loader.Launcher.launch(Launcher.java:104)
In server.log:
2016-09-01 05:18:57,356 [localhost-startStop-1] INFO c.h.c.LifecycleService - [localhost]:5701 [dcc-prod] [3.2.3] Address[localhost]:5701 is STARTED
2016-09-01 05:18:58,062 [localhost-startStop-1] WARN o.o.s.RealmVerifier - RP discovery / realm validation disabled;
2016-09-01 05:19:01,930 [localhost-startStop-1] INFO o.i.d.p.s.c.SearchConfig - Initializing Elasticsearch Transport Client with settings: {client.transport.sniff=true}
2016-09-01 05:19:02,248 [localhost-startStop-1] INFO o.e.plugins - [Jean DeWolff] loaded [], sites []
2016-09-01 05:19:06,201 [localhost-startStop-1] INFO o.i.d.p.s.r.TermsLookupRepository - Checking index 'terms-lookup' for existence...
2016-09-01 05:19:06,213 [localhost-startStop-1] INFO o.i.d.p.s.r.TermsLookupRepository - Creating index 'terms-lookup'...
2016-09-01 05:19:07,144 [localhost-startStop-1] ERROR o.s.b.c.e.t.TomcatStarter - Error starting Tomcat context: org.springframework.beans.factory.UnsatisfiedDependencyException
2016-09-01 05:19:07,227 [WrapperSimpleAppMain] WARN o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
2016-09-01 05:19:07,331 [WrapperSimpleAppMain] INFO c.h.c.LifecycleService - [localhost]:5701 [dcc-prod] [3.2.3] Address[localhost]:5701 is SHUTTING_DOWN
2016-09-01 05:19:07,435 [WrapperSimpleAppMain] INFO c.h.initializer - [localhost]:5701 [dcc-prod] [3.2.3] Destroying node initializer.
2016-09-01 05:19:07,437 [WrapperSimpleAppMain] INFO c.h.instance.Node - [localhost]:5701 [dcc-prod] [3.2.3] Hazelcast Shutdown is completed in 104 ms.
2016-09-01 05:19:07,437 [WrapperSimpleAppMain] INFO c.h.c.LifecycleService - [localhost]:5701 [dcc-prod] [3.2.3] Address[localhost]:5701 is SHUTDOWN
2016-09-01 05:19:07,454 [WrapperSimpleAppMain] ERROR o.s.b.SpringApplication - Application startup failed
org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:133)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:760)
at org.springframework.boot.SpringApplication.createAndRefreshContext(SpringApplication.java:360)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:306)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1185)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1174)
at org.icgc.dcc.portal.server.ServerMain.main(ServerMain.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:54)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:104)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:61)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'resourceConfig' defined in class path resource [org/icgc/dcc/portal/server/config/WebConfig.class]: Unsatisfied dependency expressed through constructor argument with index 1 of type [java.util.Set]: Error creating bean with name 'versionResource' defined in URL [jar:file:/srv/dcc-portal-server-4.3.6-SNAPSHOT/lib/dcc-portal-server.jar!/org/icgc/dcc/portal/server/resource/core/VersionResource.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.icgc.dcc.portal.server.model.Versions]: Error creating bean with name 'versions' defined in class path resource [org/icgc/dcc/portal/server/config/SearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.portal.server.model.Versions]: Factory method 'versions' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'releaseIndexMetadata' defined in class path resource [org/icgc/dcc/portal/server/config/SearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.Map]: Factory method 'releaseIndexMetadata' threw exception; nested exception is java.util.NoSuchElementException; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'versions' defined in class path resource [org/icgc/dcc/portal/server/config/SearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.portal.server.model.Versions]: Factory method 'versions' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'releaseIndexMetadata' defined in class path resource [org/icgc/dcc/portal/server/config/SearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.Map]: Factory method 'releaseIndexMetadata' threw exception; nested exception is java.util.NoSuchElementException; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'versionResource' defined in URL [jar:file:/srv/dcc-portal-server-4.3.6-SNAPSHOT/lib/dcc-portal-server.jar!/org/icgc/dcc/portal/server/resource/core/VersionResource.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.icgc.dcc.portal.server.model.Versions]: Error creating bean with name 'versions' defined in class path resource [org/icgc/dcc/portal/server/config/SearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.portal.server.model.Versions]: Factory method 'versions' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'releaseIndexMetadata' defined in class path resource [org/icgc/dcc/portal/server/config/SearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.Map]: Factory method 'releaseIndexMetadata' threw exception; nested exception is java.util.NoSuchElementException; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'versions' defined in class path resource [org/icgc/dcc/portal/server/config/SearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.portal.server.model.Versions]: Factory method 'versions' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'releaseIndexMetadata' defined in class path resource [org/icgc/dcc/portal/server/config/SearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.Map]: Factory method 'releaseIndexMetadata' threw exception; nested exception is java.util.NoSuchElementException
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:464)
...
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'versions' defined in class path resource [org/icgc/dcc/portal/server/config/SearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.icgc.dcc.portal.server.model.Versions]: Factory method 'versions' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'releaseIndexMetadata' defined in class path resource [org/icgc/dcc/portal/server/config/SearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.Map]: Factory method 'releaseIndexMetadata' threw exception; nested exception is java.util.NoSuchElementException
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:11
Please take a look. I can upload logs if necessary.
Thanks,
Brady