Hi @andricDu,
Thanks for your updating.
I erased all ES indices,and tried to import release.tar / repository.tar.gz to ES5.1.2
Import repository.tar.gz
using dcc-download-import-5.0.0.jar:
root@elastic-server-01:/data/icgc_daily_index# java -jar /data/bin/dcc-download-import-5.0.0.jar -i repository.tar.gz -es es://elastic-server-01:9300
2017-03-07 11:27:25,017 [main] INFO o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-03-07 11:27:25,020 [main] INFO o.i.d.d.i.ClientMain - Running with ClientOptions(inputFile=repository.tar.gz, esUrl=es://elastic-server-01:9300, project=null, version=false, help=false)
2017-03-07 11:27:25,020 [main] INFO o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-03-07 11:27:25,022 [main] INFO o.i.d.d.i.c.IndexClientCommand - Creating tar reader for file repository.tar.gz
2017-03-07 11:27:25,037 [main] INFO o.i.d.d.c.e.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01/10.50.10.80', port = 9300
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
2017-03-07 11:27:26,469 [main] INFO o.i.d.d.c.e.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01/10.50.10.80', port = 9300
2017-03-07 11:27:26,717 [main] INFO o.i.d.d.c.e.i.DefaultDocumentWriter - [1232457650] Created ES document writer.
2017-03-07 11:27:26,733 [main] INFO o.i.d.d.i.s.IndexService - Checking index 'icgc-repository-20170306' for existence...
2017-03-07 11:27:26,751 [main] INFO o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170306'...
2017-03-07 11:27:27,340 [main] INFO o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170306' mapping for type 'repository'...
2017-03-07 11:27:27,372 [main] INFO o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170306' mapping for type 'file-centric'...
2017-03-07 11:27:27,404 [main] INFO o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170306' mapping for type 'file-text'...
2017-03-07 11:27:27,427 [main] INFO o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170306' mapping for type 'donor-text'...
2017-03-07 11:27:27,589 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 1,000
2017-03-07 11:27:27,833 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 2,000
2017-03-07 11:27:27,915 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 3,000
2017-03-07 11:27:27,959 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 4,000
2017-03-07 11:27:28,005 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 5,000
2017-03-07 11:27:28,050 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 6,000
2017-03-07 11:27:28,096 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 7,000
2017-03-07 11:27:28,141 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 8,000
2017-03-07 11:27:28,191 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 9,000
2017-03-07 11:27:28,228 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 10,000
2017-03-07 11:27:28,262 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 11,000
2017-03-07 11:27:28,295 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 12,000
2017-03-07 11:27:28,328 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 13,000
2017-03-07 11:27:28,360 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 14,000
2017-03-07 11:27:28,391 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 15,000
2017-03-07 11:27:28,426 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 16,000
2017-03-07 11:27:28,463 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 17,000
2017-03-07 11:27:28,567 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 18,000
2017-03-07 11:27:28,598 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 19,000
2017-03-07 11:27:28,628 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 20,000
2017-03-07 11:27:28,672 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 21,000
2017-03-07 11:27:28,725 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 22,000
2017-03-07 11:27:28,777 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 23,000
2017-03-07 11:27:28,796 [main] INFO o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Sending bulk request '1' with 23,281 items (37.8 MB bytes)
2017-03-07 11:27:31,646 [main] INFO o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Successfully loaded bulk request '1'.
2017-03-07 11:27:31,681 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 24,000
2017-03-07 11:27:31,727 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 25,000
2017-03-07 11:27:31,774 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 26,000
2017-03-07 11:27:31,817 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 27,000
2017-03-07 11:27:31,858 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 28,000
2017-03-07 11:27:31,895 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 29,000
2017-03-07 11:27:31,931 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 30,000
2017-03-07 11:27:31,976 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 31,000
2017-03-07 11:27:32,024 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 32,000
2017-03-07 11:27:32,110 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 33,000
2017-03-07 11:27:32,142 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 34,000
2017-03-07 11:27:32,171 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 35,000
2017-03-07 11:27:32,200 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 36,000
2017-03-07 11:27:32,231 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 37,000
2017-03-07 11:27:32,262 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 38,000
2017-03-07 11:27:32,293 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 39,000
2017-03-07 11:27:32,323 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 40,000
2017-03-07 11:27:32,357 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 41,000
2017-03-07 11:27:32,393 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 42,000
2017-03-07 11:27:32,410 [main] INFO o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Sending bulk request '2' with 19,238 items (37.7 MB bytes)
2017-03-07 11:27:35,056 [main] INFO o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Successfully loaded bulk request '2'.
......
2017-03-07 11:28:19,454 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 350,000
2017-03-07 11:28:19,461 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 351,000
2017-03-07 11:28:19,467 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 352,000
2017-03-07 11:28:19,474 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 353,000
2017-03-07 11:28:19,480 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 354,000
2017-03-07 11:28:19,487 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 355,000
2017-03-07 11:28:19,493 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 356,000
2017-03-07 11:28:19,500 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 357,000
2017-03-07 11:28:19,506 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 358,000
2017-03-07 11:28:19,513 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 359,000
2017-03-07 11:28:19,519 [main] INFO o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 360,000
2017-03-07 11:28:19,521 [main] INFO o.i.d.d.i.l.RepositoryFileLoader - Finished indexing file repository.tar.gz in 52 seconds.
2017-03-07 11:28:19,521 [main] INFO o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Sending bulk request '12' with 67,649 items (34.4 MB bytes)
2017-03-07 11:28:25,626 [main] INFO o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Successfully loaded bulk request '12'.
2017-03-07 11:28:25,626 [main] INFO o.i.d.d.c.e.i.DefaultDocumentWriter - [1232457650] Closing bulk processor...
2017-03-07 11:28:31,635 [main] INFO o.i.d.d.c.e.i.DefaultDocumentWriter - [1232457650] Finished indexing 360,259 documents
2017-03-07 11:28:31,635 [main] INFO o.i.d.d.i.c.IndexClientCommand - Finished processing repository.tar.gz in 66 seconds.
Import release.tar(with index icgc23-9) using dcc-download-import-5.0.0.jar:
root@elastic-server-01:/source_data/release-23# java -Xmx4g -jar /data/bin/dcc-download-import-5.0.0.jar -i release.tar -es es://elastic-server-01:9300 -p LAML-CN
2017-03-07 10:22:13,417 [main] INFO o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-03-07 10:22:13,421 [main] INFO o.i.d.d.i.ClientMain - Running with ClientOptions(inputFile=release.tar, esUrl=es://elastic-server-01:9300, project=LAML-CN, version=false, help=false)
2017-03-07 10:22:13,421 [main] INFO o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-03-07 10:22:13,423 [main] INFO o.i.d.d.i.c.IndexClientCommand - Creating tar reader for file release.tar
2017-03-07 10:22:13,527 [main] INFO o.i.d.d.i.l.ReleaseFileLoader - Indexing file 'icgc23-9_diagram.tar.gz' into index 'icgc23-9'
2017-03-07 10:22:13,532 [main] INFO o.i.d.d.c.e.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01/10.50.10.80', port = 9300
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
2017-03-07 10:22:15,029 [main] INFO o.i.d.d.c.e.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01/10.50.10.80', port = 9300
2017-03-07 10:22:15,110 [main] INFO o.i.d.d.c.e.i.DefaultDocumentWriter - [1978180753] Created ES document writer.
2017-03-07 10:22:15,127 [main] INFO o.i.d.d.i.s.IndexService - Checking index 'icgc23-9' for existence...
2017-03-07 10:22:15,146 [main] INFO o.i.d.d.i.s.IndexService - Creating index 'icgc23-9'...
2017-03-07 10:22:15,238 [main] INFO o.i.d.d.c.e.i.DefaultDocumentWriter - [1978180753] Closing bulk processor...
2017-03-07 10:22:20,723 [main] INFO o.i.d.d.c.e.i.DefaultDocumentWriter - [1978180753] Finished indexing 0 documents
2017-03-07 10:22:20,725 [main] ERROR o.i.d.d.i.ClientMain - Unknown error:
java.lang.IllegalArgumentException: unknown setting [index.cache.query.enable] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:278)
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:246)
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:234)
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.onlyCreateIndex(MetaDataCreateIndexService.java:213)
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.createIndex(MetaDataCreateIndexService.java:193)
at org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.masterOperation(TransportCreateIndexAction.java:81)
at org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.masterOperation(TransportCreateIndexAction.java:39)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:86)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:170)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:167)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:132)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:103)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:53)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173)
at org.elasticsearch.action.ingest.IngestActionFilter.apply(IngestActionFilter.java:82)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:145)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1343)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1242)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:536)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:490)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
at java.lang.Thread.run(Thread.java:745)
Suppressed: java.lang.IllegalArgumentException: unknown setting [index.store.compress.stored] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
... 53 common frames omitted
Suppressed: java.lang.IllegalArgumentException: unknown setting [index.store.compress.tv] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
... 53 common frames omitted
Unknow error. Please check the log for detailed error messages: unknown setting [index.cache.query.enable] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
Is release.tar
file compatible with ES5.1.2 ?