Failed to import repository.tar.gz using dcc-download-import.jar recently


#1

Hi,

I used to run a crontab script to get and import repository.tar.gz daily, it works just fine until 17th February.

The import command is:
java -jar /data/bin/dcc-download-import-4.3.9.jar -i /data/icgc_daily_index/repository.tar.gz -es es://lxv-icgc-elastic01:9300

And i made a symbol link for repository.tar.gz

-rw-r--r-- 1 root root 79461328 Feb 11 15:01 repository-2017-02-11.tar.gz
-rw-r--r-- 1 root root 79461328 Feb 12 15:01 repository-2017-02-12.tar.gz
-rw-r--r-- 1 root root 79461328 Feb 13 15:01 repository-2017-02-13.tar.gz
-rw-r--r-- 1 root root 79461328 Feb 14 15:01 repository-2017-02-14.tar.gz
-rw-r--r-- 1 root root 79461328 Feb 15 15:01 repository-2017-02-15.tar.gz
-rw-r--r-- 1 root root 79461328 Feb 16 15:01 repository-2017-02-16.tar.gz
-rw-r--r-- 1 root root 79411879 Feb 17 15:01 repository-2017-02-17.tar.gz
-rw-r--r-- 1 root root 79411299 Feb 18 15:01 repository-2017-02-18.tar.gz
-rw-r--r-- 1 root root 79419704 Feb 19 15:01 repository-2017-02-19.tar.gz
lrwxrwxrwx 1 root root       28 Feb 20 14:04 repository.tar.gz -> repository-2017-02-19.tar.gz

When running the import command above, i got these errors

root@lxv-icgc-elastic01:/data/icgc_daily_index# curl -XDELETE http://lxv-icgc-elastic01:9200/icgc-repository-20170219
{"acknowledged":true}root@lxv-icgc-elastic01:java -jar /data/bin/dcc-download-import-4.3.9.jar -i /data/icgc_daily_index/repository.tar.gz -es es://lxv-icgc-elastic01:9300 
2017-02-20 14:19:36,409 [main] INFO  o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-02-20 14:19:36,412 [main] INFO  o.i.d.d.i.ClientMain - Running with ClientOptions(inputFile=/data/icgc_daily_index/repository.tar.gz, esUrl=es://lxv-icgc-elastic01:9300, project=null, version=false, help=false)
2017-02-20 14:19:36,413 [main] INFO  o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-02-20 14:19:36,416 [main] INFO  o.i.d.d.i.c.IndexClientCommand - Creating tar reader for file /data/icgc_daily_index/repository.tar.gz
2017-02-20 14:19:36,430 [main] INFO  o.i.d.r.j.i.f.TransportClientFactory - Creating ES transport client from URI 'es://lxv-icgc-elastic01:9300': host = 'lxv-icgc-elastic01', port = 9300
2017-02-20 14:19:37,108 [main] INFO  o.i.d.d.c.e.DocumentWriterFactory - Creating ES transport client from URI 'es://lxv-icgc-elastic01:9300': host = 'lxv-icgc-elastic01', port = 9300
2017-02-20 14:19:37,356 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1219469946] Created ES document writer.
2017-02-20 14:19:37,375 [main] INFO  o.i.d.d.i.s.IndexService - Checking index 'icgc-repository-20170219' for existence...
2017-02-20 14:19:37,380 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170219'...
2017-02-20 14:19:38,035 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170219' mapping for type 'repository'...
2017-02-20 14:19:38,335 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170219' mapping for type 'file-centric'...
2017-02-20 14:19:38,610 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170219' mapping for type 'file-text'...
2017-02-20 14:19:38,775 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1219469946] Closing bulk processor...
2017-02-20 14:19:38,798 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1219469946] Finished indexing 0 documents
2017-02-20 14:19:38,800 [main] ERROR o.i.d.d.i.ClientMain - Unknown error: 
org.elasticsearch.index.mapper.MapperParsingException: No handler for type [keyword] declared on field [donor_id]
        at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:288)
        at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214)
        at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:189)
        at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:290)
        at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214)
        at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:136)
        at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:209)
        at org.elasticsearch.index.mapper.DocumentMapperParser.parseCompressed(DocumentMapperParser.java:190)
        at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:444)
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$4.execute(MetaDataMappingService.java:505)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:352)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:184)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:154)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Unknow error. Please check the log for detailed error messages: No handler for type [keyword] declared on field [donor_id]

The last usable index should be icgc-repository-20170208, which can be imported successfully.

Any ideas about what was going on, cause i can no longer open repository page of our mirror site anymore ?

Thanks!


#2

Now using index icgc-repository-20170208 for our site, waiting for suggestions.

More info:
Elasticsearch version: 1.4.4
Import tool: dcc-download-import-4.3.9.jar


#3

I get it.

You are using a new ES field type called ‘keyword’ which only available on Elasticsearch 5.x.

It seems that i really have to update es cluster.


#4

Updated ES cluster to 5.1.2 with no plugin installed.

When running command to import data, error occured

root@elastic-server-01:/source_data/release-23# java -Xmx4g -jar /data/bin/dcc-download-import-4.3.1.jar -i release.tar -es es://elastic-server-01:9300 -p LAML-CN   
2017-02-21 10:45:33,649 [main] INFO  o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-02-21 10:45:33,653 [main] INFO  o.i.d.d.i.ClientMain - Running with ClientOptions(inputFile=release.tar, esUrl=es://elastic-server-01:9300, project=LAML-CN, version=false, help=false)
2017-02-21 10:45:33,653 [main] INFO  o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-02-21 10:45:33,657 [main] INFO  o.i.d.d.i.c.IndexClientCommand - Creating tar reader for file release.tar
2017-02-21 10:45:33,832 [main] INFO  o.i.d.d.i.l.ReleaseFileLoader - Indexing file 'icgc23-9_diagram.tar.gz' into index 'icgc23-9'
2017-02-21 10:45:33,840 [main] INFO  o.i.d.r.j.i.f.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01', port = 9300
2017-02-21 10:45:34,666 [main] INFO  o.i.d.d.c.e.DocumentWriterFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01', port = 9300
2017-02-21 10:45:34,733 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1722871118] Created ES document writer.
2017-02-21 10:45:34,751 [main] INFO  o.i.d.d.i.s.IndexService - Checking index 'icgc23-9' for existence...
2017-02-21 10:45:34,754 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1722871118] Closing bulk processor...
2017-02-21 10:45:34,782 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1722871118] Finished indexing 0 documents
2017-02-21 10:45:34,784 [main] ERROR o.i.d.d.i.ClientMain - Unknown error: 
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
        at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:278)
        at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:197)
        at org.elasticsearch.client.transport.support.InternalTransportIndicesAdminClient.execute(InternalTransportIndicesAdminClient.java:86)
        at org.elasticsearch.client.support.AbstractIndicesAdminClient.exists(AbstractIndicesAdminClient.java:170)
        at org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsRequestBuilder.doExecute(IndicesExistsRequestBuilder.java:53)
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
        at org.icgc.dcc.download.imports.service.IndexService.ensureIndexNotExists(IndexService.java:85)
        at org.icgc.dcc.download.imports.service.IndexService.applySettings(IndexService.java:56)
        at org.icgc.dcc.download.imports.io.BaseTarArchiveEntryCallback.onSettings(BaseTarArchiveEntryCallback.java:60)
        at org.icgc.dcc.download.imports.io.TarArchiveDocumentReader.read(TarArchiveDocumentReader.java:82)
        at org.icgc.dcc.download.imports.load.ReleaseFileLoader.processTypeTarEntry(ReleaseFileLoader.java:95)
        at org.icgc.dcc.download.imports.load.ReleaseFileLoader.loadFile(ReleaseFileLoader.java:69)
        at org.icgc.dcc.download.imports.command.IndexClientCommand.execute(IndexClientCommand.java:49)
        at org.icgc.dcc.download.imports.ClientMain.execute(ClientMain.java:90)
        at org.icgc.dcc.download.imports.ClientMain.main(ClientMain.java:74)
Unknow error. Please check the log for detailed error messages: None of the configured nodes are available: []

ES master node settings:

root@elastic-server-01:/source_data/release-23# grep -v '^#' /etc/elasticsearch/elasticsearch.yml
cluster.name: elasticsearch
node.name: elastic-server-01
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.50.10.80 
network.publish_host: 10.50.10.80
network.bind_host: 10.50.10.80
http.port: 9200
discovery.zen.ping.unicast.hosts: ["elastic-server-02", "elastic-server-03", "elastic-server-04", "elastic-server-05"]
discovery.zen.fd.ping_timeout: 120s
discovery.zen.fd.ping_retries: 6
discovery.zen.fd.ping_interval: 30s

gateway.recover_after_nodes: 3

script:
    inline: true
    stored: true

node:
  master: true
  data: false

action:
  destructive_requires_name: true

http:
  max_initial_line_length: 1000kb
  max_chunk_size: 1000kb
  max_content_length: 1500mb
  max_header_size: 100kb

#5

The importer compatible with Elaticsearch 5 has not yet been released.


#6

Hi @felix,

Version 5.0.0 of the importer has been released and is compatible with Elasticsearch 5.1.2.


#7

Hi @andricDu,

Thanks for your updating.

I erased all ES indices,and tried to import release.tar / repository.tar.gz to ES5.1.2

Import repository.tar.gz using dcc-download-import-5.0.0.jar:

root@elastic-server-01:/data/icgc_daily_index# java -jar /data/bin/dcc-download-import-5.0.0.jar -i repository.tar.gz -es es://elastic-server-01:9300
2017-03-07 11:27:25,017 [main] INFO  o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-03-07 11:27:25,020 [main] INFO  o.i.d.d.i.ClientMain - Running with ClientOptions(inputFile=repository.tar.gz, esUrl=es://elastic-server-01:9300, project=null, version=false, help=false)
2017-03-07 11:27:25,020 [main] INFO  o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-03-07 11:27:25,022 [main] INFO  o.i.d.d.i.c.IndexClientCommand - Creating tar reader for file repository.tar.gz
2017-03-07 11:27:25,037 [main] INFO  o.i.d.d.c.e.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01/10.50.10.80', port = 9300
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
2017-03-07 11:27:26,469 [main] INFO  o.i.d.d.c.e.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01/10.50.10.80', port = 9300
2017-03-07 11:27:26,717 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1232457650] Created ES document writer.
2017-03-07 11:27:26,733 [main] INFO  o.i.d.d.i.s.IndexService - Checking index 'icgc-repository-20170306' for existence...
2017-03-07 11:27:26,751 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170306'...
2017-03-07 11:27:27,340 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170306' mapping for type 'repository'...
2017-03-07 11:27:27,372 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170306' mapping for type 'file-centric'...
2017-03-07 11:27:27,404 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170306' mapping for type 'file-text'...
2017-03-07 11:27:27,427 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc-repository-20170306' mapping for type 'donor-text'...
2017-03-07 11:27:27,589 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 1,000
2017-03-07 11:27:27,833 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 2,000
2017-03-07 11:27:27,915 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 3,000
2017-03-07 11:27:27,959 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 4,000
2017-03-07 11:27:28,005 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 5,000
2017-03-07 11:27:28,050 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 6,000
2017-03-07 11:27:28,096 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 7,000
2017-03-07 11:27:28,141 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 8,000
2017-03-07 11:27:28,191 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 9,000
2017-03-07 11:27:28,228 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 10,000
2017-03-07 11:27:28,262 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 11,000
2017-03-07 11:27:28,295 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 12,000
2017-03-07 11:27:28,328 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 13,000
2017-03-07 11:27:28,360 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 14,000
2017-03-07 11:27:28,391 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 15,000
2017-03-07 11:27:28,426 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 16,000
2017-03-07 11:27:28,463 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 17,000
2017-03-07 11:27:28,567 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 18,000
2017-03-07 11:27:28,598 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 19,000
2017-03-07 11:27:28,628 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 20,000
2017-03-07 11:27:28,672 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 21,000
2017-03-07 11:27:28,725 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 22,000
2017-03-07 11:27:28,777 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 23,000
2017-03-07 11:27:28,796 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Sending bulk request '1' with 23,281 items (37.8 MB bytes)
2017-03-07 11:27:31,646 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Successfully loaded bulk request '1'.
2017-03-07 11:27:31,681 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 24,000
2017-03-07 11:27:31,727 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 25,000
2017-03-07 11:27:31,774 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 26,000
2017-03-07 11:27:31,817 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 27,000
2017-03-07 11:27:31,858 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 28,000
2017-03-07 11:27:31,895 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 29,000
2017-03-07 11:27:31,931 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 30,000
2017-03-07 11:27:31,976 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 31,000
2017-03-07 11:27:32,024 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 32,000
2017-03-07 11:27:32,110 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 33,000
2017-03-07 11:27:32,142 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 34,000
2017-03-07 11:27:32,171 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 35,000
2017-03-07 11:27:32,200 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 36,000
2017-03-07 11:27:32,231 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 37,000
2017-03-07 11:27:32,262 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 38,000
2017-03-07 11:27:32,293 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 39,000
2017-03-07 11:27:32,323 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 40,000
2017-03-07 11:27:32,357 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 41,000
2017-03-07 11:27:32,393 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 42,000
2017-03-07 11:27:32,410 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Sending bulk request '2' with 19,238 items (37.7 MB bytes)
2017-03-07 11:27:35,056 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Successfully loaded bulk request '2'.
......
2017-03-07 11:28:19,454 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 350,000
2017-03-07 11:28:19,461 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 351,000
2017-03-07 11:28:19,467 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 352,000
2017-03-07 11:28:19,474 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 353,000
2017-03-07 11:28:19,480 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 354,000
2017-03-07 11:28:19,487 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 355,000
2017-03-07 11:28:19,493 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 356,000
2017-03-07 11:28:19,500 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 357,000
2017-03-07 11:28:19,506 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 358,000
2017-03-07 11:28:19,513 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 359,000
2017-03-07 11:28:19,519 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 360,000
2017-03-07 11:28:19,521 [main] INFO  o.i.d.d.i.l.RepositoryFileLoader - Finished indexing file repository.tar.gz in 52 seconds.
2017-03-07 11:28:19,521 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Sending bulk request '12' with 67,649 items (34.4 MB bytes)
2017-03-07 11:28:25,626 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [1232457650] Successfully loaded bulk request '12'.
2017-03-07 11:28:25,626 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1232457650] Closing bulk processor...
2017-03-07 11:28:31,635 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1232457650] Finished indexing 360,259 documents
2017-03-07 11:28:31,635 [main] INFO  o.i.d.d.i.c.IndexClientCommand - Finished processing repository.tar.gz in 66 seconds.

Import release.tar(with index icgc23-9) using dcc-download-import-5.0.0.jar:

root@elastic-server-01:/source_data/release-23# java -Xmx4g -jar /data/bin/dcc-download-import-5.0.0.jar -i release.tar -es es://elastic-server-01:9300 -p LAML-CN   
2017-03-07 10:22:13,417 [main] INFO  o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-03-07 10:22:13,421 [main] INFO  o.i.d.d.i.ClientMain - Running with ClientOptions(inputFile=release.tar, esUrl=es://elastic-server-01:9300, project=LAML-CN, version=false, help=false)
2017-03-07 10:22:13,421 [main] INFO  o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-03-07 10:22:13,423 [main] INFO  o.i.d.d.i.c.IndexClientCommand - Creating tar reader for file release.tar
2017-03-07 10:22:13,527 [main] INFO  o.i.d.d.i.l.ReleaseFileLoader - Indexing file 'icgc23-9_diagram.tar.gz' into index 'icgc23-9'
2017-03-07 10:22:13,532 [main] INFO  o.i.d.d.c.e.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01/10.50.10.80', port = 9300
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
2017-03-07 10:22:15,029 [main] INFO  o.i.d.d.c.e.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01/10.50.10.80', port = 9300
2017-03-07 10:22:15,110 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1978180753] Created ES document writer.
2017-03-07 10:22:15,127 [main] INFO  o.i.d.d.i.s.IndexService - Checking index 'icgc23-9' for existence...
2017-03-07 10:22:15,146 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc23-9'...
2017-03-07 10:22:15,238 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1978180753] Closing bulk processor...
2017-03-07 10:22:20,723 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [1978180753] Finished indexing 0 documents
2017-03-07 10:22:20,725 [main] ERROR o.i.d.d.i.ClientMain - Unknown error: 
java.lang.IllegalArgumentException: unknown setting [index.cache.query.enable] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:278)
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:246)
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:234)
        at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.onlyCreateIndex(MetaDataCreateIndexService.java:213)
        at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.createIndex(MetaDataCreateIndexService.java:193)
        at org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.masterOperation(TransportCreateIndexAction.java:81)
        at org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.masterOperation(TransportCreateIndexAction.java:39)
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:86)
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:170)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109)
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:167)
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:132)
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:103)
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:53)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173)
        at org.elasticsearch.action.ingest.IngestActionFilter.apply(IngestActionFilter.java:82)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:145)
        at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
        at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
        at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109)
        at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1343)
        at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1242)
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:536)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:490)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
        at java.lang.Thread.run(Thread.java:745)
        Suppressed: java.lang.IllegalArgumentException: unknown setting [index.store.compress.stored] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                ... 53 common frames omitted
        Suppressed: java.lang.IllegalArgumentException: unknown setting [index.store.compress.tv] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                ... 53 common frames omitted
Unknow error. Please check the log for detailed error messages: unknown setting [index.cache.query.enable] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Is release.tar file compatible with ES5.1.2 ?


#8

Hi Felix, thanks for pointing this out, it looks like we need to regenerate our export of the release index.

We will let you know when that is done.


#9

Hi Felix,

We have regenerated the release index. Please give the new files on the download server a try.

Let us know if you have any more problems.

Andy


#10

Thanks @andricDu and @ayang,

Appreciate for what you’ve done.

And @ayang, would you be so kind to provide size(in byte) and md5 info for the new release.tar file ?

We will need those information to verify the new release.tar file.


#11

Import release.tar on our testing environment, get these errors:

root@elastic-server-01:/source_data/release-23# java -Xmx4g -jar /data/bin/dcc-download-import-5.0.0.jar -i release.tar -es es://elastic-server-01:9300 -p LAML-CN
2017-03-10 16:10:23,241 [main] INFO  o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-03-10 16:10:23,245 [main] INFO  o.i.d.d.i.ClientMain - Running with ClientOptions(inputFile=release.tar, esUrl=es://elastic-server-01:9300, project=LAML-CN, version=false, help=false)
2017-03-10 16:10:23,246 [main] INFO  o.i.d.d.i.ClientMain - ----------------------------------------------------------------------------------------------------
2017-03-10 16:10:23,248 [main] INFO  o.i.d.d.i.c.IndexClientCommand - Creating tar reader for file release.tar
2017-03-10 16:10:23,350 [main] INFO  o.i.d.d.i.l.ReleaseFileLoader - Indexing file 'icgc23-34_diagram.tar.gz' into index 'icgc23-34'
2017-03-10 16:10:23,355 [main] INFO  o.i.d.d.c.e.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01/10.50.10.80', port = 9300
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
2017-03-10 16:10:24,731 [main] INFO  o.i.d.d.c.e.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01/10.50.10.80', port = 9300
2017-03-10 16:10:24,868 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [563452345] Created ES document writer.
2017-03-10 16:10:24,885 [main] INFO  o.i.d.d.i.s.IndexService - Checking index 'icgc23-34' for existence...
2017-03-10 16:10:24,900 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc23-34'...
2017-03-10 16:10:25,226 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc23-34' mapping for type 'diagram'...
2017-03-10 16:10:26,015 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [563452345] Sending bulk request '1' with 410 items (37.9 MB bytes)
2017-03-10 16:10:27,051 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [563452345] Successfully loaded bulk request '1'.
2017-03-10 16:10:27,662 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [563452345] Sending bulk request '2' with 457 items (37.9 MB bytes)
2017-03-10 16:10:28,694 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [563452345] Successfully loaded bulk request '2'.
2017-03-10 16:10:28,902 [main] INFO  o.i.d.d.i.i.BaseTarArchiveEntryCallback - Document count: 1,000
2017-03-10 16:10:29,144 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [563452345] Sending bulk request '3' with 353 items (37.8 MB bytes)
2017-03-10 16:10:30,132 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [563452345] Successfully loaded bulk request '3'.
2017-03-10 16:10:30,511 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [563452345] Sending bulk request '4' with 428 items (37.8 MB bytes)
2017-03-10 16:10:31,553 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [563452345] Successfully loaded bulk request '4'.
2017-03-10 16:10:31,935 [main] INFO  o.i.d.d.i.l.ReleaseFileLoader - Finished indexing file icgc23-34_diagram.tar.gz in 7 seconds.
2017-03-10 16:10:31,935 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [563452345] Sending bulk request '5' with 342 items (34.0 MB bytes)
2017-03-10 16:10:32,822 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [563452345] Successfully loaded bulk request '5'.
2017-03-10 16:10:32,823 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [563452345] Closing bulk processor...
2017-03-10 16:10:37,831 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [563452345] Finished indexing 1,990 documents
2017-03-10 16:10:37,832 [main] INFO  o.i.d.d.i.l.ReleaseFileLoader - Indexing file 'icgc23-34_donor-centric.tar.gz' into index 'icgc23-34'
2017-03-10 16:10:37,832 [main] INFO  o.i.d.d.c.e.TransportClientFactory - Creating ES transport client from URI 'es://elastic-server-01:9300': host = 'elastic-server-01/10.50.10.80', port = 9300
2017-03-10 16:10:37,870 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [998724994] Created ES document writer.
2017-03-10 16:10:37,871 [main] INFO  o.i.d.d.i.s.IndexService - Creating index 'icgc23-34' mapping for type 'donor-centric'...
Exception in thread "elasticsearch[_client_][generic][T#3]" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at sun.net.www.ParseUtil.encodePath(ParseUtil.java:146)
        at sun.misc.URLClassPath$JarLoader.checkResource(URLClassPath.java:922)
        at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1007)
        at sun.misc.URLClassPath.getResource(URLClassPath.java:212)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:863)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2017-03-10 17:41:28,222 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [998724994] Sending bulk request '1' with 100 items (1.3 MB bytes)
2017-03-10 17:41:29,506 [main] INFO  o.i.d.d.c.e.i.BulkProcessorListener - [998724994] Successfully loaded bulk request '1'.
2017-03-10 17:41:29,509 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [998724994] Closing bulk processor...
2017-03-10 17:41:34,535 [main] INFO  o.i.d.d.c.e.i.DefaultDocumentWriter - [998724994] Finished indexing 100 documents
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at com.fasterxml.jackson.databind.node.IntNode.valueOf(IntNode.java:46)
        at com.fasterxml.jackson.databind.node.JsonNodeFactory.numberNode(JsonNodeFactory.java:156)
        at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeAny(JsonNodeDeserializer.java:316)
        at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:231)
        at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:254)
        at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:225)
        at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:63)
        at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:14)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2993)
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:1737)
        at org.icgc.dcc.download.imports.io.TarArchiveDocumentReader.readSource(TarArchiveDocumentReader.java:116)
        at org.icgc.dcc.download.imports.io.TarArchiveDocumentReader.read(TarArchiveDocumentReader.java:90)
        at org.icgc.dcc.download.imports.load.ReleaseFileLoader.processTypeTarEntry(ReleaseFileLoader.java:95)
        at org.icgc.dcc.download.imports.load.ReleaseFileLoader.loadFile(ReleaseFileLoader.java:69)
        at org.icgc.dcc.download.imports.command.IndexClientCommand.execute(IndexClientCommand.java:49)
        at org.icgc.dcc.download.imports.ClientMain.execute(ClientMain.java:90)
        at org.icgc.dcc.download.imports.ClientMain.main(ClientMain.java:74)

Servers detail:
ecs_id Component Hostname IP CPU RAM (GB) MOUNTED DISK (GB) OS swap(GB)
i-88595blpi Elasticsearch Nodes lxv-icgc-stg2-elastic01 10.50.10.80 4 8 100 Ubuntu 14.04.5 LTS(64) 10
i-8836nxedt Elasticsearch Nodes lxv-icgc-stg2-elastic02 10.50.10.81 4 8 100 Ubuntu 14.04.5 LTS(64) 10
i-88d6ylv5b Elasticsearch Nodes lxv-icgc-stg2-elastic03 10.50.10.18 4 8 100 Ubuntu 14.04.5 LTS(64) 10
i-88ocxxap1 Elasticsearch Nodes lxv-icgc-stg2-elastic04 10.50.10.39 4 8 100 Ubuntu 14.04.5 LTS(64) 10
i-889dzxkze Elasticsearch Nodes lxv-icgc-stg2-elastic05 10.50.10.44 4 8 100 Ubuntu 14.04.5 LTS(64) 10