Uploading data problem

ETL
1

Hi all,

We have installed one elasticsearch database node in an Ubuntu 14.04 with 64Gb RAM. We have set ES_HEAP_SIZE=30g and default settings.

We have been following the steps https://github.com/icgc-dcc/dcc-download/tree/develop/dcc-download-import to use the dcc-load-importer to import some data. As a result we have now two indexes in our database:

But the icgc22-13 index hasn’t uploaded complete. We have used this command:

java -jar dcc-download-import-.jar
-i </path/to/the/elasticsearch/archive>
-es <elasticsearch_url>
-p <project_code>
To upload project code CLLE-ES but during the import the shards have turn in red and the upload has started to fail giving this error on the upload:

captura_import.png1598×226 67 KB

And this one in the database:

captura_elastic.png1599×695 152 KB

Do you know what would be the problem? Do we need to configure any more elasticsearch nodes?

Thanks for your attention,

Lorena

2

Revising the database reports I’ve seen the reported error was given before the import failed. This was really the elasticsearch log for the import failure:

[2016-10-28 14:50:06,717][DEBUG][action.bulk ] [Hobgoblin] [icgc22-13][6] failed to execute bulk item (index) index {[icgc22-13][mutation-text][MU59949570], source[{“mutation”:“chr1:g.183425464G>A”,“id”:“MU59949570”,“geneMutations”:[],“type”:“mutation”,“start”:“183425464”}]}
org.elasticsearch.index.engine.IndexFailedEngineException: [icgc22-13][6] Index failed for [mutation-text#MU59949570]
at org.elasticsearch.index.engine.internal.InternalEngine.index(InternalEngine.java:530)
at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:425)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:439)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:150)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:512)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Input/output error: _1r1_es090_0.tim
at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:189)
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
at org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:221)
at org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:152)
at org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:506)
at org.elasticsearch.common.lucene.uid.PerThreadIDAndVersionLookup.lookup(PerThreadIDAndVersionLookup.java:104)
at org.elasticsearch.common.lucene.uid.Versions.loadDocIdAndVersion(Versions.java:150)
at org.elasticsearch.common.lucene.uid.Versions.loadVersion(Versions.java:161)
at org.elasticsearch.index.engine.internal.InternalEngine.loadCurrentVersionFromIndex(InternalEngine.java:1423)
at org.elasticsearch.index.engine.internal.InternalEngine.innerIndex(InternalEngine.java:567)
at org.elasticsearch.index.engine.internal.InternalEngine.index(InternalEngine.java:522)
… 8 more
Caused by: java.io.IOException: Input/output error
at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:741)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
… 20 more

Excuse me and thanks for your help,

Lorena

3

This looks like a failed batch. Perhaps you were overwhelming the cluster. Is it repeatable?