ES_CPU飚高

一、第一次CPU飚高

1）执行命令查看线程

$ GET _nodes/hot_threads

hot_threads_1.txt

2）关键堆栈：

org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.elasticsearch.index.store.ByteSizeCachingDirectory.estimateSizeInBytes(ByteSizeCachingDirectory.java:55)       org.elasticsearch.index.store.ByteSizeCachingDirectory.access$200(ByteSizeCachingDirectory.java:36)
org.elasticsearch.index.store.ByteSizeCachingDirectory$1.refresh(ByteSizeCachingDirectory.java:89)       org.elasticsearch.index.store.ByteSizeCachingDirectory$1.refresh(ByteSizeCachingDirectory.java:71)       org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.ByteSizeCachingDirectory.estimateSizeInBytes(ByteSizeCachingDirectory.java:120)
org.elasticsearch.index.store.Store$StoreDirectory.estimateSize(Store.java:723)
org.elasticsearch.index.store.Store.stats(Store.java:362)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:971)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:180)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:178)       org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:48)

3）结论：

GET /indices/_stats命令导致CPU飚高。

https://github.com/elastic/elasticsearch/issues/19313

look at this hot thread stack, it seem like you request Get /indices/_stats request to es, which would sum all index files size. And this is very expensive, for this request will make many linux system call.
If you have many index, and the index’s file num is big. Get /indices/_stats will led the node’s cpu usage high.

I had test this situation in my ES cluster, if have more than 100 index, and send Get /indices/_stats every second, the cpu usage will keep 80% above.

二、第二次CPU飚高

1）执行命令查看线程

$ GET _nodes/hot_threads

hot_threads_2.txt

2）关键堆栈

org.wltea.analyzer.lucene.IKTokenizer.incrementToken(IKTokenizer.java:88)
org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:787)
org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430)
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1609)
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1228)
org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1125)
org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1070)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:897)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:772)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:741)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:705)
org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:461)
org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$2609/1493261797.get(Unknown Source)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:483)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:459)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79)

3）结论

设置了IK_smart分词的字段，CPU高时写入的document该字段长度过大，导致CPU飚高。