kafka消費(fèi)不到數(shù)據(jù)的排查過程
kafka消費(fèi)不到數(shù)據(jù)的排查
集群上新安裝并啟動(dòng)了3個(gè)kafka Broker，代碼打包上傳至集群，運(yùn)行后發(fā)現(xiàn)一直消費(fèi)不到數(shù)據(jù)，
本地idea中debug后發(fā)現(xiàn)，程序一直阻塞在如下程序中，陷入了死循環(huán)。
/**
* Block until the coordinator for this group is known and is ready to receive requests.
* 等待直到我們和服務(wù)端的GroupCoordinator取得連接
*/
public void ensureCoordinatorReady() {
while (coordinatorUnknown()) {//無法獲取GroupCoordinator
RequestFuture<Void> future = sendGroupCoordinatorRequest();//發(fā)送請(qǐng)求
client.poll(future);//同步等待異步調(diào)用的結(jié)果
if (future.failed()) {
if (future.isRetriable())
client.awaitMetadataUpdate();
else
throw future.exception();
} else if (coordinator != null && client.connectionFailed(coordinator)) {
// we found the coordinator, but the connection has failed, so mark
// it dead and backoff before retrying discovery
coordinatorDead();
time.sleep(retryBackoffMs);//等待一段時(shí)間，然后重試
}
}
}
流程大概說就是
- consumer會(huì)從集群中選取一個(gè)broker作為coordinator
- 然后group中的consumer會(huì)向coordinator發(fā)請(qǐng)求申請(qǐng)成為consumergroup中的leader
- 最后有1個(gè)consumer會(huì)成為consumerLeader ,其他consumer成為follower
- consumerLeader做分區(qū)分配任務(wù),同步給coordinator
- consumerFollower從coordinator同步分區(qū)分配數(shù)據(jù)
問題出現(xiàn)在第一步，意思就是說Consumer和服務(wù)端的GroupCoordinator無法取得連接，所以程序一直在等待狀態(tài)。
看了下__consumer_offsets 這個(gè)topic情況,50個(gè)分區(qū)全在broker id為152的broker上
bin/kafka-topics.sh --describe --zookeeper localhost:2182 --topic __consumer_offsets
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 152 Replicas: 152 Isr:152
Topic: __consumer_offsets Partition: 1 Leader: 152 Replicas: 152 Isr:152
Topic: __consumer_offsets Partition: 2 Leader: 152 Replicas: 152 Isr:152
Topic: __consumer_offsets Partition: 3 Leader: 152
......
但是集群上并沒有broker id為152的節(jié)點(diǎn)，想到該集群kafka節(jié)點(diǎn)曾經(jīng)添加刪除過節(jié)點(diǎn)，初步斷定152是之前的kafka節(jié)點(diǎn)，后來該節(jié)點(diǎn)去掉后又加入新的節(jié)點(diǎn)但是zookeeper中的數(shù)據(jù)并沒有更新。
所以就關(guān)閉broker，進(jìn)入zookeeper客戶端，將brokers節(jié)點(diǎn)下的topics節(jié)點(diǎn)下的__consumer_offsets刪除，然后重啟broker，注意，此時(shí)zookeeper上__consumer_offsets還并沒有生成,要開啟消費(fèi)者之后才會(huì)生成.
然后再觀察__consumer_offsets,分區(qū)已經(jīng)均勻分布在三個(gè)broker上面了
bin/kafka-topics.sh --zookeeper localhost:2182 --describe --topic __consumer_offsets
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 1 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 2 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 3 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 4 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 5 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 6 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 7 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 8 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 9 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 10 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 11 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 12 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 13 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 14 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 15 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 16 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 17 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 18 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 19 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 20 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 21 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 22 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 23 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 24 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 25 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 26 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 27 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 28 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 29 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 30 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 31 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 32 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 33 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 34 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 35 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 36 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 37 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 38 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 39 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 40 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 41 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 42 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 43 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 44 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 45 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 46 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 47 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 48 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 49 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
這個(gè)時(shí)候重啟程序，發(fā)現(xiàn)已經(jīng)可以正常消費(fèi)了，問題解決。
參考資料：
總結(jié)
以上為個(gè)人經(jīng)驗(yàn)，希望能給大家一個(gè)參考，也希望大家多多支持腳本之家。
相關(guān)文章
SpringBoot全局異常捕獲處理實(shí)現(xiàn)方案
這篇文章主要詳細(xì)介紹了SpringBoot全局異常捕獲處理實(shí)現(xiàn)方案,文章通過代碼示例給大家介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或工作有一定的幫助,需要的朋友可以參考下2024-02-02
Java.try catch finally 的執(zhí)行順序說明
這篇文章主要介紹了Java.try catch finally 的執(zhí)行順序說明，具有很好的參考價(jià)值，希望對(duì)大家有所幫助。一起跟隨小編過來看看吧2020-10-10
最大子數(shù)組和Java實(shí)現(xiàn)代碼示例
這篇文章主要介紹了最大子數(shù)組和Java實(shí)現(xiàn)的相關(guān)資料,文中介紹了兩種方法來解決尋找具有最大和的連續(xù)子數(shù)組的問題,第一種方法是動(dòng)態(tài)規(guī)劃,第二種方法是分治法,需要的朋友可以參考下2024-11-11
你必須得會(huì)的SpringBoot全局統(tǒng)一處理異常詳解
程序在運(yùn)行的過程中，不可避免會(huì)產(chǎn)生各種各樣的錯(cuò)誤，這個(gè)時(shí)候就需要進(jìn)行異常處理，本文主要為大家介紹了SpringBoot實(shí)現(xiàn)全局統(tǒng)一處理異常的方法，需要的可以參考一下2023-06-06
基于Java+SpringBoot+Vue前后端分離實(shí)現(xiàn)倉庫管理系統(tǒng)
這篇文章主要介紹了一個(gè)完整的倉庫管理系統(tǒng)是基于Java+Springboot + Vue前后端分離編寫的，本文給大家介紹的非常詳細(xì)，對(duì)大家的學(xué)習(xí)或工作具有一定的參考借鑒價(jià)值，需要的朋友可以參考下2023-06-06
解決springboot導(dǎo)入失敗,yml未識(shí)別的問題
這篇文章主要介紹了解決springboot導(dǎo)入失敗,yml未識(shí)別的問題,具有很好的參考價(jià)值,希望對(duì)大家有所幫助,如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2024-07-07
Java中Map接口使用以及有關(guān)集合的面試知識(shí)點(diǎn)匯總
在java面試過程中,Map時(shí)常會(huì)被作為一個(gè)面試點(diǎn)來問,下面這篇文章主要給大家介紹了關(guān)于Java中Map接口使用以及有關(guān)集合的面試知識(shí)點(diǎn)匯總的相關(guān)資料,文中通過實(shí)例代碼介紹的非常詳細(xì),需要的朋友可以參考下2022-07-07
MyBatisPlus3如何向數(shù)據(jù)庫中存入List
本文主要介紹了Mybatis Plus的類型處理器的使用,通過User.java和UserMapper.xml示例進(jìn)行詳細(xì)的解析,并提供了JSON解析器的使用方法,希望通過這篇文章,可以幫助大家更好的理解和掌握Mybatis Plus的類型處理器2024-10-10
這篇文章主要介紹了Java 8新特性 內(nèi)建函數(shù)式接口詳解的相關(guān)資料,這里主要介紹4個(gè)基本函數(shù)式接口，需要的朋友可以參考下2016-12-12
這篇文章主要介紹了自定義log4j.properties的加載位置方式，具有很好的參考價(jià)值，希望對(duì)大家有所幫助。如有錯(cuò)誤或未考慮完全的地方，望不吝賜教2021-12-12