Kafka批量消費(fèi)&逐條消費(fèi)詳解

更新時(shí)間：2026年02月06日 15:21:51 作者：C_Knight

文章介紹了Kafka消費(fèi)者的配置參數(shù),包括批量消費(fèi)和逐條消費(fèi)的設(shè)置,在逐條消費(fèi)模式下,消息會(huì)被分割成字符串?dāng)?shù)組,總結(jié)并提供了個(gè)人經(jīng)驗(yàn)供參考

Kafka批量消費(fèi)&逐條消費(fèi)

消費(fèi)者配置參數(shù)

    private Map<String, Object> defaultGoodsConsumerConfig() {
        Map<String, Object> props = Maps.newHashMap();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "ip:port");
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
        props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "100");
        props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "15000");
        props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "50");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer
");
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer
");
        props.put("listener.type", "batch");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "modify-group");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, “SASL_PLAINTEXT”);
        props.put(SaslConfigs.SASL_MECHANISM, defaultKafkaProperties.getSaslMechanism());
        props.put("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username="###" password="###";
");

        return props;
    }

    @Bean(name = "defaultListenerContainerFactory")
    public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> defaultListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory();
        factory.setConsumerFactory(new DefaultKafkaConsumerFactory(defaultGoodsConsumerConfig()));
        factory.setConcurrency(4);
        factory.setBatchListener(true);
        factory.getContainerProperties().setPollTimeout(3000);
        log.info("KafkaDefaultConsumer factory獲取實(shí)例："+ JSON.toJSONString(factory));
        return factory;
    }

       ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory();
       factory.setConsumerFactory(new DefaultKafkaConsumerFactory(defaultGoodsConsumerConfig()));
       factory.setConcurrency(4);
       //批量消費(fèi)，如果不設(shè)置默認(rèn)是單條消費(fèi)
       factory.setBatchListener(true);
       factory.getContainerProperties().setPollTimeout(3000);

消費(fèi)者監(jiān)聽(tīng)消息

    /**
     * 監(jiān)聽(tīng)goods變更消息
     */
    @KafkaListener(id="sync-modify-goods", topics = "${kafka.sync.goods.topic}", concurrency = "4", containerFactory = "defaultListenerContainerFactory")
    public void updateListener(List<ConsumerRecord<String, String>> records){
        for (ConsumerRecord<String, String> msg:records) {
            GoodsChangeMsg changeMsg = null;
            try {
                changeMsg = JSONObject.parseObject(msg.value(), GoodsChangeMsg.class);
                syncGoodsProcessor.handle(changeMsg);
            }catch (Exception exception) {
                log.error("解析失敗{}", msg, exception);
            }
        }
    }

List<ConsumerRecord<String, String>> records可以是String[] message

如果是逐條消費(fèi)，這里配置list，kafka會(huì)根據(jù)字符串中的逗號(hào)進(jìn)行分割，所以碰見(jiàn)該現(xiàn)象不要慌，看一下批量消費(fèi)的配置。