Docker部署Logstash同步Mysql數(shù)據(jù)到ES方式

更新時間：2024年10月23日 14:09:19 作者：寫....寫個大西瓜

本文介紹了如何搭建ELK（Elasticsearch, Logstash, Kibana）日志系統(tǒng),包括配置文件準備、Logstash與Elasticsearch部署、驅(qū)動文件準備、配置文件編輯、容器操作以及日志查看等步驟,文章內(nèi)容是作者的個人經(jīng)驗分享

Docker部署Logstash同步Mysql數(shù)據(jù)到ES

1、準備配置文件文件夾

2、部署logstash & elasticsearch

docker pull docker.elastic.co/logstash/logstash:7.15.0
## 替換{你的ES地址}為ES地址
docker run -d --name logstash -p 5044:5044 -p 9600:9600 -v D:\logstash\data\:/usr/share/logstash/data  -v D:\logstash\path\to\mysql-connector-j-8.0.33.jar:/usr/share/logstash/mysql-connector-j-8.0.33.jar -v D:\logstash\config\:/usr/share/logstash/config -v D:\logstash\pipeline\:/usr/share/logstash/pipeline -e "ELASTICSEARCH_HOSTS={你的ES地址}" docker.elastic.co/logstash/logstash:7.15.0

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.15.0
## 配置文件如果需要修改，修改完后刪除重新啟動容器
docker run -d --name elasticsearch -v D:\elasticsearch\config\:/usr/share/elasticsearch/config -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.15.0

頁面訪問 ip:9200端口，出現(xiàn)下面頁面部署成功

3、準備驅(qū)動文件

4、編輯配置文件

input {
  jdbc {
    jdbc_driver_library => "/usr/share/logstash/mysql-connector-j-8.0.33.jar"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://{你的MYSQL地址}/{你的數(shù)據(jù)庫}"
    jdbc_user => "{數(shù)據(jù)庫賬號}"
    jdbc_password => "{數(shù)據(jù)庫密碼}"
    statement => "SELECT * FROM user"
    jdbc_paging_enabled => true		 # 開啟分頁
    jdbc_page_size => 2000			 # 分頁每頁條數(shù)
    schedule => "*/5 * * * * * UTC"  # 每5s執(zhí)行一次
  }
}
output {
  elasticsearch {
      hosts  => ["{你的ES地址}"]         				# ElasticSearch 的地址和端口
      index  => "user"         							# 指定索引名，如果不存在es會自動創(chuàng)建
      document_id => "%{id}"
      codec  => "json"
  }
}

5、刪除容器、重新啟動

docker rm -f logstash
docker run -d --name logstash -p 5044:5044 -p 9600:9600 -v D:\logstash\data\:/usr/share/logstash/data  -v D:\logstash\path\to\mysql-connector-j-8.0.33.jar:/usr/share/logstash/mysql-connector-j-8.0.33.jar -v D:\logstash\config\:/usr/share/logstash/config -v D:\logstash\pipeline\:/usr/share/logstash/pipeline -e "ELASTICSEARCH_HOSTS={你的ES地址}" docker.elastic.co/logstash/logstash:7.15.0