clickhouse?批量插入數(shù)據(jù)及ClickHouse常用命令詳解
一.安裝使用
ClickHouse是Yandex提供的一個(gè)開源的列式存儲(chǔ)數(shù)據(jù)庫管理系統(tǒng),多用于聯(lián)機(jī)分析(OLAP)場(chǎng)景,可提供海量數(shù)據(jù)的存儲(chǔ)和分析,同時(shí)利用其數(shù)據(jù)壓縮和向量化引擎的特性,能提供快速的數(shù)據(jù)搜索。

Ⅰ).安裝
sudo yum install yum-utils sudo rpm --import https://repo.yandex.ru/clickhouse/CLICKHOUSE-KEY.GPG sudo yum-config-manager --add-repo https://repo.yandex.ru/clickhouse/rpm/stable/x86_64 sudo yum install clickhouse-server clickhouse-client sudo /etc/init.d/clickhouse-server start clickhouse-client
Ⅱ).配置
a).clickhouse-server
CLICKHOUSE_USER=username
CLICKHOUSE_LOGDIR=${CLICKHOUSE_HOME}/log/clickhoue-server
CLICKHOUSE_LOGDIR_USER=username
CLICKHOUSE_DATADIR_OLD=${CLICKHOUSE_HOME}/data/old
CLICKHOUSE_DATADIR=${CLICKHOUSE_HOME}/datab).config.xml
... ...
<!-- 配置日志參數(shù) -->
<logger>
<level>info</level>
<log>${CLICKHOUSE_HOME}/log/clickhoue-server/clickhoue-server.log</log>
<errorlog>${CLICKHOUSE_HOME}/log/clickhoue-server/clickhoue-server-error.log</errorlog>
<size>100M</size>
<count>5</count>
</logger>
<!-- 配置數(shù)據(jù)保存路徑 -->
<path>${CLICKHOUSE_HOME}</>
<tmp_path>${CLICKHOUSE_HOME}/tmp</>
<user_files_path>${CLICKHOUSE_HOME}/user_files</>
<!-- 配置監(jiān)聽 -->
<listen_host>::</listen_host>
<!-- 配置時(shí)區(qū) -->
<timezone>Asiz/Shanghai</timezone>
... ...Ⅲ).啟停服務(wù)
#### a).啟動(dòng)服務(wù) sudo service clickhouse-server start #### b).停止服務(wù) sudo service clickhouse-server stop
Ⅳ).客戶端訪問
clickhouse-client
二.常用命令
Ⅰ).創(chuàng)建表
CREATE TABLE IF NOT EXISTS database.table_name ON cluster cluster_shardNum_replicasNum(
'id' UInt64,
'name' String,
'time' UInt64,
'age' UInt8,
'flag' UInt8
)
ENGINE = MergeTree
PARTITION BY toDate(time/1000)
ORDER BY (id,name)
SETTINGS index_granularity = 8192Ⅱ).創(chuàng)建物化視圖
CREATE MATERIALIZED VIEW database.view_name ON cluster cluster_shardNum_replicasNum
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMMDD(time)
ORDER BY (id,name)
AS SELECT
toStartOfHour(toDateTime(time/1000)) as time,
id,
name,
sumState( if (flag = 1, 1, 0)) AS successCount,
sumState( if (flag = 0, 1, 0)) AS faildCount,
sumState( if ((age < 10), 1, 0)) AS rang1Age,
sumState( if ((age > 10) AND (age < 20), 2, 0)) AS rang2Age,
sumState( if ((age > 20), 3, 0)) AS rang3Age,
maxState(age) AS maxAge,
minState(age) AS minAge
FROM datasource.table_name
GROUP BY time,id,nameⅢ).插入數(shù)據(jù)
a).普通數(shù)據(jù)插入
INSERT INTO database.table_name(id, name, age, flag) VALUES(1, 'test', 15, 0)
b).Json數(shù)據(jù)插入
INSERT INTO database.table_name FORMAT JSONEachRow{"id":"1", "name":"test", "age":"11", "flag":"1"}Ⅳ).查詢數(shù)據(jù)
a).表數(shù)據(jù)查詢
SELECT * FROM database.table_name WHERE id=1
b).物化視圖查詢
SELECT id, name, sumMerge(successCount), sumMerge(faildCount), sumMerge(rang1Age), sumMerge(rang2Age), maxMerge(maxAge), minMerge(minAge) FROM database.view_name WHERE id=1 GROUP BY id, name
Ⅴ).創(chuàng)建NESTED表
CREATE TABLE IF NOT EXISTS database.table_name( 'id' UInt64, 'name' String, 'time' UInt64, 'age' UInt8, 'flag' UInt8 nested_table_name Nested ( sequence UInt32, id UInt64, name String, time UInt64, age UInt8, flag UInt8 socketAddr String, socketRemotePort UInt32, socketLocalPort UInt32, eventTime UInt64, exceptionClassName String, hashCode Int32, nextSpanId UInt64 )) ENGINE = MergeTree PARTITION BY toDate (time / 1000) ORDER BY (id, name, time) SETTINGS index_granularity = 8192
Ⅵ).NESTED表數(shù)據(jù)查詢
SELECT table1.*,table1.id FROM nest.table_name AS table1 array JOIN nested_table_name AS table2
Ⅶ).配置字典項(xiàng)
<dictionaries>
<dictionary>
<name>url</name>
<source>
<clickhouse>
<host>hostname</host>
<port>9000</port>
<user>default</user>
<password/>
<db>dict</db>
<table>url_dict</table>
</clickhouse>
</source>
<lifetime>
<min>30</min>
<max>36</max>
</lifetime>
<layout>
<hashed/>
</layout>
<structure>
<id>
<name>id</name>
</id>
<attribute>
<name>hash_code</name>
<type>String</type>
<null_value/>
</attribute>
<attribute>
<name>url</name>
<type>String</type>
<null_value/>
</attribute>
</structure>
</dictionary>
<dictionary>
<name>url_hash</name>
<source>
<clickhouse>
<host>hostname</host>
<port>9000</port>
<user>default</user>
<password/>
<db>dict</db>
<table>url_hash</table>
</clickhouse>
</source>
<lifetime>
<min>30</min>
<max>36</max>
</lifetime>
<layout>
<complex_key_hashed/>
</layout>
<structure>
<key>
<attribute>
<name>hash_code</name>
<type>String</type>
</attribute>
</key>
<attribute>
<name>url</name>
<type>String</type>
<null_value/>
</attribute>
</structure>
</dictionary>
</dictionaries>Ⅷ).字典查詢
SELECT
id,
dictGet('name', 'name', toUInt64(name)) AS name,
dictGetString('url', 'url', tuple(url)) AS url
FROM table_nameⅨ).導(dǎo)入數(shù)據(jù)
clickhouse-client --query="INSERT INTO database.table_name FORMAT CSVWithNames" < /path/import_filename.csv
Ⅹ).導(dǎo)出數(shù)據(jù)
clickhouse-client --query="SELECT * FROM database.table_name FORMAT CSV" sed 's/"http://g' > /path/export_filename.csv
Ⅺ).查看partition狀態(tài)
SELECT table, name, partition,active FROM system.parts WHERE database='database_name'
Ⅻ).清理partition
ALTER TABLE database.table_name ON cluster cluster_shardNum_replicasNum detach partition 'partition_id'
XIII).查看列的壓縮率
SELECT
database,
table,
name,
formatReadableSize(sum(data_compressed_bytes) AS c) AS comp,
formatReadableSize(sum(data_uncompressed_bytes) AS r) AS raw,
c/r AS comp_ratio
FROM system.columns
WHERE database='database_name'
AND table='table_name'
GROUP BY nameXIV).查看物化視圖的磁盤占用
clickhouse-client --query="SELECT partition,count(*) AS partition_num, formatReadableSize(sum(bytes)) AS disk_size FROM system.columns WHERE database='database_name' " --external --?le=***.sql --name=parts --structure='table String, name String, partition UInt64, engine String' -h hostname
到此這篇關(guān)于clickhouse 批量插入數(shù)據(jù)及ClickHouse常用命令的文章就介紹到這了,更多相關(guān)clickhouse 批量插入內(nèi)容請(qǐng)搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家!
相關(guān)文章
SpringSceurity實(shí)現(xiàn)短信驗(yàn)證碼登陸
這篇文章主要介紹了SpringSceurity實(shí)現(xiàn)短信驗(yàn)證碼登陸,本文通過實(shí)例代碼給大家介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或工作具有一定的參考借鑒價(jià)值,需要的朋友可以參考下2020-06-06
springboot3.4和mybatis plus的版本問題的解決
本文主要介紹了springboot3.4和mybatis plus的版本問題的解決,主要由于Spring Boot 3.4與MyBatis-Plus版本不匹配導(dǎo)致分頁功能問題,下面就來解決這個(gè)問題,感興趣的可以了解一下2025-03-03
springcloud集成nacos?使用lb?無效問題解決方案
這篇文章主要介紹了解決springcloud集成nacos?使用lb?無效,通過查看spring-cloud-starter-gateway?jar中的自動(dòng)配置類的源碼,得知,該jar包中是不支持負(fù)載均衡的,需要引入spring-cloud-starter-loadbalancer?來支持,需要的朋友可以參考下2023-04-04
詳解在Spring Boot框架下使用WebSocket實(shí)現(xiàn)消息推送
這篇文章主要介紹了詳解在Spring Boot框架下使用WebSocket實(shí)現(xiàn)消息推送,具有一定的參考價(jià)值,感興趣的小伙伴們可以參考一下。2016-12-12
實(shí)例分析Java中public static void main(String args[])是什么意思
這篇文章主要介紹了實(shí)例分析Java中public static void main(String args[])的意義,詳細(xì)分析了Java主函數(shù)main關(guān)鍵字聲明的具體含義和用法,需要的朋友可以參考下2015-12-12
Java設(shè)計(jì)模式之Prototype原型模式
這篇文章主要為大家詳細(xì)介紹了Java設(shè)計(jì)模式之Prototype原型模式的相關(guān)資料,文中示例代碼介紹的非常詳細(xì),具有一定的參考價(jià)值,感興趣的小伙伴們可以參考一下2018-03-03

