kafka 修改topic partitions

张映 发表于 2019-04-29

分类目录: hadoop/spark/scala

标签:, ,

kafka topic的partitions为1时,存取的数据,只会在kafka集群中的一台机器上面,时间一长,必然会造成,资源不均衡。

1,修改topic partitions数量

[root@bigserver2 kafka]# ./bin/kafka-topics.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --alter --topic track_pc --partitions 3
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

[root@bigserver2 kafka]# bin/kafka-topics.sh --describe --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --topic track_pc
Topic:track_pc PartitionCount:3 ReplicationFactor:1 Configs:
 Topic: track_pc Partition: 0 Leader: 1 Replicas: 1 Isr: 1
 Topic: track_pc Partition: 1 Leader: 2 Replicas: 2 Isr: 2
 Topic: track_pc Partition: 2 Leader: 0 Replicas: 0 Isr: 0

分区由原来一个变成了三个,并且集群里面的所有kafka,data目录都会产生相应的文件夹

2,生成均衡配置文件

[root@bigserver2 kafka]# cat track_pc.json //创建配置文件
{
    "topics": [
        {"topic": "track_pc"}
    ],
    "version": 1
}

[root@bigserver2 kafka]# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --topics-to-move-json-file track_pc.json --broker-list "0,1,2" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"track_pc","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"track_pc","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"track_pc","partition":1,"replicas":[2],"log_dirs":["any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"track_pc","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"track_pc","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"track_pc","partition":1,"replicas":[1],"log_dirs":["any"]}]}

注意:0,1,2是kafka配置文件中的broker.id,在这里只是生成配置文件,并没有真正均衡

3,均衡partitions

将2中,生成的json,copy到一个json文件中,执行以下命令

# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --reassignment-json-file exec.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"track_pc","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"track_pc","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"track_pc","partition":1,"replicas":[2],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

[root@bigserver2 kafka]# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --reassignment-json-file exec.json --verify
Status of partition reassignment:
Reassignment of partition track_pc-0 is still in progress
Reassignment of partition track_pc-2 is still in progress
Reassignment of partition track_pc-1 is still in progress

[root@bigserver2 kafka]# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --reassignment-json-file exec.json --verify
Status of partition reassignment:
Reassignment of partition track_pc-0 completed successfully
Reassignment of partition track_pc-2 completed successfully
Reassignment of partition track_pc-1 completed successfully

注意:到这里均衡已结束,但是对已有的数据,并不能起到均衡作用。



转载请注明
作者:海底苍鹰
地址:http://blog.51yip.com/hadoop/2131.html