kafka 修改topic partitions

kafka topic的partitions为1时，存取的数据，只会在kafka集群中的一台机器上面，时间一长，必然会造成，资源不均衡。

1，修改topic partitions数量

[root@bigserver2 kafka]# ./bin/kafka-topics.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --alter --topic track_pc --partitions 3
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

[root@bigserver2 kafka]# bin/kafka-topics.sh --describe --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --topic track_pc
Topic:track_pc PartitionCount:3 ReplicationFactor:1 Configs:
 Topic: track_pc Partition: 0 Leader: 1 Replicas: 1 Isr: 1
 Topic: track_pc Partition: 1 Leader: 2 Replicas: 2 Isr: 2
 Topic: track_pc Partition: 2 Leader: 0 Replicas: 0 Isr: 0

分区由原来一个变成了三个，并且集群里面的所有kafka，data目录都会产生相应的文件夹

2，生成均衡配置文件

[root@bigserver2 kafka]# cat track_pc.json //创建配置文件
{
    "topics": [
        {"topic": "track_pc"}
    ],
    "version": 1
}

[root@bigserver2 kafka]# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --topics-to-move-json-file track_pc.json --broker-list "0,1,2" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"track_pc","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"track_pc","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"track_pc","partition":1,"replicas":[2],"log_dirs":["any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"track_pc","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"track_pc","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"track_pc","partition":1,"replicas":[1],"log_dirs":["any"]}]}

注意：0，1，2是kafka配置文件中的broker.id，在这里只是生成配置文件，并没有真正均衡

3，均衡partitions

将2中，生成的json，copy到一个json文件中，执行以下命令

# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --reassignment-json-file exec.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"track_pc","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"track_pc","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"track_pc","partition":1,"replicas":[2],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

[root@bigserver2 kafka]# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --reassignment-json-file exec.json --verify
Status of partition reassignment:
Reassignment of partition track_pc-0 is still in progress
Reassignment of partition track_pc-2 is still in progress
Reassignment of partition track_pc-1 is still in progress

[root@bigserver2 kafka]# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --reassignment-json-file exec.json --verify
Status of partition reassignment:
Reassignment of partition track_pc-0 completed successfully
Reassignment of partition track_pc-2 completed successfully
Reassignment of partition track_pc-1 completed successfully

注意：到这里均衡已结束，但是对已有的数据，并不能起到均衡作用。