kafka实战：如何重分配分区
1、确定需要执行重分区的topic，可以指定多个，建议每次重分区一个；（重分区之前检查下topic大小）
# vi topics-to-move.json
{
"topics": [{
"topic": "test-topic"
}],
"version": 1
}2、生成分区方案
Current partition replica assignment 当前分区分布（用于回滚）
Proposed partition reassignment configuration重分区之后的分区方案（分区后的效果）
注意broker-list指定重分配的节点，执行之前先确认每个节点的broker id
不一定需要执行改步骤，也可以手动分配
#bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3,4,5" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test-topic","partition":2,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":3,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":6,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":4,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":1,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":9,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":5,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":7,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":0,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":8,"replicas":,"log_dirs":["any","any"]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test-topic","partition":4,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":7,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":1,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":9,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":6,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":3,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":8,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":0,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":5,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":2,"replicas":,"log_dirs":["any","any"]}]}手动分配分区文件如下所示：
{
"version": 1,
"partitions": [{
"topic": "testTopic",
"partition": 0,
"replicas":
}, {
"topic": "testTopic",
"partition": 1,
"replicas":
}, {
"topic": "testTopic",
"partition": 2,
"replicas":
}, {
"topic": "testTopic",
"partition": 3,
"replicas":
}, {
"topic": "testTopic",
"partition": 4,
"replicas":
}, {
"topic": "testTopic",
"partition": 5,
"replicas":
}, {
"topic": "testTopic",
"partition": 6,
"replicas":
}, {
"topic": "testTopic",
"partition": 7,
"replicas":
}, {
"topic": "testTopic",
"partition": 8,
"replicas":
}, {
"topic": "testTopic",
"partition": 9,
"replicas":
}]
}3、将生成的分区方案保存至json文件中
#vi reassignment.json
{"version":1,"partitions":[{"topic":"test-topic","partition":4,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":7,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":1,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":9,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":6,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":3,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":8,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":0,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":5,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":2,"replicas":,"log_dirs":["any","any"]}]}4、执行分区方案
# bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file reassignment.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test-topic","partition":4,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":7,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":1,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":9,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":6,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":3,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":8,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":0,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":5,"replicas":,"log_dirs":["any","any"]},{"topic":"test-topic","partition":2,"replicas":,"log_dirs":["any","any"]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.5、执行后登录Kafdrop检查分区状态
如果分区完成之后Preferred Replicas不是100%，需要手动重新为分区选举Leader
情况如下：
# cat leader.json
{
"version": 1,
"partitions": [{
"topic": "showItems",
"partition": 0
},{
"topic": "showItems",
"partition": 1
},{
"topic": "showItems",
"partition": 2
},{
"topic": "showItems",
"partition": 8
},{
"topic": "showItems",
"partition": 9
}]
}重新选举
# bin/kafka-preferred-replica-election.sh --zookeeper localhost:2181 --path-to-json-file leader.json
执行后检查分区状态是否正常
感谢分享
页:
[1]