利用elasticdump迁移es数据

elasticdump 是一个开源免费且用于导入和导出 elasticsearch 数据的命令行工具，下面详细讲述如何利用elasticdump将某es索引从源节点迁移到目标节点，示例环境如下：

源头es索引：172.31.220.52:9201/sam_index_3.1.1
目标es节点：10.30.4.101:9200

• 1.确认要迁移的索引，比如 sam_index_3.1.1

#.源头es索引
[root@sam-es-0001 ~]# curl http://172.31.220.52:9201/_cat/indices?v
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   sam_index_3.1.1   AmE8hjmoQXqqYvyz68HJ6Q   5   1      37863         7391      3.1gb          3.1gb

#.目标es索引
[root@sam-es-0001 ~]# curl http://10.30.4.101:9200/_cat/indices?v
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   sam_index_3.3.1   d6KcUa8TT9ym4KkByVh7aQ   5   1          0            0        1kb            1kb
yellow open   test              r5Npjhg_TpafcyyuH3RZ_Q   5   1          0            0        1kb            1kb

• 2.先装 Node.js 再用npm安装 elasticdump

#.安装 node.js 16.20.2
curl -sL http://iso.sqlfans.cn/linux/install_nodejs_16202.sh | bash
npm --version

#.在线安装 elasticdump 6.110.0
# npm install -g elasticdump
# ln -sv /usr/local/nodejs/lib/node_modules/elasticdump/bin/elasticdump /usr/bin/
# elasticdump --version

#.离线安装 elasticdump 6.110.0
curl -L http://iso.sqlfans.cn/linux/elasticdump-6.110.0.zip -o /opt/elasticdump-6.110.0.zip
unzip -n -d /usr/local /opt/elasticdump-6.110.0.zip
ln -sv /usr/local/elasticdump-6.110.0/bin/elasticdump  /usr/bin/
elasticdump --version

• 3.利用 elasticdump 迁移 es 索引，比如 sam_index_3.1.1

#.迁移单个索引
  # elasticdump --input=http://172.31.220.52:9201/sam_index_3.1.1 --output=http://10.30.4.101:9200/sam_index_3.1.1 --type=settings
  # elasticdump --input=http://172.31.220.52:9201/sam_index_3.1.1 --output=http://10.30.4.101:9200/sam_index_3.1.1 --type=mapping
elasticdump --input=http://172.31.220.52:9201/sam_index_3.1.1 --output=http://10.30.4.101:9200/sam_index_3.1.1 --type=data

#.2024.04.28.实测
  1.添加 --concurrency=4 并没有太大作用
  2.添加 --limit=n 若n>200反而会报错 RangeError: Invalid string length，而n=50或100或200对导入时间影响不大

• 遇到的问题1：若es索引过大（比如超过2GB）而迁移报错，可尝试先将es索引备份成.json，然后再将此.json导入目标节点

#.备份单个索引 elasticdump --input=http://user:passwd@localhost:9200/index --output=/path/index.json --type=data
#.还原单个索引 elasticdump --input=/path/index.json --output=http://user:passwd@localhost:9200/index --type=data

• 遇到的问题2：若es索引过大（比如超过2GB）而报错 Data too large，则有可能是分配给ES的堆内存过小（默认1g）导致的，调大-Xms及-Xmx并重启es进程，再重试就好了（注:源节点与目标节点都要调大）

[root@localhost]# elasticdump --input=/opt/sam_index_3.1.1.json --output=http://127.0.0.1:9200/sam_index_3.1.1 --type=data
Sun, 28 Apr 2024 12:58:47 GMT | starting dump
Sun, 28 Apr 2024 12:58:54 GMT | got 100 objects from source file (offset: 0)
Sun, 28 Apr 2024 12:59:33 GMT | Error Emitted => {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1513523696/1.4gb], which is larger than the limit of [1020054732/972.7mb], real usage: [599126528/571.3mb]

#.注意,源节点与目标节点均要调大
[root@localhost]# elasticdump --input=http://10.30.3.231:9200/sam_index_3.1.1 --output=http://10.30.3.232:9200/sam_index_3.1.1 --type=data
Sun, 28 Apr 2024 12:58:47 GMT | starting dump
Sun, 28 Apr 2024 12:58:54 GMT | got 100 objects from source file (offset: 0)
Sun, 28 Apr 2024 14:12:23 GMT | dump ended with error (get phase) => TOO_MANY_REQUESTS: {"error":{"root_cause":[{"type":"es_rejected_execution_exception","reason":"rejected execution of coordinating operation [coordinating_and_primary_bytes=0, replica_bytes=0, all_bytes=0, coordinating_operation_bytes=183491464, max_coordinating_and_primary_bytes=107374182]"

[root@localhost]# cat /data/elasticsearch_9200/config/jvm.options | grep "^\-Xm"
-Xms1g
-Xmx1g