ElasticSearch 集群部署
概述
我们将ES的服务部署和同步服务部署分为两套,因为后期我们数据同步完成可以将异构数据同步的服务停止掉
服务布局
我们整体采用Docker方式进行布局,以下是我们需要部署的服务
服务名称 |
服务名称 |
开放端口 |
内存限制 |
ES-node1 |
node-1 |
9200 |
1G |
ES-node2 |
node-2 |
9201 |
1G |
ES-node3 |
node-3 |
9202 |
1G |
ES-cerebro |
cerebro |
9100 |
不限 |
kibana |
kibana |
5601 |
不限 |
准备工作
创建挂载目录
1 2 3 4 5 6
| #创建配置目录 mkdir -p /tmp/etc/elasticsearch/node-{1..3}/{config,plugins} # 创建kibana的配置目录 mkdir -p /tmp/data/kibana/config # 创建数据目录 mkdir -p /tmp/data/elasticsearch/node-{1..3}/{data,log}
|
目录授权
1 2 3
| chmod 777 /tmp/etc/elasticsearch/node-{1..3}/{config,plugins} chmod 777 /tmp/etc/kibana/config chmod 777 /tmp/data/elasticsearch/node-{1..3}/{data,log}
|
修改Linux句柄数
查看当前最大句柄数
1
| sysctl -a | grep vm.max_map_count
|
修改句柄数
1
| + vm.max_map_count=262144
|
生效配置
修改后需要重启才能生效,不想重启可以设置临时生效
1
| sysctl -w vm.max_map_count=262144
|
修改句柄数和最大线程数
配置修改
修改后需要重新登录生效
1
| vi /etc/security/limits.conf
|
1 2 3 4 5
| # 添加以下内容 * soft nofile 65536 * hard nofile 65536 * soft nproc 4096 * hard nproc 4096
|
重启服务
添加IK分词器
因为后面要用到IK分词,所以我们要安装以下IK分词器
查找
在github中下载对应版本的分词器
1
| https://github.com/medcl/elasticsearch-analysis-ik/releases
|
根据自己的ES版本选择相应版本的IK分词器,因为安装的ES是7.17.5
,所以也下载相应的IK分词器
解压
将下载的分词器复制到ES安装目录的plugins
目录中并进行解压
1 2
| mkdir ik && cd ik unzip elasticsearch-analysis-ik-7.17.5.zip
|
安装
1 2 3
| cp -R ik/ /tmp/etc/elasticsearch/node-1/plugins/ cp -R ik/ /tmp/etc/elasticsearch/node-2/plugins/ cp -R ik/ /tmp/etc/elasticsearch/node-3/plugins/
|
编写配置文件
node-1
1
| vi /tmp/data/elasticsearch/node-1/config/elasticsearch.yml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
| cluster.name: elastic
node.name: node-1
node.master: true
node.data: true
node.max_local_storage_nodes: 3
path.data: /usr/share/elasticsearch/data
path.logs: /usr/share/elasticsearch/log
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
discovery.seed_hosts: ["node-1","node-2","node-3"]
cluster.initial_master_nodes: ["node-1","node-2","node-3"]
gateway.recover_after_nodes: 2
xpack.security.enabled: false
|
node-2
1
| vi /tmp/data/elasticsearch/node-2/config/elasticsearch.yml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
| cluster.name: elastic
node.name: node-2
node.master: true
node.data: true
node.max_local_storage_nodes: 3
path.data: /usr/share/elasticsearch/data
path.logs: /usr/share/elasticsearch/log
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
discovery.seed_hosts: ["node-1","node-2","node-3"]
cluster.initial_master_nodes: ["node-1","node-2","node-3"]
gateway.recover_after_nodes: 2
xpack.security.enabled: false
|
node-3
1
| vi /tmp/data/elasticsearch/node-3/config/elasticsearch.yml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
| cluster.name: elastic
node.name: node-3
node.master: true
node.data: true
node.max_local_storage_nodes: 3
path.data: /usr/share/elasticsearch/data
path.logs: /usr/share/elasticsearch/log
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
discovery.seed_hosts: ["node-1","node-2","node-3"]
cluster.initial_master_nodes: ["node-1","node-2","node-3"]
gateway.recover_after_nodes: 2
xpack.security.enabled: false
|
kibana
1
| vi /tmp/data/kibana/config/kibana.yml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
| server.host: 0.0.0.0
server.port: 5601 server.name: "kibana"
elasticsearch.hosts: ["http://node-1:9200","http://node-2:9201","http://node-3:9202"] monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.username: "kibana" elasticsearch.password: "12345"
logging.dest: stdout
logging.silent: false
logging.quiet: false
logging.verbose: false
ops.interval: 5000
i18n.locale: "zh-CN"
|
编写部署文档
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101
| version: "3" services: node-1: image: elasticsearch:7.17.5 container_name: node-1 environment: - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" - "TZ=Asia/Shanghai" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 ports: - "9200:9200" logging: driver: "json-file" options: max-size: "50m" volumes: - /tmp/etc/elasticsearch/node-1/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - /tmp/etc/elasticsearch/node-1/plugins:/usr/share/elasticsearch/plugins - /tmp/data/elasticsearch/node-1/data:/usr/share/elasticsearch/data - /tmp/data/elasticsearch/node-1/log:/usr/share/elasticsearch/log networks: - elastic node-2: image: elasticsearch:7.17.5 container_name: node-2 environment: - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" - "TZ=Asia/Shanghai" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 ports: - "9201:9200" logging: driver: "json-file" options: max-size: "50m" volumes: - /tmp/etc/elasticsearch/node-2/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - /tmp/etc/elasticsearch/node-2/plugins:/usr/share/elasticsearch/plugins - /tmp/data/elasticsearch/node-2/data:/usr/share/elasticsearch/data - /tmp/data/elasticsearch/node-2/log:/usr/share/elasticsearch/log networks: - elastic node-3: image: elasticsearch:7.17.5 container_name: node-3 environment: - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" - "TZ=Asia/Shanghai" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 ports: - "9202:9200" logging: driver: "json-file" options: max-size: "50m" volumes: - /tmp/etc/elasticsearch/node-3/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - /tmp/etc/elasticsearch/node-3/plugins:/usr/share/elasticsearch/plugins - /tmp/data/elasticsearch/node-3/data:/usr/share/elasticsearch/data - /tmp/data/elasticsearch/node-3/log:/usr/share/elasticsearch/log networks: - elastic kibana: container_name: kibana image: kibana:7.17.5 volumes: - /tmp/data/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml ports: - 5601:5601 networks: - elastic elasticsearch-head: image: wallbase/elasticsearch-head:6-alpine container_name: elasticsearch-head environment: TZ: 'Asia/Shanghai' ports: - '9100:9100' networks: - elastic networks: elastic: driver: bridge
|
启动服务
使用以下命令就可以启动服务了