ubuntu elasticsearch,logstash,kibana,filebeat安装配置

elk 简单说是一个分布式的日志管理系统，包括elasticsearch,logstash,kibana，程序可以通过网页查看日志信息，解决问题，减轻运维人员工作。

一，elasticsearch,logstash,kibana,filebeat功能介绍

elasticsearch搜索工具，服务端安装

logstash日志收集工具，收集filebeat输出的日志，服务端安装

kibana图形管理工具，服务端安装

filebeat日志输出工具，客户端安装

 二，安装java8

# apt-get install openjdk-8-jdk

# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

三，添加elasticsearch,logstash,kibana,filebeat源文件

# wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add -

# echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
# echo "deb http://packages.elastic.co/kibana/4.5/debian stable main" | tee -a /etc/apt/sources.list
# echo "deb http://packages.elastic.co/logstash/2.3/debian stable main" | tee -a /etc/apt/sources.list
# echo "deb https://packages.elastic.co/beats/apt stable main" |  tee -a /etc/apt/sources.list.d/beats.list

# apt-get update  //更新包

四，安装elasticsearch,logstash,kibana,filebeat

# apt-get install elasticsearch logstash kibana filebeat

五，配置elasticsearch logstash kibana filebeat nginx

1，配置，启动elasticsearch

# cat /etc/elasticsearch/elasticsearch.yml|awk '{if($0 !~ /^$/ && $0 !~ /^#/) {print $0}}'  
network.host: localhost
http.port: 9200

# systemctl start elasticsearch

# ps aux |grep  elasticsearch
elastic+ 17572  1.8  9.1 3567152 361768 ?      Ssl  14:00   1:28 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
 -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.4.6.jar:/usr/share/
 elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/
 share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch
root     20515  0.0  0.0  15992   904 pts/1    S+   15:20   0:00 grep --color=auto elasticsearch

注意：一开始的没有添加elasticsearch的源，因为默认的有。但是安装完了以后，启动不起来，原因是没有可执行文件。

2，配置，启动kibana

# cat /opt/kibana/config/kibana.yml |awk '{if($0 !~ /^$/ && $0 !~ /^#/) {print $0}}'  
server.host: "localhost"

# /etc/init.d/kibana start

# ps aux |grep kibana
kibana   11846  0.3  7.6 1474504 300980 ?      Ssl  10:54   0:54 /opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/cli
root     20769  0.0  0.0  15992   972 pts/1    S+   15:28   0:00 grep --color=auto kibana

3，配置 启动logstash

# cat /etc/logstash/conf.d/02-beats-input.conf | awk '{if($0 !~ /^$/ && $0 !~ /^#/) {print $0}}'
input {
  beats {
    port => 5044
  }
}

# cat /etc/logstash/conf.d/10-syslog-filter.conf | awk '{if($0 !~ /^$/ && $0 !~ /^#/) {print $0}}'
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

# cat /etc/logstash/conf.d/30-elasticsearch-output.conf | awk '{if($0 !~ /^$/ && $0 !~ /^#/) {print $0}}'
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

# service logstash configtest   //测试配置
Configuration OK

# /etc/init.d/logstash start
# ps aux|grep logstash

4，（客户端）配置 启动filebeat

# cat /etc/filebeat/filebeat.yml |awk '{gsub(/^ +#/,"#",$0);if($0 !~ /^$/ && $0 !~ /^#/) {print $0}}'
filebeat:
  prospectors:
    -
      paths:
        - /var/log/syslog
        - /var/log/auth.log
      input_type: log
      document_type: syslog
  registry_file: /var/lib/filebeat/registry
output:
  logstash:
    hosts: ["192.168.0.90:5044"]  //服务端IP
    bulk_max_size: 1024
shipper:
logging:
  files:
    rotateeverybytes: 10485760 # = 10MB

# /etc/init.d/filebeat start

# ps aux |grep filebeat
root     11033  0.0  0.2 285796  9220 ?        Ssl  13:14   0:04 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root     19867  0.0  0.0  15984   964 pts/21   S+   19:17   0:00 grep --color=auto filebeat

6，配置nginx

server {
    listen 80;

    server_name 192.168.0.90;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}

 六，服务端安装filebeat仪表板，模板

# curl -L -O https://download.elastic.co/beats/dashboards/beats-dashboards-1.2.2.zip
# unzip beats-dashboards-1.2.2.zip
# cd beats-dashboards-1.2.2
# ./load.sh

# curl -O https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json
# curl -XPUT 'http://localhost:9200/_template/filebeat?pretty' -d@filebeat-index-template.json

如果不加，登录kibana后，会提示No default index pattern. You must select or create one to continue.当然也可以自己添加