elasticsearch + logstash + kibana3

目的

搭建该平台的目的就是为了运维、研发很方便的进行日志的查询

简介

  1. logstsh

    • 收集日志转到任意地方
    • inputs >> codecs >> filters >> outputs
    • 支持的插件（很多 官网）
    • ruby

  2. ElasticSearch

    • 基于lucene的开源搜索引擎
    • 实时、分布式、高可用、文档型、restful api
    • java

  3. kibana

    • kibana是一个功能强大的elasticsearch数据显示客户端
    • 纯html+js客户端，可以很方便的部署到Apache、Nginx等Http服务器
    • 酷图

架构图

安装

依赖jdk
安装过程参考

实例

标注输入

input {
  stdin{}
}

filter {
  grok {
    match => [ "message", "(?<ttt>\d{1,3})" ]
  }
}

output{
 stdout{
   codec => rubydebug
 }
}

输入结果

nginx 访问日志

nginx 配置

http {
        log_format logstash_json '{ "@timestamp": "$time_iso8601", '
                                 '"@fields": { '
                                 '"remote_addr": "$remote_addr", '
                                 '"remote_user": "$remote_user", '
                                 '"body_bytes_sent": "$body_bytes_sent", '
                                 '"request_time": "$request_time", '
                                 '"status": "$status", '
                                 '"request": "$request", '
                                 '"request_method": "$request_method", '
                                 '"http_referrer": "$http_referer", '
                                 '"http_user_agent": "$http_user_agent" } }';
}

server {
    access_log /var/log/nginx/xiaoma.log logstash_json;
}

nginx-access-json.conf

input{
  file {
     path => "/usr/local/logstash/conf/xiaoma.log"
     type => "nginx_json"
     start_position => "beginning"
     format => "json_event"
     sincedb_path => "/dev/null"
  }
}

filter {
  if [type] == "nginx_json" {
     geoip {
       source => "remote_addr"
     }
  }
}
output{
   stdout{
       codec => rubydebug
   }

   elasticsearch {
        host => "127.0.0.1"
   }
}

输入结果

mysql慢查询

input {
  file {
    type => "mysql-slow"
    path => "/usr/local/src/mysql-slow.log"
    start_position => "beginning"
  }
}

filter {
      grep {
        match => [ "@message", "^# Time: " ]
        negate => true
      }

      grok {
        singles => true
        pattern => [
          "^# User@Host: %{USER:user}\[[^\]]+\] @ %{HOST:host} \[%{IP:ip}?]",
          "^# Query_time: %{NUMBER:duration:float} \s*Lock_time: %{NUMBER:lock_wait:float} \s*Rows_sent: %{NUMBER:results:int} \s*Rows_examined: %{NUMBER:scanned:int}",
          "^SET timestamp=%{NUMBER:timestamp};"
        ]
      }

      multiline {
        pattern => "^# User@Host: "
        negate => true
        what => previous
      }

      date {
        match => ["timestamp", UNIX]
      }

      mutate {
        remove => "timestamp"
      }
    }

output {
  stdout {
    codec => rubydebug
  }

  elasticsearch {
     host => "168.192.122.58"
     protocol => "http"
     workers => 5
  }
}

输入结果

学习资料