日志通过logstash收集到redis，之后从logstash从redis读取数据存入到ES
1. logstash使用redis测试
通过标准输入到redis中
logstash配置与启动
1 [yun@mini03 config]$ pwd 2 /app/logstash/config 3 [yun@mini03 config]$ cat redis_test.conf 4 input{ 5 stdin{} 6 } 7 8 filter{ 9 }10 11 output{12 redis {13 data_type => "list"14 # 生产环境需要规划15 db => 116 host => "mini03"17 port => 637918 key => "redis_test"19 }20 }21 22 ### 使用yun用户即可23 [yun@mini03 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/redis_test.conf 24 …………25 11126 22227 33328 44429 12355530 123431 654321zhags
redis查看
1 [root@mini03 ~]# redis-cli -h mini03 -p 6379 2 mini03:6379> select 1 3 OK 4 mini03:6379[1]> KEYS * # 生产环境禁止使用该命令 5 1) "redis_test" 6 mini03:6379[1]> type redis_test 7 list 8 mini03:6379[1]> llen redis_test 9 (integer) 710 mini03:6379[1]> lindex redis_test -111 "{\"host\":\"mini03\",\"message\":\"654321zhags\",\"@timestamp\":\"2018-08-29T13:58:02.184Z\",\"@version\":\"1\"}"
2. httpd日志收集到redis中
logstash配置与启动
1 [yun@mini03 config]$ pwd 2 /app/logstash/config 3 [yun@mini03 config]$ cat redis_httpd_test.conf 4 input{ 5 file{ 6 path => ["/var/log/httpd/access_log"] 7 type => "httpd-access-log" 8 start_position => "beginning" 9 }10 }11 12 filter{13 }14 15 output{16 redis {17 data_type => "list"18 # 生产环境需要规划19 db => 120 host => "mini03"21 port => 637922 key => "apache-access-log"23 }24 }25 26 #### 使用root用户，涉及权限27 [root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/redis_httpd_test.conf # 使用root用户
使用谷歌、火狐或者IE浏览器访问
redis查看
[root@mini03 ~]# redis-cli -h mini03 -p 6379mini03:6379> select 1OKmini03:6379[1]> KEYS *1) "apache-access-log"2) "redis_test"mini03:6379[1]> llen apache-access-log(integer) 28mini03:6379[1]> lindex apache-access-log -1"{\"message\":\"10.0.0.1 - - [29/Aug/2018:22:08:30 +0800] \\\"GET /aaabbb/?aaa=bbb HTTP/1.1\\\" 404 205 \\\"-\\\" \\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0\\\"\",\"type\":\"httpd-access-log\",\"path\":\"/var/log/httpd/access_log\",\"host\":\"mini03\",\"@timestamp\":\"2018-08-29T14:08:31.442Z\",\"@version\":\"1\"}"
3. logstash从redis读取数据标准输出
注意：该logstash在mini02上读取mini03上redis的数据
读取之后先使用grok进行过滤
之后进行标准输出【命令行输出】
logstash配置与启动
[yun@mini02 config]$ pwd/app/logstash/config[yun@mini02 config]$ cat redis_stdout.conf input{ redis { data_type => "list" db => 1 host => "mini03" port => 6379 key => "apache-access-log" }}filter{ grok { match => { "message" => "%{HTTPD_COMBINEDLOG}" } }}output{ stdout { codec => rubydebug }}###### 使用yun用户即可[yun@mini02 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/redis_stdout.conf……………………{ "request" => "/noindex/css/fonts/Bold/OpenSans-Bold.ttf", "message" => "10.0.0.1 - - [30/Aug/2018:17:22:13 +0800] \"GET /noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1\" 404 238 \"http://mini03/noindex/css/open-sans.css\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"", "@version" => "1", "bytes" => "238", "auth" => "-", "referrer" => "\"http://mini03/noindex/css/open-sans.css\"", "response" => "404", "type" => "httpd-access-log", "clientip" => "10.0.0.1", "@timestamp" => 2018-08-30T09:22:13.950Z, "ident" => "-", "verb" => "GET", "path" => "/var/log/httpd/access_log", "host" => "mini03", "agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"", "timestamp" => "30/Aug/2018:17:22:13 +0800", "httpversion" => "1.1"}{ "request" => "/?refresh=1m&orgId=1", "message" => "10.0.0.1 - - [30/Aug/2018:17:22:13 +0800] \"GET /?refresh=1m&orgId=1 HTTP/1.1\" 403 4897 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"", "@version" => "1", "bytes" => "4897", "auth" => "-", "referrer" => "\"-\"", "response" => "403", "type" => "httpd-access-log", "clientip" => "10.0.0.1", "@timestamp" => 2018-08-30T09:22:13.949Z, "ident" => "-", "verb" => "GET", "path" => "/var/log/httpd/access_log", "host" => "mini03", "agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"", "timestamp" => "30/Aug/2018:17:22:13 +0800", "httpversion" => "1.1"}……………………
4. elkstack-使用redis作为消息队列【汇总】
在mini03的logstash读取httpd的日志，并存储到redis
4.1. mini03的 logstash配置如下：
1 [yun@mini03 config]$ pwd 2 /app/logstash/config 3 [yun@mini03 config]$ cat redis_httpd_test.conf 4 input{ 5 file{ 6 path => ["/var/log/httpd/access_log"] 7 type => "httpd-access-log" 8 start_position => "beginning" 9 }10 }11 12 filter{13 }14 15 output{16 redis {17 data_type => "list"18 # 生产环境需要规划19 db => 120 host => "mini03"21 port => 637922 key => "apache-access-log"23 }24 }25 26 ######## 使用root用户，涉及权限27 [root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/redis_httpd_test.conf 28 ………………
在mini02的logstash读取redis信息，并存储在ES
4.2. mini02的logstash配置
1 [yun@mini02 config]$ pwd 2 /app/logstash/config 3 [yun@mini02 config]$ cat redis_es.conf 4 input{ 5 redis { 6 data_type => "list" 7 db => 1 8 host => "mini03" 9 port => 637910 key => "apache-access-log"11 }12 }13 14 filter{15 grok {16 match => { "message" => "%{HTTPD_COMBINEDLOG}" }17 }18 }19 20 output{21 # es有3台，随便指定一台即可 也可以是多台如 ["127.0.0.1:9200","127.0.0.2:9200"]22 elasticsearch {23 hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]24 index => "httpd-access-log-%{+YYYY.MM.dd}"25 }26 }27 28 ####### 使用yun用户即可29 [yun@mini02 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/redis_es.conf 30 ………………
4.3. 浏览器访问httpd
浏览器
1 # 可以通过谷歌、火狐、IE访问2 http://mini03/ 3 http://mini03/indweg.html
Linux命令行访问
1 [yun@mini02 ~]$ ab -n40 -c 1 http://mini03/2 [yun@mini02 ~]$ ab -n40 -c 1 http://mini03/wet/bdhw/
4.4. 信息查看
elasticsearch-head查看
kibana查看
