Filebeat实时采集Nginx日志

　　说明Filebeat版本为5.3.0

　　之所以使用Beats系列的Filebeat代替Logstash的原因是，Logstash消耗了太多资源（请忽略那些拥有足够服务器资源的用户）

　　在官方网站上，Logstash的下载量为89M，而Filebeat的下载量仅为840M，这很明显

　　Logstash可以配置jvm参数. 经过我自己的调试，内存分配很小，启动缓慢，有时甚至根本无法启动. 如果分配量很大，其他服务将没有资源

　　对于低配置服务器而言，选择Filebeat是最佳选择，并且既然Filebeat已开始取代Logstash，仍然有必要修改nginx日志格式nginx.config

　　更改日志记录的格式

　　 log_format json '{ "@timestamp": "$time_iso8601", '

'"time": "$time_iso8601", '

'"remote_addr": "$remote_addr", '

'"remote_user": "$remote_user", '

'"body_bytes_sent": "$body_bytes_sent", '

'"request_time": "$request_time", '

'"status": "$status", '

'"host": "$host", '

'"request": "$request", '

'"request_method": "$request_method", '

'"uri": "$uri", '

'"http_referrer": "$http_referer", '

'"body_bytes_sent":"$body_bytes_sent", '

'"http_x_forwarded_for": "$http_x_forwarded_for", '

'"http_user_agent": "$http_user_agent" '

'}';

access_log /var/log/nginx/access.log json;

　　filebeat.yml

　　 #=========================== Filebeat prospectors =============================

filebeat.prospectors:

- input_type: log

# Paths that should be crawled and fetched. Glob based paths.

paths:

- /var/log/nginx/*access*.log

json.keys_under_root: true

json.overwrite_keys: true

#-------------------------- Elasticsearch output ------------------------------

output.elasticsearch:

# Array of hosts to connect to.

hosts: ["ip:port","ip:port"]

index: "filebeat_server_nginx_%{+YYYY-MM}"

　　这里要注意的是

　　json.keys_under_root: 默认值为FALSE，这意味着我们的json日志将在解析后放置在json密钥上. 设置为TRUE，所有键都将放置在根节点中

　　json.overwrite_keys: 是否覆盖原创密钥，这是密钥配置. 将keys_under_root设置为TRUE之后，然后将overwrite_keys设置为TRUE以覆盖filebeat默认密钥值

　　还有其他配置

　　json.add_error_key: 添加json_error密钥以记录json解析失败错误

　　json.message_key: 指定解析后放置json日志的键，默认为json，还可以指定日志等.

　　坦率地说，区别在于配置前的elasticsearch数据如下:

　　 {

"_index": "filebeat_server_nginx_2018-05",

"_type": "log",

"_id": "AWM9sVOkCcRcg0IPg399",

"_version": 1,

"_score": 1,

"_source": {

"@timestamp": "2018-05-08T03:00:17.544Z",

"beat": {

"hostname": "VM_252_18_centos",

"name": "VM_252_18_centos",

"version": "5.3.0"

},

"input_type": "log",

"json": {},

"message": "{ "@timestamp": "2018-05-08T11:00:11+08:00", "time": "2018-05-08T11:00:11+08:00", "remote_addr": "113.16.251.67", "remote_user": "-", "body_bytes_sent": "403", "request_time": "0.000", "status": "200", "host": "blog.joylau.cn", "request": "GET /img/%E7%BD%91%E6%98%93%E4%BA%91%E9%9F%B3%E4%B9%90.png HTTP/1.1", "request_method": "GET", "uri": "/img/\xE7\xBD\x91\xE6\x98\x93\xE4\xBA\x91\xE9\x9F\xB3\xE4\xB9\x90.png", "http_referrer": "http://blog.joylau.cn/css/style.css", "body_bytes_sent":"403", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36" }",

"offset": 7633,

"source": "/var/log/nginx/access.log",

"type": "log"

}

}

　　配置后，它看起来像这样:

　　 {

"_index": "filebeat_server_nginx_2018-05",

"_type": "log",

"_id": "AWM9rjLd8mVZNgvhdnN9",

"_version": 1,

"_score": 1,

"_source": {

"@timestamp": "2018-05-08T02:56:50.000Z",

"beat": {

"hostname": "VM_252_18_centos",

"name": "VM_252_18_centos",

"version": "5.3.0"

},

"body_bytes_sent": "12576",

"host": "blog.joylau.cn",

"http_referrer": "http://blog.joylau.cn/",

"http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36",

"http_x_forwarded_for": "-",

"input_type": "log",

"offset": 3916,

"remote_addr": "60.166.12.138",

"remote_user": "-",

"request": "GET /2018/03/01/JDK8-Stream-Distinct/ HTTP/1.1",

"request_method": "GET",

"request_time": "0.000",

"source": "/var/log/nginx/access.log",

"status": "200",

"time": "2018-05-08T10:56:50+08:00",

"type": "log",

"uri": "/2018/03/01/JDK8-Stream-Distinct/index.html"

}

}

　　这样看起来很舒服

　　启动FileBeat

　　进入Filebeat目录

　　 nohup sudo ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &

　　更新

　　如果nginx日志收录中文，则中文将转换为Unicode编码. 如果没有，只需添加escape = json参数.

　　 log_format json escape=json '{ "@timestamp": "$time_iso8601", '

'"time": "$time_iso8601", '

'"remote_addr": "$remote_addr", '

'"remote_user": "$remote_user", '

'"body_bytes_sent": "$body_bytes_sent", '

'"request_time": "$request_time", '

'"status": "$status", '

'"host": "$host", '

'"request": "$request", '

'"request_method": "$request_method", '

'"uri": "$uri", '

'"http_referrer": "$http_referer", '

'"body_bytes_sent":"$body_bytes_sent", '

'"http_x_forwarded_for": "$http_x_forwarded_for", '

'"http_user_agent": "$http_user_agent" '

'}';

access_log /var/log/nginx/access.log json;

　　消息