使用日志飞行员进行日志采集

　　log-Pilot是一个智能容器日志采集工具. 它不仅可以高效便捷地将容器日志采集和输出到各种存储日志后端，还可以动态发现并采集容器内部的日志文件. 更多咨询，您可以在这里移动.

　　log-Pilot当前支持两种工具来采集日志，即Fluentd插件和Filebeat插件.

　　Log-Pilot支持容器事件管理. 它可以动态监视容器的事件变化，然后根据容器标签进行解析，生成日志采集配置文件，然后将其移交给采集插件以进行日志采集.

　　在Kubernetes下，Log-Pilot可以基于环境变量aliyun_logs_ $ name = $ path来动态生成日志采集配置文件，该环境变量收录两个变量:

　　部署示例:

　　apiVersion: apps/v1

kind: DaemonSet

metadata:

name: log-pilot

labels:

app: log-pilot

namespace: kube-ops

spec:

selector:

matchLabels:

app: log-pilot

updateStrategy:

type: RollingUpdate

template:

metadata:

labels:

app: log-pilot

annotations:

scheduler.alpha.kubernetes.io/critical-pod: ''

spec:

tolerations:

- key: node-role.kubernetes.io/master

effect: NoSchedule

containers:

- name: log-pilot

image: registry.cn-hangzhou.aliyuncs.com/acs/log-pilot:0.9.7-filebeat

resources:

limits:

memory: 500Mi

requests:

cpu: 200m

memory: 200Mi

env:

- name: "NODE_NAME"

valueFrom:

fieldRef:

fieldPath: spec.nodeName

# 日志收集前缀

- name: PILOT_LOG_PREFIX

value: aliyun

- name: "LOGGING_OUTPUT"

value: "elasticsearch"

# 请确保集群到ES网络可达

- name: "ELASTICSEARCH_HOSTS"

value: "elasticsearch:9200"

# 配置ES访问权限

#- name: "ELASTICSEARCH_USER"

# value: "{es_username}"

#- name: "ELASTICSEARCH_PASSWORD"

# value: "{es_password}"

volumeMounts:

- name: sock

mountPath: /var/run/docker.sock

- name: root

mountPath: /host

readOnly: true

- name: varlib

mountPath: /var/lib/filebeat

- name: varlog

mountPath: /var/log/filebeat

- name: localtime

mountPath: /etc/localtime

readOnly: true

livenessProbe:

failureThreshold: 3

exec:

command:

- /pilot/healthz

initialDelaySeconds: 10

periodSeconds: 10

successThreshold: 1

timeoutSeconds: 2

securityContext:

capabilities:

add:

- SYS_ADMIN

terminationGracePeriodSeconds: 30

volumes:

- name: sock

hostPath:

path: /var/run/docker.sock

- name: root

hostPath:

path: /

- name: varlib

hostPath:

path: /var/lib/filebeat

type: DirectoryOrCreate

- name: varlog

hostPath:

path: /var/log/filebeat

type: DirectoryOrCreate

- name: localtime

hostPath:

path: /etc/localtim

　　创建广告连播测试

　　apiVersion: v1

kind: Pod

metadata:

name: tomcat

spec:

containers:

- name: tomcat

image: "tomcat:8.0"

env:

# 1、stdout为约定关键字，表示采集标准输出日志

# 2、配置标准输出日志采集到ES的catalina索引下

- name: aliyun_logs_catalina

value: "stdout"

# 1、配置采集容器内文件日志，支持通配符

# 2、配置该日志采集到ES的access索引下

- name: aliyun_logs_access

value: "/usr/local/tomcat/logs/catalina.*.log"

# 容器内文件日志路径需要配置emptyDir

volumeMounts:

- name: tomcat-log

mountPath: /usr/local/tomcat/logs

volumes:

- name: tomcat-log

emptyDir: {}

　　然后，我们检查索引并查看access-和catalina-的索引

　　# curl -XGET 'localhost:9200/_cat/indices?v&pretty'

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

green open access-2020.06.23 0LS6STfpQ4yHt7makuSI1g 5 1 40 0 205.5kb 102.5kb

green open logstash-2020.06.23 HR62innTQi6HjObIzf6DHw 5 1 99 0 296kb 148kb

green open catalina-2020.06.23 dSFGcZlPS6-wieFKrOWV-g 5 1 40 0 227.1kb 133.3kb

green open .kibana H-TAto8QTxmi-jI_4mIUrg 1 1 2 0 20.4kb 10.2kb

green open logstash-2020.06.22 8-IFAOj_SqiipqOXN6Soxw 5 1 43784 0 30.6mb 15.3mb

　　然后在页面上添加索引.

　　当然，除了直接输出到es之外，日志输出还可以输出到其他地方. 如果您正在使用filebeat，则可以单击此处查看. 如果需要，请单击此处.