Go实现海量日志收集系统

　　Go实现海量日志收集系统

　　再次整理了一下这个日志收集系统的框，如下图

　　这次要实现的代码的整体逻辑为：

　　完整代码地址为:

　　etcd介绍

　　高可用的分布式key-value存储，可以用于配置共享和服务发现

　　类似的项目：zookeeper和consul

　　开发语言：go

　　接口：提供restful的接口，使用简单

　　实现算法：基于raft算法的强一致性，高可用的服务存储目录

　　etcd的应用场景：

　　官网对etcd的有一个非常简明的介绍：

　　etcd搭建：

　　下载地址：

　　根据自己的环境下载对应的版本然后启动起来就可以了

　　启动之后可以通过如下命令验证一下：

　　[root@localhost etcd-v3.2.18-linux-amd64]# ./etcdctl set name zhaofan <br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /><br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" />zhaofan<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" />[root@localhost etcd-v3.2.18-linux-amd64]# ./etcdctl get name<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" />zhaofan<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" />[root@localhost etcd-v3.2.18-linux-amd64]#

　　context 介绍和使用

　　其实这个东西翻译过来就是上下文管理，那么context的作用是做什么，主要有如下两个作用：

　　通过下面一个简单的例子进行理解：

<p>package main<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /><br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" />import (<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> "fmt"<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> "time"<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> "net/http"<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> "context"<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> "io/ioutil"<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" />)<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /><br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /><br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" />type Result struct{<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> r *http.Response<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> err error<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" />}<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /><br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" />func process(){<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> ctx,cancel := context.WithTimeout(context.Background(),2*time.Second)<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> defer cancel()<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> tr := &http.Transport{}<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> client := &http.Client{Transport:tr}<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> c := make(chan Result,1)<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> req,err := http.NewRequest("GET","http://www.google.com",nil)<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> if err != nil{<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> fmt.Println("http request failed,err:",err)<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> return<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> }<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> // 如果请求成功了会将数据存入到管道中<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> go func(){<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> resp,err := client.Do(req)<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> pack := Result{resp,err}<br style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;" /> c