Grok配置案例：
成都创新互联公司服务项目包括宿豫网站建设、
宿豫网站制作、宿豫网页制作以及宿豫网络营销策划等。多年来，我们专注于互联网行业，利用自身积累的技术优势、行业经验、深度合作伙伴关系等，向广大中小型企业、政府机构等提供互联网行业的解决方案，
宿豫网站推广取得了明显的社会效益与经济效益。目前，我们服务的客户以成都为中心已经辐射到宿豫省份的部分城市，未来相信会继续扩大服务区域并继续获得客户的支持与信任！##启动文件配置：
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
stdin{}
}
filter {
grok {
match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\
%{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"]
}
}
output {
stdout{
codec => "rubydebug"
}
}
##输出文件内容
172.16.213.132 [07/Feb/2018:16:24:19 +0800] "GET / HTTP/1.1" 403 5039
##显示内容
{
"@version" => "1",
"@timestamp" => 2019-11-10T06:02:42.865Z,
"host" => "localhost.localdomain",
"message" => "172.16.213.132 [07/Feb/2018:16:24:19 +0800] \"GET / HTTP/1.1\" 403 5039",
"timestamp" => "07/Feb/2018:16:24:19 +0800",
"bytes" => "5039",
"response" => "403",
"clientip" => "172.16.213.132",
"referrer" => "\"GET / HTTP/1.1\""
}
Grok 过滤重复字段
## 配置文件
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
stdin{
}
}
filter {
grok {
match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\
%{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"]
remove_field => ["message"]
}
}
output {
stdout{
codec => "rubydebug"
}
}
Grok搭配Date时间插件配置
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
stdin{
}
}
filter {
grok {
match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\
%{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"]
remove_field => ["message"]
}
date {
match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"]
}
}
output {
stdout{
codec => "rubydebug"
}
}
Date 过滤重复得字段配置
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
stdin{
}
}
filter {
grok {
match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\
%{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"]
remove_field => ["message"]
}
date {
match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"]
}
mutate {
remove_field => [ "timestamp" ]
}
}
output {
stdout{
codec => "rubydebug"
}
}
综合练习配置参数
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
stdin{
}
}
filter {
grok {
match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\
%{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"]
remove_field => ["message"]
}
date {
match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"]
}
mutate{
rename => {"response" => "response_new"}
gsub => ["referrer", "\"", ""]
remove_field => [ "timestamp" ]
split => ["clientip", "."]
}
}
output {
stdout{
codec => "rubydebug"
}
}
Geoip 地理位置插件操作方式
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
stdin{
}
}
filter {
grok {
match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\
%{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"]
remove_field => ["message"]
}
date {
match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"]
}
mutate{
remove_field => [ "timestamp" ]
}
geoip {
source => "clientip"
database => "/usr/local/include/GeoLite2-ASN_20191105/GeoLite2-ASN.mmdb"
}
}
output {
stdout{
codec => "rubydebug"
}
}
Geoip输出指定属性值
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
stdin{
}
}
filter {
grok {
match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\
%{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"]
remove_field => ["message"]
}
date {
match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"]
}
mutate{
remove_field => [ "timestamp" ]
}
geoip {
source => "clientip"
#database => "/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb"
database => "/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb"
fields => ["city_name", "region_name", "country_name", "ip", "latitude", "longitude", "timezone"]
}
}
output {
stdout{
codec => "rubydebug"
}
}
模拟数据：
36.7.152.182 [07/Feb/2018:16:24:19 +0800] "GET / HTTP/1.1" 403 5039
综合实战
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
stdin{}
}
filter{
grok{
match => {"message" => "%{TIMESTAMP_ISO8601:localtime}\|\~\|%{IP:clientip}
\|\~\|%{GREEDYDATA:http_user_agent}\|\~\|%{GREEDYDATA:url}
\|\~\|%{GREEDYDATA:mediaid}\|\~\|%{GREEDYDATA:osid}"}
remove_field => [ "message" ]
}
date {
match => ["localtime", "yyyy-MM-dd'T'HH:mm:ssZZ"]
target => "@timestamp"
}
mutate {
remove_field => ["localtime"]
}
geoip {
source => "clientip"
#database => "/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb"
database => "/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb"
fields => ["city_name", "region_name", "country_name", "ip", "latitude", "longitude", "timezone"]
}
}
output {
stdout {
codec => "rubydebug"
}
}
示例：2018-02-09T10:57:42+08:00|~|123.87.240.97|~|Mozilla/5.0
(iPhone;CPU iPhone OS 11_2_2 like Mac OS X)
AppleWebKit/604.4.7 Version/11.0 Mobile/15C202 Safari/604.1
|~|http://m.sina.cn/cm/ads_ck_wap.html
|~|12434785489009|~|DF45566587855P
另外有需要云服务器可以了解下创新互联cdcxhl.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。
新闻名称：Logstash基础操作-Filter-创新互联
分享链接：
http://www.cdkjz.cn/article/dcepoj.html
