logstash处理文件进度记录机制
logstash,处理进度2016-11-24
input {
file {
path => "/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/common-sql-*.log"
start_position => "beginning"
type => "sql"
codec => json {
charset => "UTF-8"
}
}
}
output {
if "_grokparsefailure" in [tags] {
}else{
if [type] == "sql"{
elasticsearch {
hosts => ["http://192.168.33.10:9200"]
index => "common-sql-%{+YYYY.MM.dd}"
}
}
}
}
所有匹配common-sql-*.log的文件都将被处理
第一次从头开始处理文件
处理后以json格式输出到elasticsearch
177037 0 64768 25951716
176956 0 64768 32955178
[vagrant@hb-localhost ~]$ stat logstash/logstash-2.2.2/dbpool-logs/dev/common-sql-2016-11-24.log
File: `logstash/logstash-2.2.2/dbpool-logs/dev/common-sql-2016-11-24.log'
Size: 32955178 Blocks: 64368 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 176956 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 501/ vagrant) Gid: ( 501/ vagrant)
Access: 2016-11-24 02:08:19.058506565 +0000
Modify: 2016-11-24 01:46:14.000000000 +0000
Change: 2016-11-24 02:05:06.194122690 +0000
[vagrant@hb-localhost ~]$ stat logstash/logstash-2.2.2/dbpool-logs/dev/common-sql-2016-11-23.log
File: `logstash/logstash-2.2.2/dbpool-logs/dev/common-sql-2016-11-23.log'
Size: 25951716 Blocks: 50688 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 177037 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 501/ vagrant) Gid: ( 501/ vagrant)
Access: 2016-11-24 02:15:28.217978771 +0000
Modify: 2016-11-23 13:19:16.000000000 +0000
Change: 2016-11-24 02:15:27.913826772 +0000