unexpected error error="undefined method `[]=' for nil:NilClass" #5
Description
We got the following error when sending messages from filebeat to fluentd:
unexpected error error="undefined method `[]=' for nil:NilClass"
With the traceback:
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/fluent/plugin/in_beats.rb:103:in `block (3 levels) in run'
/var/lib/gems/2.3.0/gems/fluentd-0.14.20/lib/fluent/plugin/parser_json.rb:65:in `rescue in parse'
/var/lib/gems/2.3.0/gems/fluentd-0.14.20/lib/fluent/plugin/parser_json.rb:61:in `parse'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/fluent/plugin/in_beats.rb:102:in `block (2 levels) in run'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:386:in `data'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:361:in `block (2 levels) in read_socket'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:373:in `ack_if_needed'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:357:in `block in read_socket'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:246:in `json_data_payload'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:163:in `feed'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:296:in `compressed_payload'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:163:in `feed'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:342:in `read_socket'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:319:in `run'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/fluent/plugin/in_beats.rb:97:in `block in run'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:331:in `run_task'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (3 levels) in create_worker'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `loop'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `block (2 levels) in create_worker'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `catch'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `block in create_worker'
The version info:
- filebeat
5.6.0 - fluentd
0.14.20 - fluent-plugin-beats
0.1.2
The filebeat config:
filebeat.prospectors:
- input_type: log
paths:
- /tmp/logs/*.log
output.logstash:
enabled: true
hosts: ["fluentd:5044"]
logging.level: debug
The fluentd config:
<source>
@type beats
port 5044
tag mytag.beats
format json
time_key @timestamp
time_format %Y-%m-%dT%H:%M:%S.%L%Z
</source>
The filebeat debug log shows that the message failed to send:
filebeat_1 | 2017/09/13 11:15:20.033374 log_file.go:84: DBG End of file reached: /tmp/logs/test.log; Backoff now.
filebeat_1 | 2017/09/13 11:15:21.977556 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 1
filebeat_1 | 2017/09/13 11:15:21.977825 client.go:214: DBG Publish: {
filebeat_1 | "@timestamp": "2017-09-13T11:15:17.032Z",
filebeat_1 | "beat": {
filebeat_1 | "hostname": "c80bd1b1ef19",
filebeat_1 | "name": "c80bd1b1ef19",
filebeat_1 | "version": "5.6.0"
filebeat_1 | },
filebeat_1 | "input_type": "log",
filebeat_1 | "message": "12345 aaaaa",
filebeat_1 | "offset": 12,
filebeat_1 | "source": "/tmp/logs/test.log",
filebeat_1 | "type": "log"
filebeat_1 | }
filebeat_1 | 2017/09/13 11:15:21.977846 output.go:109: DBG output worker: publish 1 events
filebeat_1 | 2017/09/13 11:15:21.977869 sync.go:53: DBG connect
filebeat_1 | 2017/09/13 11:15:21.983998 sync.go:107: DBG Try to publish 1 events to logstash with window size 10
filebeat_1 | 2017/09/13 11:15:21.985179 client.go:194: DBG handle error: EOF
filebeat_1 | 2017/09/13 11:15:21.985239 client.go:110: DBG closing
filebeat_1 | 2017/09/13 11:15:21.985533 sync.go:78: DBG 0 events out of 1 events sent to logstash. Continue sending
filebeat_1 | 2017/09/13 11:15:21.985593 sync.go:58: DBG close connection
filebeat_1 | 2017/09/13 11:15:21.985631 sync.go:85: ERR Failed to publish events caused by: EOF
filebeat_1 | 2017/09/13 11:15:21.985646 single.go:91: INFO Error publishing events (retrying): EOF
filebeat_1 | 2017/09/13 11:15:21.985652 sync.go:58: DBG close connection
filebeat_1 | 2017/09/13 11:15:21.985657 single.go:156: DBG send fail
fluentd_1 | 2017-09-13 11:15:21 +0000 [error]: #0 unexpected error error="undefined method `[]=' for nil:NilClass"
....
Is there anything that I missed or did wrong? Thanks.