Skip to content

parser_csv: skip empty or unparseable lines#5355

Open
Watson1978 wants to merge 1 commit intofluent:masterfrom
Watson1978:fix-parser_csv
Open

parser_csv: skip empty or unparseable lines#5355
Watson1978 wants to merge 1 commit intofluent:masterfrom
Watson1978:fix-parser_csv

Conversation

@Watson1978
Copy link
Copy Markdown
Contributor

@Watson1978 Watson1978 commented May 9, 2026

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
This PR fixes an issue in parser_csv where processing an empty line causes a TypeError.

When parser_csv receives an empty line, CSV.parse_line returns nil.
Currently, passing this nil value to Array#zip throws a TypeError: wrong argument type NilClass (must respond to :each).

Reproduce

config:

<source>
  @type tail
  path "#{File.expand_path '~/tmp/fluentd/test.csv'}"
  tag test.csv
  read_from_head true
  <parse>
    @type csv
    keys k1,k2,k3
  </parse>
</source>

<match test.csv>
  @type stdout
</match>

Prepare input data:

$ echo "A,B,C"  > ~/tmp/fluentd/test.csv
$ echo ""      >> ~/tmp/fluentd/test.csv
$ echo "D,E,F" >> ~/tmp/fluentd/test.csv

Result of before changing:

2026-05-09 17:23:14 +0900 [info]: #0 following tail of /home/watson/tmp/fluentd/test.csv
2026-05-09 17:23:14 +0900 [warn]: #0 invalid line found 
2026-05-09 17:23:14 +0900 [warn]: #0 this parameter is highly recommended to save the position to resume tailing.
2026-05-09 17:23:14 +0900 [info]: #0 starting fluentd worker pid=41671 ppid=41646 worker=0
2026-05-09 17:23:14 +0900 [info]: #0 following tail of /home/watson/tmp/fluentd/test.csv
2026-05-09 17:23:14 +0900 [warn]: #0 invalid line found file="/home/watson/tmp/fluentd/test.csv" line="" error="wrong argument type NilClass (must respond to :each)"
2026-05-09 17:23:14.692710744 +0900 test.csv: {"k1":"A","k2":"B","k3":"C"}
2026-05-09 17:23:14.692801343 +0900 test.csv: {"k1":"D","k2":"E","k3":"F"}
2026-05-09 17:23:14 +0900 [info]: #0 fluentd worker is now running worker=0

Docs Changes:
N/A

Release Note:

  • parser_csv: skip empty or unparseable lines

@Watson1978 Watson1978 requested a review from kenhys May 9, 2026 08:25
@Watson1978 Watson1978 added the backport to v1.19 We will backport this fix to the LTS branch label May 9, 2026
@Watson1978 Watson1978 added this to the v1.20.0 milestone May 9, 2026
@Watson1978
Copy link
Copy Markdown
Contributor Author

Simple reproduction:

irb(main):001> require 'csv'
=> true
irb(main):002> @keys = ('k1'..'k3').to_a
=> ["k1", "k2", "k3"]
irb(main):003> values = CSV.parse_line('')
=> nil
irb(main):004> Hash[@keys.zip(values)]
(irb):4:in 'Array#zip': wrong argument type NilClass (must respond to :each) (TypeError)
        from (irb):4:in '<main>'
        from /home/watson/.rbenv/versions/4.0.3/lib/ruby/gems/4.0.0/gems/irb-1.18.0/exe/irb:9:in '<top (required)>'
        from /home/watson/.rbenv/versions/4.0.3/lib/ruby/site_ruby/4.0.0/rubygems.rb:305:in 'Kernel#load'
        from /home/watson/.rbenv/versions/4.0.3/lib/ruby/site_ruby/4.0.0/rubygems.rb:305:in 'Gem.activate_and_load_bin_path'
        from /home/watson/.rbenv/versions/4.0.3/bin/irb:25:in '<main>'

Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport to v1.19 We will backport this fix to the LTS branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant