Skip to content

parser_csv: skip empty or unparseable lines#5355

Merged
kenhys merged 1 commit intofluent:masterfrom
Watson1978:fix-parser_csv
May 11, 2026
Merged

parser_csv: skip empty or unparseable lines#5355
kenhys merged 1 commit intofluent:masterfrom
Watson1978:fix-parser_csv

Conversation

@Watson1978
Copy link
Copy Markdown
Contributor

@Watson1978 Watson1978 commented May 9, 2026

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
This PR fixes an issue in parser_csv where processing an empty line causes a TypeError.

When parser_csv receives an empty line, CSV.parse_line returns nil.
Currently, passing this nil value to Array#zip throws a TypeError: wrong argument type NilClass (must respond to :each).

Reproduce

config:

<source>
  @type tail
  path "#{File.expand_path '~/tmp/fluentd/test.csv'}"
  tag test.csv
  read_from_head true
  <parse>
    @type csv
    keys k1,k2,k3
  </parse>
</source>

<match test.csv>
  @type stdout
</match>

Prepare input data:

$ echo "A,B,C"  > ~/tmp/fluentd/test.csv
$ echo ""      >> ~/tmp/fluentd/test.csv
$ echo "D,E,F" >> ~/tmp/fluentd/test.csv

Result of before changing:

2026-05-09 17:23:14 +0900 [info]: #0 following tail of /home/watson/tmp/fluentd/test.csv
2026-05-09 17:23:14 +0900 [warn]: #0 invalid line found 
2026-05-09 17:23:14 +0900 [warn]: #0 this parameter is highly recommended to save the position to resume tailing.
2026-05-09 17:23:14 +0900 [info]: #0 starting fluentd worker pid=41671 ppid=41646 worker=0
2026-05-09 17:23:14 +0900 [info]: #0 following tail of /home/watson/tmp/fluentd/test.csv
2026-05-09 17:23:14 +0900 [warn]: #0 invalid line found file="/home/watson/tmp/fluentd/test.csv" line="" error="wrong argument type NilClass (must respond to :each)"
2026-05-09 17:23:14.692710744 +0900 test.csv: {"k1":"A","k2":"B","k3":"C"}
2026-05-09 17:23:14.692801343 +0900 test.csv: {"k1":"D","k2":"E","k3":"F"}
2026-05-09 17:23:14 +0900 [info]: #0 fluentd worker is now running worker=0

Docs Changes:
N/A

Release Note:

  • parser_csv: skip empty or unparseable lines

@Watson1978 Watson1978 requested a review from kenhys May 9, 2026 08:25
@Watson1978 Watson1978 added the backport to v1.19 We will backport this fix to the LTS branch label May 9, 2026
@Watson1978 Watson1978 added this to the v1.20.0 milestone May 9, 2026
@Watson1978
Copy link
Copy Markdown
Contributor Author

Simple reproduction:

irb(main):001> require 'csv'
=> true
irb(main):002> @keys = ('k1'..'k3').to_a
=> ["k1", "k2", "k3"]
irb(main):003> values = CSV.parse_line('')
=> nil
irb(main):004> Hash[@keys.zip(values)]
(irb):4:in 'Array#zip': wrong argument type NilClass (must respond to :each) (TypeError)
        from (irb):4:in '<main>'
        from /home/watson/.rbenv/versions/4.0.3/lib/ruby/gems/4.0.0/gems/irb-1.18.0/exe/irb:9:in '<top (required)>'
        from /home/watson/.rbenv/versions/4.0.3/lib/ruby/site_ruby/4.0.0/rubygems.rb:305:in 'Kernel#load'
        from /home/watson/.rbenv/versions/4.0.3/lib/ruby/site_ruby/4.0.0/rubygems.rb:305:in 'Gem.activate_and_load_bin_path'
        from /home/watson/.rbenv/versions/4.0.3/bin/irb:25:in '<main>'

Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
Copy link
Copy Markdown
Contributor

@kenhys kenhys left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kenhys kenhys merged commit 0547ff9 into fluent:master May 11, 2026
21 checks passed
@Watson1978 Watson1978 deleted the fix-parser_csv branch May 11, 2026 01:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport to v1.19 We will backport this fix to the LTS branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants