Skip to content

2.4.x field type json format bug#308

Open
CharlesTHN wants to merge 5 commits into
brokercap:v2.4.xfrom
CharlesTHN:2.4.x-field-type-json-format-bug
Open

2.4.x field type json format bug#308
CharlesTHN wants to merge 5 commits into
brokercap:v2.4.xfrom
CharlesTHN:2.4.x-field-type-json-format-bug

Conversation

@CharlesTHN

Copy link
Copy Markdown

修复问题:
运行过程中频发事件解析的空指针错误

parseEvent err:parseEvent err recover err:runtime error: invalid memory address or nil pointer dereference ;lastMapEvent:*mysql.TableMapEvent ;binlogFileName:mysql-bin.004480 ;binlogPosition:497749251
2025/10/20 11:14:03 goroutine 2909 [running]:
runtime/debug.Stack()
/Users/xiezuofu/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.8.darwin-arm64/src/runtime/debug/stack.go:26 +0x5e
github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog0.func1.1()
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/conn_dump.go:132 +0x149
panic({0x10f7a00?, 0x1c7adc0?})
/Users/xiezuofu/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.8.darwin-arm64/src/runtime/panic.go:791 +0x132
encoding/binary.Read({0x141c160, 0xc00064f230}, {0x14283b0, 0x1cc5e20}, {0x0, 0x0})
/Users/xiezuofu/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.8.darwin-arm64/src/encoding/binary/binary.go:264 +0x23f
github.com/brokercap/Bifrost/Bristol/mysql.read_offset_or_inline(0x141c160?, 0x30?)
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/field_type_json_format.go:273 +0x149
github.com/brokercap/Bifrost/Bristol/mysql.read_binary_json_object(0xc00064f230, 0x1454a, 0x1)
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/field_type_json_format.go:222 +0x395
github.com/brokercap/Bifrost/Bristol/mysql.get_field_json_data0(0x10?, 0xa0?, 0xc00084d560?)
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/field_type_json_format.go:49 +0x46
github.com/brokercap/Bifrost/Bristol/mysql.get_field_json_data({0xc002cc60b1, 0x1454b, 0x289c1}, 0x1454b)
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/field_type_json_format.go:39 +0xa5
github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseEventRow(0x10e85e0?, 0xc00064f1d0, 0xc0015d2210, {0xc004ea8780, 0xd, 0xc0015d4c60?})
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/event_row.go:588 +0x3205
github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseRowsEvent(0xc00089e120, 0xc00064f1d0)
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/event_row.go:62 +0x352
github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseEvent(0xc00089e120, {0xc002cc6001, 0x28a71, 0x28a71})
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/parser.go:256 +0xa39
github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog0.func1(0xc0004c1e20, 0xc0015d22c0?, 0xc0004c1e18, {0xc002cc6000?, 0x1?, 0x28?})
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/conn_dump.go:135 +0x85
github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog0(0xc00223ae70, 0xc00089e120, 0xc000138390)
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/conn_dump.go:136 +0x205
github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlogMySQLGtid(0xc00223ae70, 0xc00089e120, 0xc000138390)
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/conn_dump.go:66 +0x165
github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlogGtid(0xc00223ae70, 0xc00089e120, 0xc000138390)
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/conn_dump.go:52 +0x105
github.com/brokercap/Bifrost/Bristol/mysql.(*BinlogDump).startConnAndDumpBinlog.func2()
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/binlog.go:267 +0x58
created by github.com/brokercap/Bifrost/Bristol/mysql.(*BinlogDump).startConnAndDumpBinlog in goroutine 73
/Users/xiezuofu/go/src/github.com/brokercap/Bifrost/Bristol/mysql/binlog.go:261 +0x43f

修复项:
1.修复了数组类型判断错误(之前数组分支错误地检查了 JSONB_TYPE_LARGE_OBJECT,已改为检查 JSONB_TYPE_LARGE_ARRAY)。
2.修复了在 large object 分支中 key offsets 的索引错误(把原来错误的 key_offset_lengths[0] = ... 改为 key_offset_lengths[i] = ...)。
3.修改了打包语句,Go 1.13+ 的 -trimpath 参数不需要指定路径值,它会自动移除所有本地路径信息。

单元测试已过,运行 一周没有再复现此异常

tianhaonan and others added 5 commits October 20, 2025 14:28
1.修复了数组类型判断错误(之前数组分支错误地检查了 JSONB_TYPE_LARGE_OBJECT,已改为检查 JSONB_TYPE_LARGE_ARRAY)。
2.修复了在 large object 分支中 key offsets 的索引错误(把原来错误的 key_offset_lengths[0] = ... 改为 key_offset_lengths[i] = ...)。
Go 1.13+ 的 -trimpath 参数不需要指定路径值,它会自动移除所有本地路径信息。
- event_row:按 len(data) 传入解析,与 Next() 截断行为一致
- field_type_json_format:钳制 length 至 buf 剩余、嵌套递归用 buf.Len()、元素数上限、变量长字符串与对象值解析错误处理
- 增加声明长度远大于 payload 的回归测试

Made-with: Cursor
1. 修复 read_offset_or_inline 对 INT64/UINT64/DOUBLE 的处理逻辑,这些类型在 value-entry 中存储偏移量而非内联值,修复了导致解析错位的根本原因。
2. 修复 parsePartialJSONBinlog 解析顺序,符合 MySQL 8 真实格式(op -> path_len -> path -> value_len -> value)。
3. 修复 read_binary_json_object/array 中冗余判断和死代码 fallback。
4. 为 readJSONVaruintLength 添加位宽溢出防护,防止损坏数据导致死循环或溢出。
5. 在 getPluginAndSetParam 中添加边界检查,修复潜在的空指针引用问题。
6. 更新版本号至 v2.4.5-lc。

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant