Consider extracting sendwal from Klio into a standalone Go module

[joao00001]

Hi all 😄. I've been reading through Klio's core/internal/client/sendwal package (the WAL streaming receiver: IDENTIFY_SYSTEM, replication slots, START_REPLICATION, timeline handling, built on pgx + pglogrepl). It's already written with a clean sink interface (buffer.Handler) that isn't Klio-specific, and it's one of very few working, tested implementations of Postgres WAL streaming in Go.

I think it's worth considering as a standalone module rather than something buried in Klio's internal/ tree, for a few reasons:

• cloudnative-pg/plugin-barman-cloud currently shells out to pg_receivewal/barman-cloud-wal-archive, so a native Go receiver would let it drop that dependency on external binaries.
• Any future CNPG-I backup plugin needs this same piece and shouldn't have to either vendor it from Klio's internals or reimplement it.
• Full disclosure: I'm on the Barman side at EDB, and we'd also like to use this in a Go-based Barman component down the line. Wanted to say that plainly rather than present this as purely altruistic. I think the module stands on its own merits for the reasons above, but you should know the interest exists.

Two things would need to happen for it to actually be extractable cleanly: right now receiver.go pulls in the full cloudnative-pg/cloudnative-pg operator module just for one function (SegmentFromName), and it talks to Klio's gRPC client directly instead of through the Handler interface it already has for the write side. Neither is a big lift, but they're real before/after conditions for this to be a genuinely lightweight library.

Not proposing pulling WAL filename/LSN parsing into this, since that already lives in machinery and should probably stay there. Curious whether this seems worth an issue + small prototype PR, and who from Klio's maintainers would want to be involved early given it touches module boundaries.

[gbartolini]

Thanks @joao00001. Sounds like a great idea to me, but I leave it up to @fcanovai, who's currently leading the Klio project, to respond.

[fcanovai]

Hi @joao00001 ! If we can share the sendwal between the two projects I'm happy about it, it means a fundamental piece of Klio gets tested twice 😄

Although probable should change name at that point, because it won't be sending WALs to the Klio server anymore, it will just delegate that to a Klio handler, whatever shape that will take.

SegmentFromName (and all that is related) is something that could go to machinery without issues.
About the grpc, I think #98 will require some additional change, so it may not be it's definitive shape.

By the way, this is an opinionated receiver. E.g.: it creates a replication slot. Which I think is fine for a backup tool, but could be made more generic later.

If you have time for the prototype I can help (and also the other owners).

[joao00001]

I agree on the sendwal side, happy to bikeshed later, not blocking on it now. On the gRPC side I looked at #98, and I think it changes the calculus a bit: if the sink needs to move to some ack-based/bidirectional protocol to fix the flush-durability issue, the interface we'd be extracting isn't stable yet. Rather than extract now and redesign the sink again once #98 lands, would it make sense to treat that redesign as part of the same piece of work, instead of doing it twice?
On the receiver being opinionated (auto-creating the slot), I guess it's fine to keep as the default for now, since every consumer we can think of is a backup tool that needs a slot anyway. Can open it up later if someone needs otherwise.

If you're up for it, I can start a small prototype PR along these lines, with you and the other owners weighing in on the sink/slot-lifecycle shape before it goes too far.