Key-value DB using gob for encoding/decoding

[vqhuy]

I have made an improvement in commit 84fc324bc27fee50c0619a9538c0d91421b804d1. This implementation's performance is better than #37 now.

Update (description below)

This pull uses encoding/gob to encode/decode the merkletree structs to/from a buffer, and it implements the db backend hook for the merkletree package.

The package merkletree is now exporting some functions for encoding/decoding the PAD, STR, Merkletree and Policies from a buffer (any types that implement the io.Writer and io.Reader interface). This way, I think we can hide the detail implementations of the encoding/decoding from the developers.

Move to the hook implementation, there are two cases when we want to use a persistent storage:

• Store the current directory (pad).
• Store the old STRs.

This hook exports APIs for store/load the PAD, STR to/from the db.

[liamsi]

Looking good. I added a few questions. Hope to do a more detailed review before our hangouts session later.

[liamsi]

Arent' those constants the same as in https://github.com/coniks-sys/coniks-go/blob/84fc324bc27fee50c0619a9538c0d91421b804d1/merkletree/storage.go ?

[vqhuy]

Oh, this is a old commit that has been deleted.

[liamsi]

I guess all this values need to public to correctly encode now?

[vqhuy]

Yes, since we didn't export the structs so I think it still ensures our preservation, i.e., there's no way to change or read the node's fields.
Btw, the reason is (mostly) because I didn't want to implement the GobEncode/GobDecode interface manually.

[liamsi]

Some suggestions/questions.

[liamsi]

Why is the comment not relevant anymore?

[liamsi]

Nitpicking: We would need this naming to be consistent. I liked the identifier (we already have so many different "keys" ;-) Here, we still stick with identifier: https://github.com/coniks-sys/coniks-go/pull/88/files#diff-7e689735c327b0bd737cd6e956e20b4aR4
Anyways, choose what you prefer. I just think it should be consistent.

[vqhuy]

Oops, it must be my mistake. I have no intention to change this! Thanks!

[liamsi]

This could be unexported or not? If this should stay exported it would need a more speaking name.

[vqhuy]

Do you have any suggestion for this? :D

[liamsi]

If it is unexported the name is OK.

[liamsi]

This and LoadSTR currently aren't used. I guess this will change?

[vqhuy]

They would be used to load the old STRs which were deleted from the pad.snapshot.

[liamsi]

Cool, thanks. Can we have a test for that?

[vqhuy]

Yup!

[liamsi]

From my point of view this PR looks very good (just a few questions and nitpicks).

[vqhuy]

Thanks!
Update: I made some changes in e8e9169c14efeb33810e1053b8f92d9a160d691e according to @liamsi's comments.

[masomel]

The encoding, decoding and reconstruction functions aren't part of the core data structure, they are only relevant to the DB, and they are ultimately only used in EncodePAD, which is really only used by StorePAD in padkv.go. As such, all gob encoding and decoding functions related to the DB should be part of the storage/kv/merkletreekv package, not the core merkletree package.

I agree that the encoding format is an implementation detail we should hide from the developer, but I don't see the reason why these functions need to be part of the core data structure if they are used only for a separate feature (the DB) of coniks-go.

[vqhuy]

PTAL. I need to use this in extensions implementation ;)