fix: emit block line numbers when with_meta is set (#291) - #333
Draft
livingstaccato wants to merge 3 commits into
Draft
fix: emit block line numbers when with_meta is set (#291)#333livingstaccato wants to merge 3 commits into
livingstaccato wants to merge 3 commits into
Conversation
…#291) `SerializationOptions.with_meta` documented `__start_line__` and `__end_line__` keys, `hcl2tojson` exposed it as `--with-meta`, and the v8 migration guide told readers the v7 keys were "still available". None of that was true: v7 added the keys in `RuleTransformer.block`, the v8 rewrite moved block serialization into `BlockRule.serialize`, and the option went unread anywhere in the package. Emit them from `BlockRule.serialize`, on the same innermost dict the labels nest around and next to `__is_block__` -- the position v7 used. The output is byte-identical to 7.3.1's for the same input, nested blocks included. Two details the old implementation did not have to handle: - A tree built by the deserializer carries an empty `Meta`, whose `.line` raises. Serializing one with `with_meta` set now skips the keys rather than inventing zeros. - `dumps()` would otherwise write the metadata back out as HCL attributes, so both keys join the deserializer's reserved set. The two existing tests only asserted that the option was accepted, which is why nothing noticed; they now assert on the metadata itself.
`with_meta` puts `__start_line__` and `__end_line__` in the serialized output, which is the right place for a dict but an awkward one for a query: reading a block's span meant serializing the block and descending past however many labels it has, or reaching into the rule's private `_meta`. Both are what a downstream Terraform block reader ended up doing. Add `start_line` and `end_line` to `BlockView`. They report the same numbers `with_meta` serializes -- asserted against it rather than restated -- and are `None` for a tree built by the deserializer, which carries no positions and whose empty `Meta` has no line attributes at all. `hq` picks them up through its existing property accessors, so `hq 'resource[*] | .start_line' main.tf` works with no further wiring.
The keys travel in-band, in the same dict as the block's attributes, which is where v7 put them. That means the deserializer cannot tell a key it wrote from one the document declared, so `dumps(loads(...))` drops an attribute genuinely named `__start_line__`, and `with_meta` overwrites its value -- exactly as both already happened for `__is_block__`, `__comments__` and `__inline_comments__`. These tests state that rather than leave it to be discovered. Nothing short of moving all five keys out of band would make the metadata unambiguous, and that is a breaking change to the serialized shape, not a fix belonging to this option.
Contributor
Author
|
Please hold off on merging this one for now — I want to do another review pass over it before it goes in. Opened as a draft for that reason; I will mark it ready and say so here once I am done. |
This was referenced Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #291.
What
SerializationOptions.with_metaproduced no metadata. The option, thehcl2tojson --with-metaflag and the v8 migration guide's promise that the v7 keys are "still available" all survived the rewrite; the code that emitted them did not. v7 wrote the keys fromRuleTransformer.block, and v8 moved block serialization intoBlockRule.serializewithout carrying them over, so the option was read nowhere in the package.BlockRule.serializenow emits__start_line__and__end_line__alongside the body -- the same innermost dict the labels nest around, which is where v7 put them. The line numbers in the tests were checked against python-hcl2 7.3.1 on the same input, so they are v7's values rather than merely self-consistent.A tree built by the deserializer carries an empty
Meta, so it reports no keys rather than inventing zeros.Also here
BlockView.start_line/.end_line, so a span can be read from the query API without serializing the block.with_metaputs the numbers in the output dict, which otherwise meant reaching them through the label nesting or through the rule's private_meta. Both areNonefor a deserialized tree.hqpicks them up through its property accessors:hq 'resource[*] | .start_line' main.tf.dumps(loads(...))drops such an attribute andwith_metaoverwrites it -- as already happened for__is_block__and__comments__before these two keys existed. That collision is filed separately as__is_block__,__comments__and thewith_metakeys silently displace attributes of the same name #331, and this PR makes it worse rather than merely inheriting it: before,__start_line__was an ordinary attribute that survived a round trip, and after, it does not -- on the default path, withwith_metaoff. The tests state that rather than defend it.__is_block__,__comments__and thewith_metakeys silently displace attributes of the same name #331 now has an out-of-band fix in progress, which is where the collision goes away for all five names; if that lands first this PR should rebase onto it instead of reserving the two names in-band.Note for merge order
This branch and the one for #328 both touch
hcl2/query/blocks.pyon adjacent lines. Whichever lands second will want a keep-both resolution.Merging
It touches the same code as #332 (
hcl2/query/blocks.py), #345 (hcl2/deserializer.py and hcl2/rules/base.py). Whichever of those lands first, this one needs a rebase rather than a merge — the overlaps are real edits to the same methods, not adjacent lines, so resolving them by hand risks losing one of the two fixes. Say the word and I will rebase and re-run the suite.This pull request, and the investigation behind it, were produced by an AI assistant (Claude) working on behalf of the author. Every reproduction, test run and benchmark cited was executed rather than inferred, but please review with that provenance in mind.