A heredoc whose interpolation spans lines cannot be flattened to valid HCL

[livingstaccato]

Summary

preserve_heredocs=False returns a heredoc as quoted-string source. That form
cannot represent a body whose interpolation spans lines: the newlines inside
${...} are expression source, where neither a raw newline nor an escaped one
is valid.

Reproduction

locals {
  a = <<EOT
${
  1 + 2
}
EOT
}

OpenTofu v1.12.5 evaluates that to "3\n".

Flattening it produces "${\n 1 + 2\n}\n" with the newlines escaped, and
OpenTofu rejects that source with

This character is not used within the language.

Leaving them raw instead gives a quoted string spanning three physical lines,
which it rejects as an invalid multi-line string. Neither form is valid, so the
flattened output cannot be read back by Terraform whichever way it is written.

Why it is filed separately

The escapes-and-spans work in #339 makes the surrounding cases correct -- a
single-line interpolation now survives flattening and restoring intact -- but
it cannot fix this one, because there is no valid quoted spelling to produce.
The options are all judgement calls rather than corrections:

• collapse the interpolation onto one line, which is semantically identical for
  most expressions but not for one containing a # comment or a nested
  heredoc;
• leave the heredoc unflattened, which contradicts what the option promises;
• or raise, which turns a silent corruption into a loud failure but breaks
  callers who are flattening documents that contain one.

Worth deciding deliberately rather than by whichever fix lands first.

---

This issue, and the investigation behind it, were produced by an AI assistant (Claude) working on behalf of the author. Please review with that provenance in mind.

[livingstaccato]

A suggestion, since all three options here trade something different away.

My preference is to refuse the conversion and leave the heredoc alone, because it is the only option that never changes what a document means. preserve_heredocs=False would then have one documented case where it declines, in the same shape as the two refusals strings_to_heredocs already makes: a value that does not end in a newline stays quoted, and one carrying a lone carriage return stays quoted, both because the target form cannot express them. A multi-line interpolation is the same kind of thing — the quoted form cannot express it — so declining is consistent rather than a special case.

Collapsing the interpolation onto one line is tempting and mostly works, but "mostly" is doing real work there: a # comment inside the expression swallows everything after it on the joined line, and a nested heredoc cannot be collapsed at all. Both are rare, and both fail silently by producing a valid file that means something else, which is the worst failure mode of the three.

Raising is defensible, and better than the silent corruption in place today, but it breaks callers who are flattening documents that happen to contain one — including anyone using hcl2tojson in a pipeline, where the failure would be a stack trace rather than a diagnostic.

If declining is acceptable, the check is cheap: the span scanner added in #346 already identifies interpolation spans, so it is a test for a newline inside one.

[livingstaccato]

Superseded by #354, which takes the first of the three options above.

Updating this since the issue text says the choice is open and it no longer is, at least as a proposal: the PR declines the conversion and hands the heredoc back in the form preserve_heredocs=True produces, so the document round-trips unchanged. The two alternatives are still described above and still available — if either is preferred, the PR is small enough to redirect.

Worth restating what the current behaviour is, since the issue frames this as a choice between designs and one of them is "keep emitting output that cannot be read": preserve_heredocs=False writes "${\n 1 + 2\n}\n" today, and hcl2.loads of that raises UnexpectedToken. Whichever option is chosen, that part is a defect rather than a preference.