Skip to content

Fix typeOf failures with higher-rank types in effect handlers#875

Open
TimWhiting wants to merge 1 commit intokoka-lang:devfrom
TimWhiting:fix/typeOf-higher-rank
Open

Fix typeOf failures with higher-rank types in effect handlers#875
TimWhiting wants to merge 1 commit intokoka-lang:devfrom
TimWhiting:fix/typeOf-higher-rank

Conversation

@TimWhiting
Copy link
Copy Markdown
Collaborator

Summary

Fixes typeOf crash (KindMismatch assertion in subNew) when compiling recursive functions with higher-rank effect handler types.

Root cause

In inferRecDef2, the |~> (term substitution) replaces recursive Var references with the final generalized type, but leaves surrounding TypeApp arguments unchanged. This causes two issues:

  1. Forall variable ordering mismatch: The assumed type's forall variable order (from createGammas) can differ from the generalized type's order (from generalizeX using ofuv), because ofuv occurrence order depends on type structure which changes after unification.

  2. Forall variable count mismatch: When normalizeX closes effect tail variables (unifying open effect rows to total for closed definitions), the generalized type has fewer forall variables than the assumed type, but the TypeApp arguments still reflected the assumed type's count.

Fix

Replace the |~> substitution in all three inferRecDef2 cases (TypeLam, divergent, normal) with replaceRecCallEx, which matches TypeApp (Var name) [args] as a unit and replaces the entire construct with TypeApp newVar [correct_args], ensuring the TypeApp argument count and order always matches the Var's forall variables.

Test plan

  • stack exec koka -- -e test/cgen/type-of.kk prints "runs" (fresh .koka cache)
  • stack test --fast — 407 examples, 0 new failures (2 pre-existing unicode locale failures)
  • Clean rebuild with rm -rf .koka && mkdir .koka passes all tests

Supersedes #873. See also #872 for potential future capture-avoiding substitution improvement.

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings March 22, 2026 03:27
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a typeOf compilation crash (KindMismatch in subNew) triggered by recursive functions whose higher-rank effect-handler types cause TypeApp argument lists to get out of sync with the final generalized scheme.

Changes:

  • Update inferRecDef2 to replace recursive references via replaceRecCallEx, rewriting TypeApp (Var f) [...] as a unit to keep type-argument count/order aligned with the generalized Var scheme.
  • Add a new regression test exercising higher-rank handler types with recursion (test/cgen/type-of.kk) and its expected output.
  • Introduce replaceRecCallEx AST traversal helper in src/Type/Infer.hs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
src/Type/Infer.hs Replaces previous `(
test/cgen/type-of.kk Adds a regression program covering the higher-rank handler/recursive-call scenario.
test/cgen/type-of.kk.out Adds expected output for the new regression test.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/Type/Infer.hs
Comment thread src/Type/Infer.hs Outdated
Replace the |~> (term substitution) in inferRecDef2 with replaceRecCallEx,
which matches TypeApp (Var name) [args] as a unit and replaces the entire
construct with the correct type application. This fixes two issues:

1. Forall variable ordering mismatch: when the assumed type's forall order
   differs from the generalized type's order, the old |~> would replace
   the Var inside a TypeApp without updating the TypeApp arguments.

2. Forall variable count mismatch: when normalizeX closes effect tail
   variables (e.g., unifying open effect rows to total), the generalized
   type has fewer forall variables than the assumed type, but the TypeApp
   arguments still reflected the assumed type's variable count.

Both issues manifested as KindMismatch assertions in typeOf's TypeApp case.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@TimWhiting TimWhiting force-pushed the fix/typeOf-higher-rank branch from 0486899 to 7e6197b Compare March 22, 2026 03:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants