feat: Add LumpFeatures transfomer - #1941
Closed
dylanw-oss wants to merge 3 commits into
Closed
Conversation
dylanw-oss
requested a review
from Mark Hamilton (mhamilton723)
as a code owner
April 26, 2023 02:19
|
Hey dylanw-oss 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
dylanw-oss
changed the title
Daniel Ciborowski (dciborow)
changed the title
Contributor
Author
|
if there is no objective for this feature, I'll add document. sarahshy Jason Wang (@memoryz) |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary by GPT-4
The LumpFeatures transformer is a custom transformer that takes a DataFrame and a list of lumping rules as input and returns a DataFrame comprised of the original columns, but the columns defined in lumping rules will be indexed and lumped to top k. This transformer can be used to handle high cardinality skewed categorical features before doing encoding.
In the given code, the LumpFeatures class extends Transformer and implements the following methods:
-
transform: This method takes an input dataset and applies the lumping rules to it. It first creates a pipeline with StringIndexer transformers for each column specified in the lumping rules. Then, it fits and transforms the input dataset using this pipeline. Finally, it keeps only the top k levels for each categorical column according to the lumping rules. -
transformSchema: This method returns the schema of the output DataFrame after applying the transformation. -
copy: This method creates a copy of this instance with extra parameters.
The test suite LumpFeaturesSuite tests this transformer's basic functionality by creating an input DataFrame with categorical columns, applying lumping rules using an instance of LumpFeatures, and comparing the output DataFrame with an expected result.
In summary, this custom transformer helps in handling high cardinality skewed categorical features by indexing and lumping them according to specified rules before encoding them.
Suggestions
The changes in this PR look good and no suggestions are needed.
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 1, 2026
## Summary Add dedicated transformer fuzzing coverage for LumpFeaturesModel so global experiment, serialization, Python, and R coverage gates recognize the persisted model. ## Prompting Intent Repair the concrete UnitTests core failure from PR microsoft#2596 after Azure build 229219360 reported that LumpFeaturesModel had no directly registered fuzzers, while preserving all estimator tests. ## Linked Sources - Pull request: microsoft#2596 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229219360 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 ## Rationale Register a real TransformerFuzzing test object instead of exempting the model. This exercises deterministic transforms and model persistence while generating Python and R correspondence coverage expected by the repository-wide FuzzingTest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 1, 2026
## Summary Persist each fitted top-K alongside retained values and reject incompatible LumpFeaturesModel lumpRules mutations across direct setters, generic generated-binding transfer, copy overrides, and loaded models. ## Prompting Intent Address the independent medium-severity API review finding on PR microsoft#2596 without removing API-compatible params. Ensure a fitted model can never silently score with learned values that disagree with a post-fit K, and cover persistence, copy, Scala, Java, JSON, and generated-binding paths. ## Linked Sources - Pull request: microsoft#2596 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 - Independent review finding supplied in the PR follow-up request - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Encode the fitted top-K inside the existing model-only keptValuesJson state instead of adding another generated mutable parameter. Direct model setters fail immediately, while transform-time state validation protects generic Param paths used by generated bindings. Exact no-op rule assignment remains allowed, and incompatible copy overrides fail before returning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 4, 2026
## Summary Add LumpFeatures as a Spark ML estimator with a persisted model, deterministic top-K learning, explicit other-bucket and null semantics, schema-safe transforms, generated bindings, and comprehensive tests. ## Prompting Intent Recreate the valuable proposal from GitHub PR microsoft#1941 for current SynapseML without fitting during transform. Preserve lumpRules compatibility while covering persistence, copy behavior, special column names, unseen values, collisions, and Scala-first Python code generation. ## Linked Sources - Original proposal PR: microsoft#1941 - Feature request: microsoft#1891 - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Learn category frequencies once in fit and persist only retained values so scoring is stable and side-effect free. Restrict v1 to string columns, rank ties by value, preserve nulls by default, and reject other-bucket collisions rather than silently merging real categories. Use Spark SQL expressions instead of UDFs and retain the multi-column lumpRules API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 4, 2026
## Summary Add dedicated transformer fuzzing coverage for LumpFeaturesModel so global experiment, serialization, Python, and R coverage gates recognize the persisted model. ## Prompting Intent Repair the concrete UnitTests core failure from PR microsoft#2596 after Azure build 229219360 reported that LumpFeaturesModel had no directly registered fuzzers, while preserving all estimator tests. ## Linked Sources - Pull request: microsoft#2596 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229219360 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 ## Rationale Register a real TransformerFuzzing test object instead of exempting the model. This exercises deterministic transforms and model persistence while generating Python and R correspondence coverage expected by the repository-wide FuzzingTest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 4, 2026
## Summary Persist each fitted top-K alongside retained values and reject incompatible LumpFeaturesModel lumpRules mutations across direct setters, generic generated-binding transfer, copy overrides, and loaded models. ## Prompting Intent Address the independent medium-severity API review finding on PR microsoft#2596 without removing API-compatible params. Ensure a fitted model can never silently score with learned values that disagree with a post-fit K, and cover persistence, copy, Scala, Java, JSON, and generated-binding paths. ## Linked Sources - Pull request: microsoft#2596 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 - Independent review finding supplied in the PR follow-up request - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Encode the fitted top-K inside the existing model-only keptValuesJson state instead of adding another generated mutable parameter. Direct model setters fail immediately, while transform-time state validation protects generic Param paths used by generated bindings. Exact no-op rule assignment remains allowed, and incompatible copy overrides fail before returning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 7, 2026
## Summary Add LumpFeatures as a Spark ML estimator with a persisted model, deterministic top-K learning, explicit other-bucket and null semantics, schema-safe transforms, generated bindings, and comprehensive tests. ## Prompting Intent Recreate the valuable proposal from GitHub PR microsoft#1941 for current SynapseML without fitting during transform. Preserve lumpRules compatibility while covering persistence, copy behavior, special column names, unseen values, collisions, and Scala-first Python code generation. ## Linked Sources - Original proposal PR: microsoft#1941 - Feature request: microsoft#1891 - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Learn category frequencies once in fit and persist only retained values so scoring is stable and side-effect free. Restrict v1 to string columns, rank ties by value, preserve nulls by default, and reject other-bucket collisions rather than silently merging real categories. Use Spark SQL expressions instead of UDFs and retain the multi-column lumpRules API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 7, 2026
## Summary Add dedicated transformer fuzzing coverage for LumpFeaturesModel so global experiment, serialization, Python, and R coverage gates recognize the persisted model. ## Prompting Intent Repair the concrete UnitTests core failure from PR microsoft#2596 after Azure build 229219360 reported that LumpFeaturesModel had no directly registered fuzzers, while preserving all estimator tests. ## Linked Sources - Pull request: microsoft#2596 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229219360 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 ## Rationale Register a real TransformerFuzzing test object instead of exempting the model. This exercises deterministic transforms and model persistence while generating Python and R correspondence coverage expected by the repository-wide FuzzingTest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 7, 2026
## Summary Persist each fitted top-K alongside retained values and reject incompatible LumpFeaturesModel lumpRules mutations across direct setters, generic generated-binding transfer, copy overrides, and loaded models. ## Prompting Intent Address the independent medium-severity API review finding on PR microsoft#2596 without removing API-compatible params. Ensure a fitted model can never silently score with learned values that disagree with a post-fit K, and cover persistence, copy, Scala, Java, JSON, and generated-binding paths. ## Linked Sources - Pull request: microsoft#2596 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 - Independent review finding supplied in the PR follow-up request - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Encode the fitted top-K inside the existing model-only keptValuesJson state instead of adding another generated mutable parameter. Direct model setters fail immediately, while transform-time state validation protects generic Param paths used by generated bindings. Exact no-op rule assignment remains allowed, and incompatible copy overrides fail before returning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 7, 2026
## Summary Persist the fit-time fallback inside learned categorical state, prevent all direct and generic mutation bypasses, keep copy and load behavior atomic, and make nullability match runtime output. ## Prompting Intent Rebase PR microsoft#2596 onto current master and resolve review findings while keeping top-K scoring lean, deterministic, persisted, and compatible with Spark ML copy/load and generated language bindings. ## Linked Sources - Pull request: microsoft#2596 - Original categorical lumping proposal: microsoft#1941 - Related issue: microsoft#1891 - Nullability review: microsoft#2596 (comment) ## Rationale The reserved fallback is stored with the existing learned-state JSON rather than as a separately clearable Spark Param, so generated bindings expose no internal escape hatch. A compact in-memory snapshot restores and rejects generic mutations, copy extras are checked before transfer, and legacy artifacts are upgraded from their persisted fallback. Validation occurs once per schema/transform rather than per row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 7, 2026
## Summary Add LumpFeatures as a Spark ML estimator with a persisted model, deterministic top-K learning, explicit other-bucket and null semantics, schema-safe transforms, generated bindings, and comprehensive tests. ## Prompting Intent Recreate the valuable proposal from GitHub PR microsoft#1941 for current SynapseML without fitting during transform. Preserve lumpRules compatibility while covering persistence, copy behavior, special column names, unseen values, collisions, and Scala-first Python code generation. ## Linked Sources - Original proposal PR: microsoft#1941 - Feature request: microsoft#1891 - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Learn category frequencies once in fit and persist only retained values so scoring is stable and side-effect free. Restrict v1 to string columns, rank ties by value, preserve nulls by default, and reject other-bucket collisions rather than silently merging real categories. Use Spark SQL expressions instead of UDFs and retain the multi-column lumpRules API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 7, 2026
## Summary Add dedicated transformer fuzzing coverage for LumpFeaturesModel so global experiment, serialization, Python, and R coverage gates recognize the persisted model. ## Prompting Intent Repair the concrete UnitTests core failure from PR microsoft#2596 after Azure build 229219360 reported that LumpFeaturesModel had no directly registered fuzzers, while preserving all estimator tests. ## Linked Sources - Pull request: microsoft#2596 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229219360 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 ## Rationale Register a real TransformerFuzzing test object instead of exempting the model. This exercises deterministic transforms and model persistence while generating Python and R correspondence coverage expected by the repository-wide FuzzingTest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 7, 2026
## Summary Persist each fitted top-K alongside retained values and reject incompatible LumpFeaturesModel lumpRules mutations across direct setters, generic generated-binding transfer, copy overrides, and loaded models. ## Prompting Intent Address the independent medium-severity API review finding on PR microsoft#2596 without removing API-compatible params. Ensure a fitted model can never silently score with learned values that disagree with a post-fit K, and cover persistence, copy, Scala, Java, JSON, and generated-binding paths. ## Linked Sources - Pull request: microsoft#2596 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 - Independent review finding supplied in the PR follow-up request - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Encode the fitted top-K inside the existing model-only keptValuesJson state instead of adding another generated mutable parameter. Direct model setters fail immediately, while transform-time state validation protects generic Param paths used by generated bindings. Exact no-op rule assignment remains allowed, and incompatible copy overrides fail before returning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 7, 2026
## Summary Persist the fit-time fallback inside learned categorical state, prevent all direct and generic mutation bypasses, keep copy and load behavior atomic, and make nullability match runtime output. ## Prompting Intent Rebase PR microsoft#2596 onto current master and resolve review findings while keeping top-K scoring lean, deterministic, persisted, and compatible with Spark ML copy/load and generated language bindings. ## Linked Sources - Pull request: microsoft#2596 - Original categorical lumping proposal: microsoft#1941 - Related issue: microsoft#1891 - Nullability review: microsoft#2596 (comment) ## Rationale The reserved fallback is stored with the existing learned-state JSON rather than as a separately clearable Spark Param, so generated bindings expose no internal escape hatch. A compact in-memory snapshot restores and rejects generic mutations, copy extras are checked before transfer, and legacy artifacts are upgraded from their persisted fallback. Validation occurs once per schema/transform rather than per row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 9, 2026
## Summary Add LumpFeatures as a Spark ML estimator with a persisted model, deterministic top-K learning, explicit other-bucket and null semantics, schema-safe transforms, generated bindings, and comprehensive tests. ## Prompting Intent Recreate the valuable proposal from GitHub PR microsoft#1941 for current SynapseML without fitting during transform. Preserve lumpRules compatibility while covering persistence, copy behavior, special column names, unseen values, collisions, and Scala-first Python code generation. ## Linked Sources - Original proposal PR: microsoft#1941 - Feature request: microsoft#1891 - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Learn category frequencies once in fit and persist only retained values so scoring is stable and side-effect free. Restrict v1 to string columns, rank ties by value, preserve nulls by default, and reject other-bucket collisions rather than silently merging real categories. Use Spark SQL expressions instead of UDFs and retain the multi-column lumpRules API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 9, 2026
## Summary Add dedicated transformer fuzzing coverage for LumpFeaturesModel so global experiment, serialization, Python, and R coverage gates recognize the persisted model. ## Prompting Intent Repair the concrete UnitTests core failure from PR microsoft#2596 after Azure build 229219360 reported that LumpFeaturesModel had no directly registered fuzzers, while preserving all estimator tests. ## Linked Sources - Pull request: microsoft#2596 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229219360 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 ## Rationale Register a real TransformerFuzzing test object instead of exempting the model. This exercises deterministic transforms and model persistence while generating Python and R correspondence coverage expected by the repository-wide FuzzingTest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 9, 2026
## Summary Persist each fitted top-K alongside retained values and reject incompatible LumpFeaturesModel lumpRules mutations across direct setters, generic generated-binding transfer, copy overrides, and loaded models. ## Prompting Intent Address the independent medium-severity API review finding on PR microsoft#2596 without removing API-compatible params. Ensure a fitted model can never silently score with learned values that disagree with a post-fit K, and cover persistence, copy, Scala, Java, JSON, and generated-binding paths. ## Linked Sources - Pull request: microsoft#2596 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 - Independent review finding supplied in the PR follow-up request - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Encode the fitted top-K inside the existing model-only keptValuesJson state instead of adding another generated mutable parameter. Direct model setters fail immediately, while transform-time state validation protects generic Param paths used by generated bindings. Exact no-op rule assignment remains allowed, and incompatible copy overrides fail before returning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 9, 2026
## Summary Persist the fit-time fallback inside learned categorical state, prevent all direct and generic mutation bypasses, keep copy and load behavior atomic, and make nullability match runtime output. ## Prompting Intent Rebase PR microsoft#2596 onto current master and resolve review findings while keeping top-K scoring lean, deterministic, persisted, and compatible with Spark ML copy/load and generated language bindings. ## Linked Sources - Pull request: microsoft#2596 - Original categorical lumping proposal: microsoft#1941 - Related issue: microsoft#1891 - Nullability review: microsoft#2596 (comment) ## Rationale The reserved fallback is stored with the existing learned-state JSON rather than as a separately clearable Spark Param, so generated bindings expose no internal escape hatch. A compact in-memory snapshot restores and rejects generic mutations, copy extras are checked before transfer, and legacy artifacts are upgraded from their persisted fallback. Validation occurs once per schema/transform rather than per row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 10, 2026
## Summary Add LumpFeatures as a Spark ML estimator with a persisted model, deterministic top-K learning, explicit other-bucket and null semantics, schema-safe transforms, generated bindings, and comprehensive tests. ## Prompting Intent Recreate the valuable proposal from GitHub PR microsoft#1941 for current SynapseML without fitting during transform. Preserve lumpRules compatibility while covering persistence, copy behavior, special column names, unseen values, collisions, and Scala-first Python code generation. ## Linked Sources - Original proposal PR: microsoft#1941 - Feature request: microsoft#1891 - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Learn category frequencies once in fit and persist only retained values so scoring is stable and side-effect free. Restrict v1 to string columns, rank ties by value, preserve nulls by default, and reject other-bucket collisions rather than silently merging real categories. Use Spark SQL expressions instead of UDFs and retain the multi-column lumpRules API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 10, 2026
## Summary Add dedicated transformer fuzzing coverage for LumpFeaturesModel so global experiment, serialization, Python, and R coverage gates recognize the persisted model. ## Prompting Intent Repair the concrete UnitTests core failure from PR microsoft#2596 after Azure build 229219360 reported that LumpFeaturesModel had no directly registered fuzzers, while preserving all estimator tests. ## Linked Sources - Pull request: microsoft#2596 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229219360 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 ## Rationale Register a real TransformerFuzzing test object instead of exempting the model. This exercises deterministic transforms and model persistence while generating Python and R correspondence coverage expected by the repository-wide FuzzingTest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 10, 2026
## Summary Persist each fitted top-K alongside retained values and reject incompatible LumpFeaturesModel lumpRules mutations across direct setters, generic generated-binding transfer, copy overrides, and loaded models. ## Prompting Intent Address the independent medium-severity API review finding on PR microsoft#2596 without removing API-compatible params. Ensure a fitted model can never silently score with learned values that disagree with a post-fit K, and cover persistence, copy, Scala, Java, JSON, and generated-binding paths. ## Linked Sources - Pull request: microsoft#2596 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 - Independent review finding supplied in the PR follow-up request - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Encode the fitted top-K inside the existing model-only keptValuesJson state instead of adding another generated mutable parameter. Direct model setters fail immediately, while transform-time state validation protects generic Param paths used by generated bindings. Exact no-op rule assignment remains allowed, and incompatible copy overrides fail before returning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 10, 2026
## Summary Persist the fit-time fallback inside learned categorical state, prevent all direct and generic mutation bypasses, keep copy and load behavior atomic, and make nullability match runtime output. ## Prompting Intent Rebase PR microsoft#2596 onto current master and resolve review findings while keeping top-K scoring lean, deterministic, persisted, and compatible with Spark ML copy/load and generated language bindings. ## Linked Sources - Pull request: microsoft#2596 - Original categorical lumping proposal: microsoft#1941 - Related issue: microsoft#1891 - Nullability review: microsoft#2596 (comment) ## Rationale The reserved fallback is stored with the existing learned-state JSON rather than as a separately clearable Spark Param, so generated bindings expose no internal escape hatch. A compact in-memory snapshot restores and rejects generic mutations, copy extras are checked before transfer, and legacy artifacts are upgraded from their persisted fallback. Validation occurs once per schema/transform rather than per row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 11, 2026
## Summary Add LumpFeatures as a Spark ML estimator with a persisted model, deterministic top-K learning, explicit other-bucket and null semantics, schema-safe transforms, generated bindings, and comprehensive tests. ## Prompting Intent Recreate the valuable proposal from GitHub PR microsoft#1941 for current SynapseML without fitting during transform. Preserve lumpRules compatibility while covering persistence, copy behavior, special column names, unseen values, collisions, and Scala-first Python code generation. ## Linked Sources - Original proposal PR: microsoft#1941 - Feature request: microsoft#1891 - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Learn category frequencies once in fit and persist only retained values so scoring is stable and side-effect free. Restrict v1 to string columns, rank ties by value, preserve nulls by default, and reject other-bucket collisions rather than silently merging real categories. Use Spark SQL expressions instead of UDFs and retain the multi-column lumpRules API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 11, 2026
## Summary Add dedicated transformer fuzzing coverage for LumpFeaturesModel so global experiment, serialization, Python, and R coverage gates recognize the persisted model. ## Prompting Intent Repair the concrete UnitTests core failure from PR microsoft#2596 after Azure build 229219360 reported that LumpFeaturesModel had no directly registered fuzzers, while preserving all estimator tests. ## Linked Sources - Pull request: microsoft#2596 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229219360 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 ## Rationale Register a real TransformerFuzzing test object instead of exempting the model. This exercises deterministic transforms and model persistence while generating Python and R correspondence coverage expected by the repository-wide FuzzingTest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 11, 2026
## Summary Persist each fitted top-K alongside retained values and reject incompatible LumpFeaturesModel lumpRules mutations across direct setters, generic generated-binding transfer, copy overrides, and loaded models. ## Prompting Intent Address the independent medium-severity API review finding on PR microsoft#2596 without removing API-compatible params. Ensure a fitted model can never silently score with learned values that disagree with a post-fit K, and cover persistence, copy, Scala, Java, JSON, and generated-binding paths. ## Linked Sources - Pull request: microsoft#2596 - Original proposal: microsoft#1941 - Feature request: microsoft#1891 - Independent review finding supplied in the PR follow-up request - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Encode the fitted top-K inside the existing model-only keptValuesJson state instead of adding another generated mutable parameter. Direct model setters fail immediately, while transform-time state validation protects generic Param paths used by generated bindings. Exact no-op rule assignment remains allowed, and incompatible copy overrides fail before returning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 11, 2026
## Summary Persist the fit-time fallback inside learned categorical state, prevent all direct and generic mutation bypasses, keep copy and load behavior atomic, and make nullability match runtime output. ## Prompting Intent Rebase PR microsoft#2596 onto current master and resolve review findings while keeping top-K scoring lean, deterministic, persisted, and compatible with Spark ML copy/load and generated language bindings. ## Linked Sources - Pull request: microsoft#2596 - Original categorical lumping proposal: microsoft#1941 - Related issue: microsoft#1891 - Nullability review: microsoft#2596 (comment) ## Rationale The reserved fallback is stored with the existing learned-state JSON rather than as a separately clearable Spark Param, so generated bindings expose no internal escape hatch. A compact in-memory snapshot restores and rejects generic mutations, copy extras are checked before transfer, and legacy artifacts are upgraded from their persisted fallback. Validation occurs once per schema/transform rather than per row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128
Rana Singh (ranadeepsingh)
added a commit
that referenced
this pull request
Aug 11, 2026
* feat: add persisted categorical top-K lumping ## Summary Add LumpFeatures as a Spark ML estimator with a persisted model, deterministic top-K learning, explicit other-bucket and null semantics, schema-safe transforms, generated bindings, and comprehensive tests. ## Prompting Intent Recreate the valuable proposal from GitHub PR #1941 for current SynapseML without fitting during transform. Preserve lumpRules compatibility while covering persistence, copy behavior, special column names, unseen values, collisions, and Scala-first Python code generation. ## Linked Sources - Original proposal PR: #1941 - Feature request: #1891 - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Learn category frequencies once in fit and persist only retained values so scoring is stable and side-effect free. Restrict v1 to string columns, rank ties by value, preserve nulls by default, and reject other-bucket collisions rather than silently merging real categories. Use Spark SQL expressions instead of UDFs and retain the multi-column lumpRules API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: register LumpFeaturesModel fuzzers ## Summary Add dedicated transformer fuzzing coverage for LumpFeaturesModel so global experiment, serialization, Python, and R coverage gates recognize the persisted model. ## Prompting Intent Repair the concrete UnitTests core failure from PR #2596 after Azure build 229219360 reported that LumpFeaturesModel had no directly registered fuzzers, while preserving all estimator tests. ## Linked Sources - Pull request: #2596 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229219360 - Original proposal: #1941 - Feature request: #1891 ## Rationale Register a real TransformerFuzzing test object instead of exempting the model. This exercises deterministic transforms and model persistence while generating Python and R correspondence coverage expected by the repository-wide FuzzingTest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: reject fitted LumpFeatures rule changes ## Summary Persist each fitted top-K alongside retained values and reject incompatible LumpFeaturesModel lumpRules mutations across direct setters, generic generated-binding transfer, copy overrides, and loaded models. ## Prompting Intent Address the independent medium-severity API review finding on PR #2596 without removing API-compatible params. Ensure a fitted model can never silently score with learned values that disagree with a post-fit K, and cover persistence, copy, Scala, Java, JSON, and generated-binding paths. ## Linked Sources - Pull request: #2596 - Original proposal: #1941 - Feature request: #1891 - Independent review finding supplied in the PR follow-up request - No Azure DevOps work item was supplied; tracking is through the linked GitHub issue. ## Rationale Encode the fitted top-K inside the existing model-only keptValuesJson state instead of adding another generated mutable parameter. Direct model setters fail immediately, while transform-time state validation protects generic Param paths used by generated bindings. Exact no-op rule assignment remains allowed, and incompatible copy overrides fail before returning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: make LumpFeatures a reliable ML API and algorithm Rework the persisted top-K lumping stage so it is dependable as a training feature-engineering step, not just correct on a single-column happy path. Algorithm - Add minCount and minFreq eligibility filters, applied before the lumpRules top-K cap. This matches the order scikit-learn's OneHotEncoder uses for min_frequency and max_categories, and R forcats / feature-engine use the same primary-threshold + secondary-cap shape. Top-K alone is blind to the distribution: it lumps healthy levels in a low-cardinality column, and in a long-tailed column it retains values covering almost none of the rows. Both default to no-ops, so existing behaviour is unchanged. - Rewrite fit as a single pass. The rule columns are melted into (column, value) pairs and ranked with one windowed aggregation instead of one full scan per column plus a separate collision scan. Besides cutting fit from N+1 Spark jobs to 1, this guarantees every column is learned from the same materialization of the input; per-column jobs silently learn from different rows when the upstream plan is non-deterministic (sample, rand, unordered limit). API - Add an optional outputCols map so lumped values can be written to new columns instead of destroying the raw ones. Unset means in-place, so the default is unchanged. Destinations are validated (known source, non-empty, distinct, not already in the input schema) and ordered deterministically so transformSchema always matches transform. - Fix the nullability contract: the declared schema now derives nullability from handleNull alone instead of intersecting it with the input column's nullability, so a non-nullable input under handleNull='keep' no longer declares a non-nullable output that the expression may not honour. This is the reviewer comment on the PR; the transform expression uses a typed null literal so declared and actual schemas agree on non-nullable inputs too. - Expose the learned values to Python. LumpFeaturesModel becomes an internal wrapper with a hand-written override providing getKeptValues(), backed by a new getKeptValuesAsJson on the Scala model, so a fit can be audited from Python instead of being an opaque JSON param. Docs and tests - Document LumpFeatures in docs/Quick Examples with runnable Python and Scala examples; the stage was previously absent from the stages doc table. - Add 13 tests covering the frequency filters and their ordering against the cap, per-column denominators, joint vs single-column fit agreement, outputCols behaviour/validation/round-trip, the non-nullable schema contract, and the JSON accessor. 41 tests pass, scalastyle clean on main and test, codegen and black clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a108ab7-6879-4fa6-81de-ef2d43eb3ec5 * fix(stages): enforce fitted LumpFeatures model invariants ## Summary Persist the fit-time fallback inside learned categorical state, prevent all direct and generic mutation bypasses, keep copy and load behavior atomic, and make nullability match runtime output. ## Prompting Intent Rebase PR #2596 onto current master and resolve review findings while keeping top-K scoring lean, deterministic, persisted, and compatible with Spark ML copy/load and generated language bindings. ## Linked Sources - Pull request: #2596 - Original categorical lumping proposal: #1941 - Related issue: #1891 - Nullability review: #2596 (comment) ## Rationale The reserved fallback is stored with the existing learned-state JSON rather than as a separately clearable Spark Param, so generated bindings expose no internal escape hatch. A compact in-memory snapshot restores and rejects generic mutations, copy extras are checked before transfer, and legacy artifacts are upgraded from their persisted fallback. Validation occurs once per schema/transform rather than per row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Copilot-Session: 3a108ab7-6879-4fa6-81de-ef2d43eb3ec5 Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128
Collaborator
|
Thanks for this proposal, dylanw-oss — the idea and the API shape you designed here were both good, and they've now landed. #2596 was merged in cd9ab7b and implements categorical lumping for the current codebase, keeping the multi-column It's a fresh implementation against today's Closing this as superseded by #2596. Credit for the design belongs here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues/PRs
#1891
What changes are proposed in this pull request?
A transformer can be used to handle data with high cardinality skewed categorical before doing other featurization processing.
How is this patch tested?
unit test
Does this PR add a new feature? If so, have you added samples on website?
will add document in next commit (I'd like to ensure it makes sense before doing the next step)
website/docs/documentationfolder.Make sure you choose the correct class
estimators/transformersand namespace.DocTablepoints to correct API link.yarn run startto make sure the website renders correctly.<!--pytest-codeblocks:cont-->before each python code blocks to enable auto-tests for python samples.WebsiteSamplesTestsjob pass in the pipeline.