Skip to content

Fix sparse_keys function to bypass slow keys() call unless needed#3971

Open
idelder wants to merge 1 commit into
Pyomo:mainfrom
idelder:fix/sparse_keys_iteration
Open

Fix sparse_keys function to bypass slow keys() call unless needed#3971
idelder wants to merge 1 commit into
Pyomo:mainfrom
idelder:fix/sparse_keys_iteration

Conversation

@idelder
Copy link
Copy Markdown

@idelder idelder commented Jun 3, 2026

Fixes # .

Fix a very slow call from sparse_keys() to keys() for unsorted params.

Summary/Motivation:

The sparse_keys() function was updated to return ordered/sorted sparse indices when sorting was specified, going via the keys() function. The change, however, also routed unsorted parameters through the keys() function, collapsing to the final block where filtering was performed over the entire index cross product, keeping only sparse keys, but in index-insertion order. For very large but sparsely indexed params, this dramatically increased (by many orders of magnitude) the time to return the sparse keys set. For unsorted parameters, it makes sense to default back to old behaviour of just returning the unordered sparse keys immediately.

Changes proposed in this PR:

  • When the sort argument is not set (defaults to UNSORTED) or explicitly set to UNSORTED, just return the unordered sparse keys set as in previous pyomo versions.
  • When set to SORTED_INDICES, this will go via keys() to an explicit sorting of the sparse keys as was established in 6.10
  • When set to ORDERED_INDICES, collapses to the final block in keys() where the sparse keys set is slowly returned in index insertion order.

AI-Use Disclosure

  • AI tools were NOT used during the preparation of this PR

or

  • AI tools contributed to the development of this PR

    • AI tools generated documentation (including the PR description/comments, code comments, and/or Sphinx documentation)
    • AI tools generated tests (baselines, examples, and/or code)
    • AI tools generated code (apart from tests)

    Review process (select ONE):

    • Rewritten: All AI-generated content was rewritten by me before being committed.
    • Reviewed/verified: I retained AI-generated content and verified it before committing. Verification included (as applicable):
      • Ran the code and fixed issues
      • Added and ran tests
      • Checked correctness/logic of code and tests
      • Checked for alignment with the contribution guide
      • Considered security implications
    • As-is: AI-generated content was commited directly to the repository

Notes for reviewers (optional):
Does this correctly cover the different cases of sorting?
(This is my first PR to the pyomo codebase, forgive me if I am missing something)

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

  1. I agree my contributions are submitted under the BSD license.
  2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

Signed-off-by: Davey Elder <iandavidelder@gmail.com>
@jsiirola
Copy link
Copy Markdown
Member

jsiirola commented Jun 3, 2026

Initial thoughts on this PR:

  • This potentially changes things so that keys from sparse_keys and keys come in a different order (for the same arguments). The current implementation can be slow because we are making sure that the order that you see keys is consistent between the two methods. This is a potentially "breaking change" that will need to be discussed (likely at a developer call)
  • At a bare minimum, the ordering from sparse_keys(), sparse_values(), and sparse_items() must all match (for equivalent parameters) [this is for consistency with dict.{keys, values, items}]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants