Improve ComponentMap / ComponentSet by jsiirola · Pull Request #3970 · Pyomo/pyomo

jsiirola · 2026-06-02T23:36:56Z

Fixes #755

Summary/Motivation:

This PR Updates COmponentMap / ComponentSet to prevent key collisions between ints and the id() of unhashable Components. The final solution proposed here is to store all keys coming from id() as a 2-tuple (combined with an internal class "flag"). This ended up being significantly more efficient than building a specialized HashKey class that would store the id() and not compare equal to int.

These changes make ComponentMap / ComponentSet slightly slower (~10-12%). To mitigate this, the PR includes two new containers, 'ObjectIdMap and ObjectIdSet. These containers are more efficient than the ComponentMap / ComponentSet, and approach the efficiency of a direct (manual) "dict-of-ids" implementation (see below).

As part of developing this PR, several other improvements were implemented:

to ComponentMap:
- return formal "view" objects from .keys(), .values(), and .items() to match the behavior of dict
- make KeyErrors raised here match those raised by dict
- simplify the string representation
- improve the robustness when comparing one ComponentMap to another
- make .update() API / behavior match dict
to ComponentSet:
- simplify the string representation
- improve the robustness when comparing one ComponentSet to another
- make .update() API / behavior match set
Update the ExpressionReplacementVisitor to officially accept ComponentMap for the substitution map

Performance data

Given the following test code:

m = ConcreteModel()
m.x = Var(range(100))
m.c = Constraint(range(100), rule=lambda m, i: m.x[i] >= i)

cm2 = ComponentMap()
cs2 = ComponentSet()
d2 = {}
s2 = set()
for _ in m.x.values():
    cm2[_] = 1
    cs2.add(_)
    d2[id(_)] = 1
    s2.add(id(_))
for _ in m.c.values():
    cm2[_] = 1
    cs2.add(_)
    d2[_] = 1
    s2.add(_)
for _ in range(100):
    cm2[_] = 1
    cs2.add(_)
    d2[_] = 1
    s2.add(_)

The original ComponentMap / ComponentSet implementation (from main), tested with 100k shots on Python 3.10 - 3.14:

    3.10    3.11    3.12    3.13    3.14   test
   3.536   2.685   2.867   2.782   2.611   'for _ in m.x.values():\n  cm[_] = _'
   3.379   2.224   2.360   2.291   2.158   'for _ in m.x.values():\n  cs.add(_)'
   1.725   1.424   1.546   1.502   1.491   'for _ in m.x.values():\n  d[id(_)] = _'
   2.029   1.711   1.794   1.707   1.697   'for _ in m.x.values():\n  d[id(_)] = (_,_)'
   1.827   1.405   1.555   1.503   1.511   'for _ in m.x.values():\n  s.add(id(_))'
   3.021   2.219   2.330   2.275   2.044   'for _ in m.c.values():\n  cm[_] = _'
   2.783   1.757   1.818   1.806   1.598   'for _ in m.c.values():\n  cs.add(_)'
   1.125   0.964   1.032   1.023   0.986   'for _ in m.c.values():\n  d[_] = _'
   1.227   0.948   1.010   0.992   0.979   'for _ in m.c.values():\n  s.add(_)'
   2.036   1.431   1.460   1.401   1.239   'for _ in range(100):\n  cm[_] = _'
   2.064   0.954   0.951   0.947   0.772   'for _ in range(100):\n  cs.add(_)'
   0.309   0.221   0.217   0.236   0.205   'for _ in range(100):\n  d[_] = _'
   0.387   0.203   0.191   0.212   0.204   'for _ in range(100):\n  s.add(_)'
   3.479   2.232   2.435   2.285   2.274   'for _ in m.x.values():\n  x = cm2[_]'
   1.662   1.395   1.508   1.467   1.472   'for _ in m.x.values():\n  x = d2[id(_)]'
   2.881   1.773   1.892   1.843   1.789   'for _ in m.c.values():\n  x = cm2[_]'
   1.111   0.977   1.021   1.037   0.981   'for _ in m.c.values():\n  x = d2[_]'
   1.993   1.066   1.047   1.033   0.861   'for _ in range(100):\n  x = cm2[_]'
   0.269   0.252   0.233   0.270   0.255   'for _ in range(100):\n  x = d2[_]'
   3.233   2.329   2.624   2.468   2.376   'for _ in m.x.values():\n  x = _ in cm2'
   3.185   2.360   2.616   2.462   2.348   'for _ in m.x.values():\n  x = _ in cs2'
   1.694   1.425   1.542   1.480   1.460   'for _ in m.x.values():\n  x = id(_) in d2'
   1.665   1.386   1.514   1.481   1.464   'for _ in m.x.values():\n  x = id(_) in s2'
   2.751   1.898   2.085   2.026   1.838   'for _ in m.c.values():\n  x = _ in cm2'
   2.661   1.896   2.081   2.022   1.821   'for _ in m.c.values():\n  x = _ in cs2'
   1.131   0.995   1.035   1.019   0.984   'for _ in m.c.values():\n  x = _ in d2'
   1.098   0.960   1.003   0.998   0.970   'for _ in m.c.values():\n  x = _ in s2'
   1.840   1.149   1.271   1.227   1.087   'for _ in range(100):\n  x = _ in cm2'
   1.823   1.156   1.277   1.221   1.141   'for _ in range(100):\n  x = _ in cs2'
   0.292   0.270   0.235   0.260   0.233   'for _ in range(100):\n  x = _ in d2'
   0.342   0.291   0.284   0.290   0.274   'for _ in range(100):\n  x = _ in s2'
   0.055   0.045   0.048   0.047   0.045   'for _ in cm.keys():\n  pass'
   0.054   0.045   0.047   0.046   0.045   'for _ in cm.values():\n  pass'
   0.013   0.009   0.011   0.011   0.012   'for _ in cs:\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.keys():\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.values():\n  pass'
   0.004   0.003   0.003   0.004   0.004   'for _ in s:\n  pass'

Results for the same tests on this PR:

    3.10    3.11    3.12    3.13    3.14   test
   3.968   3.140   3.238   3.165   2.937   'for _ in m.x.values():\n  cm[_] = _'
   3.771   2.693   2.740   2.707   2.467   'for _ in m.x.values():\n  cs.add(_)'
   1.664   1.444   1.548   1.505   1.494   'for _ in m.x.values():\n  d[id(_)] = _'
   1.981   1.733   1.802   1.692   1.706   'for _ in m.x.values():\n  d[id(_)] = (_,_)'
   1.816   1.424   1.581   1.460   1.516   'for _ in m.x.values():\n  s.add(id(_))'
   2.954   2.306   2.343   2.276   2.094   'for _ in m.c.values():\n  cm[_] = _'
   2.757   1.826   1.835   1.794   1.622   'for _ in m.c.values():\n  cs.add(_)'
   1.123   0.959   1.028   1.015   1.005   'for _ in m.c.values():\n  d[_] = _'
   1.219   0.948   1.015   0.989   0.987   'for _ in m.c.values():\n  s.add(_)'
   2.036   1.439   1.465   1.393   1.233   'for _ in range(100):\n  cm[_] = _'
   2.045   0.951   0.950   0.943   0.773   'for _ in range(100):\n  cs.add(_)'
   0.314   0.221   0.213   0.233   0.209   'for _ in range(100):\n  d[_] = _'
   0.398   0.203   0.191   0.210   0.207   'for _ in range(100):\n  s.add(_)'
   3.872   2.722   2.733   2.655   2.472   'for _ in m.x.values():\n  x = cm2[_]'
   1.619   1.408   1.499   1.457   1.481   'for _ in m.x.values():\n  x = d2[id(_)]'
   2.815   1.895   1.851   1.887   1.649   'for _ in m.c.values():\n  x = cm2[_]'
   1.109   0.983   1.013   1.022   1.008   'for _ in m.c.values():\n  x = d2[_]'
   1.991   1.115   1.070   0.959   0.863   'for _ in range(100):\n  x = cm2[_]'
   0.265   0.264   0.228   0.259   0.246   'for _ in range(100):\n  x = d2[_]'
   3.539   2.802   2.964   2.854   2.771   'for _ in m.x.values():\n  x = _ in cm2'
   3.615   2.818   2.955   2.858   2.702   'for _ in m.x.values():\n  x = _ in cs2'
   1.651   1.435   1.516   1.461   1.465   'for _ in m.x.values():\n  x = id(_) in d2'
   1.612   1.397   1.469   1.419   1.440   'for _ in m.x.values():\n  x = id(_) in s2'
   2.615   1.976   2.078   2.017   1.902   'for _ in m.c.values():\n  x = _ in cm2'
   2.603   1.979   2.081   2.018   1.865   'for _ in m.c.values():\n  x = _ in cs2'
   1.126   1.004   1.035   1.013   1.014   'for _ in m.c.values():\n  x = _ in d2'
   1.079   0.959   1.000   0.973   0.960   'for _ in m.c.values():\n  x = _ in s2'
   1.781   1.208   1.302   1.189   1.095   'for _ in range(100):\n  x = _ in cm2'
   1.777   1.209   1.270   1.190   1.091   'for _ in range(100):\n  x = _ in cs2'
   0.285   0.272   0.236   0.255   0.231   'for _ in range(100):\n  x = _ in d2'
   0.337   0.298   0.288   0.271   0.274   'for _ in range(100):\n  x = _ in s2'
   0.046   0.036   0.040   0.038   0.039   'for _ in cm.keys():\n  pass'
   0.047   0.037   0.041   0.039   0.040   'for _ in cm.values():\n  pass'
   0.013   0.009   0.011   0.011   0.012   'for _ in cs:\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.keys():\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.values():\n  pass'
   0.003   0.003   0.003   0.004   0.004   'for _ in s:\n  pass'

And if we use ObjectIdMap() / ObjectIdSet in lieu of the ComponentMap / ComponentSet:

    3.10    3.11    3.12    3.13    3.14   test
   2.553   2.189   2.321   2.222   2.169   'for _ in m.x.values():\n  cm[_] = _'
   2.207   1.718   1.818   1.748   1.778   'for _ in m.x.values():\n  cs.add(_)'
   1.680   1.431   1.546   1.503   1.489   'for _ in m.x.values():\n  d[id(_)] = _'
   2.000   1.727   1.787   1.705   1.695   'for _ in m.x.values():\n  d[id(_)] = (_,_)'
   1.815   1.420   1.525   1.480   1.505   'for _ in m.x.values():\n  s.add(id(_))'
   2.539   2.159   2.322   2.230   2.170   'for _ in m.c.values():\n  cm[_] = _'
   2.209   1.716   1.823   1.745   1.784   'for _ in m.c.values():\n  cs.add(_)'
   1.125   0.958   1.030   1.032   1.006   'for _ in m.c.values():\n  d[_] = _'
   1.223   0.959   1.013   0.993   0.994   'for _ in m.c.values():\n  s.add(_)'
   1.635   1.333   1.392   1.312   1.234   'for _ in range(100):\n  cm[_] = _'
   1.371   0.880   0.906   0.850   0.760   'for _ in range(100):\n  cs.add(_)'
   0.304   0.222   0.214   0.236   0.209   'for _ in range(100):\n  d[_] = _'
   0.382   0.203   0.194   0.211   0.208   'for _ in range(100):\n  s.add(_)'
   2.322   1.718   1.832   1.730   1.725   'for _ in m.x.values():\n  x = cm2[_]'
   1.640   1.403   1.498   1.462   1.475   'for _ in m.x.values():\n  x = d2[id(_)]'
   2.598   1.762   1.890   1.781   1.778   'for _ in m.c.values():\n  x = cm2[_]'
   1.252   0.974   1.027   1.030   0.997   'for _ in m.c.values():\n  x = d2[_]'
   1.490   0.914   0.957   0.929   0.889   'for _ in range(100):\n  x = cm2[_]'
   0.270   0.260   0.235   0.267   0.245   'for _ in range(100):\n  x = d2[_]'
   2.161   1.812   2.039   1.926   1.955   'for _ in m.x.values():\n  x = _ in cm2'
   2.190   1.829   2.023   1.926   1.910   'for _ in m.x.values():\n  x = _ in cs2'
   1.856   1.438   1.519   1.459   1.469   'for _ in m.x.values():\n  x = id(_) in d2'
   1.826   1.410   1.482   1.415   1.484   'for _ in m.x.values():\n  x = id(_) in s2'
   2.462   1.880   2.077   1.990   1.967   'for _ in m.c.values():\n  x = _ in cm2'
   2.413   1.851   2.078   1.989   1.959   'for _ in m.c.values():\n  x = _ in cs2'
   1.274   1.007   1.037   1.022   1.002   'for _ in m.c.values():\n  x = _ in d2'
   1.228   0.959   1.004   0.979   0.967   'for _ in m.c.values():\n  x = _ in s2'
   1.343   1.028   1.211   1.180   1.110   'for _ in range(100):\n  x = _ in cm2'
   1.342   1.038   1.203   1.177   1.090   'for _ in range(100):\n  x = _ in cs2'
   0.297   0.277   0.239   0.258   0.236   'for _ in range(100):\n  x = _ in d2'
   0.335   0.294   0.287   0.273   0.277   'for _ in range(100):\n  x = _ in s2'
   0.046   0.036   0.040   0.040   0.040   'for _ in cm.keys():\n  pass'
   0.046   0.036   0.040   0.039   0.039   'for _ in cm.values():\n  pass'
   0.013   0.009   0.011   0.011   0.012   'for _ in cs:\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.keys():\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.values():\n  pass'
   0.003   0.003   0.003   0.004   0.004   'for _ in s:\n  pass'

Changes proposed in this PR:

(see above)

AI-Use Disclosure

AI tools were NOT used during the preparation of this PR

or

Notes for reviewers (optional):

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

…n necessary

…nt is not callable

…tVisitor

codecov · 2026-06-03T00:40:30Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.12%. Comparing base (44c4697) to head (5ebac4b).

Additional details and impacted files

@@           Coverage Diff            @@
##             main    #3970    +/-   ##
========================================
  Coverage   90.12%   90.12%            
========================================
  Files         909      909            
  Lines      108561   108676   +115     
========================================
+ Hits        97836    97947   +111     
- Misses      10725    10729     +4

Flag	Coverage Δ
builders	`29.13% <52.17%> (+0.02%)`	⬆️
default	`86.12% <100.00%> (?)`
expensive	`35.15% <57.76%> (?)`
linux	`87.64% <100.00%> (-1.98%)`	⬇️
linux_other	`87.64% <100.00%> (+0.01%)`	⬆️
oldsolvers	`28.06% <52.17%> (+0.02%)`	⬆️
osx	`83.07% <100.00%> (+0.01%)`	⬆️
win	`85.38% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jsiirola added 30 commits February 8, 2026 13:26

Return HashKey fro HashDispatcher (and not id())

c6389b8

Improve tuple handling in hash dispatcher: only generate HashKeys whe…

6e10267

…n necessary

Minor cleanup

ad99ee0

Make HashDispatcher callable (support use in place of id())

6f793d9

Minor ComponentMap performance improvements

6ba2426

Support deriving from ComponentMap/ComponentSet

59f7ba2

Standardize str() and KeyError for COmponentMap/Set

fe4d4da

Add ObjectIdMap/ObjectIdSet

c2238d3

Minor code cleanup/fixes

e60cabb

NFC: update comments/docstrings

e3c8cbd

Support cleaner fallback to ComponentMap when first positional argume…

9b3e036

…nt is not callable

Add / update tests

bfd5092

Merge branch 'main' into component-map-hashing

4cc18e8

NFC: apply black

a865f03

Avoid resolving expressions created during ComponentMap comparisons

c8823a7

Generate more human-readable str(ComponentMap)/str(COmponentSet)

dcee582

ComponetMap keys/values/items should return views

156557a

Merge branch 'main' into component-map-hashing

5458af6

Merge branch 'main' into component-map-hashing

bd7835e

Improve/test HashKey repr/str

ad1fa4f

NFC: update docs

facfa68

Make update() API consistent with dict/set

7560771

Fix typo

1a4fecc

We don't need a full HashKey class; just a flag to add in a native tuple

5c1dd71

Support ComponentMap as the substitution map for ExpressionReplacemen…

484b6f7

…tVisitor

Don't look up ComponentMap values by key id()

11cc469

Track updated exception message

7ee41f0

Catch that comparing numpy types can produce numpy bools

c455f6b

Guard against object collection under ExpressionReplacementVisitor

9365fff

Update to track simplification of string representations

295d8b6

Expose ObjectIdMap / ObjectIdSet through pyomo.common.collections

a6ac21f

Update test for pypy

5ebac4b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve ComponentMap / ComponentSet#3970

Improve ComponentMap / ComponentSet#3970
jsiirola wants to merge 32 commits into
Pyomo:mainfrom
jsiirola:component-map-hashing

jsiirola commented Jun 2, 2026

Uh oh!

codecov Bot commented Jun 3, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jsiirola commented Jun 2, 2026

Fixes #755

Summary/Motivation:

Performance data

Changes proposed in this PR:

AI-Use Disclosure

Legal Acknowledgement

Uh oh!

codecov Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov Bot commented Jun 3, 2026 •

edited

Loading