Store non-global compiled DI areas as their diff from global

[jeanmarcos-dev]

Description (*)

setup:di:compile writes a complete DI configuration file per area. On a stock Mage-OS build each of them is ~6.5 MB and the whole set adds up to 44.6 MB, even though every non-global area is nearly identical to global. crontab.php turns out to have zero differing entries — 6.5 MB that are an exact copy of global.php.

This makes each non-global area store only the entries that differ from global:

area           before      after     differing entries
adminhtml     6555.0 KB   774.5 KB   975 / 17048  (5.72%)
crontab       6502.3 KB     0.2 KB     0 / 16974  (0%)
frontend      6534.6 KB   129.9 KB   173 / 17019  (1.02%)
graphql       6543.9 KB    73.8 KB   110 / 17005  (0.65%)
webapi_rest   6508.6 KB   129.4 KB   104 / 16981  (0.61%)
webapi_soap   6507.2 KB   126.2 KB   100 / 16980  (0.59%)

generated/metadata DI total: 44.6 MB -> 7.6 MB (-83%)

Why this is equivalent

At runtime an area is always applied on top of global, and ObjectManager\Config\Compiled::extend() merges with a top-level array_replace per section. So applying only the entries that differ from global leaves the object manager in a state that is identical to applying the complete file. This is an equivalence by construction, not an approximation, as long as the diff is computed with a strict (!==) comparison on whole top-level values.

Two situations would break that reasoning, and both are handled rather than assumed away:

1. A second area applied in the same process. Config\Compiled tracks which area its state currently holds. When a delta extends an area other than the one applied, the base is loaded and merged first, restoring entries the previous area had overridden. This does happen in core — see dev/tests/api-functional/framework/Magento/TestFramework/TestCase/GraphQl/ResolverCacheAbstract.php, which switches to graphql and later restores the previous area.
2. An arbitrary configure() before the area is applied — e.g. mockCache() in DeployStaticContentCommand. Any such call clears the tracked area, so the next area is rebuilt from its base.

Files carry an explicit _extends marker, so a generated/ directory produced before this change keeps working untouched, and the interception and plugin-list files that go through the same loader are returned as they are.

Behaviour change worth calling out

ConfigLoader\Compiled itself is unchanged — load() still returns the contents of the compiled file — but for a non-global area that file is now a delta rather than the complete configuration. Every core consumer passes the result straight to ObjectManagerInterface::configure(), which resolves it; the two callers that are not areas (Interception\Config\CacheManager, PluginList) use keys whose files carry no marker and are unaffected; and the DI compiler and dev:di:info use the uncompiled loader. Third-party code that calls load() on an area to inspect the configuration rather than to apply it would now receive only the differences.

The alternative — rebuilding the complete configuration inside the loader — was measured: it costs an extra array_replace over the full configuration on every request and saves nothing in process memory, so the marker is resolved at the single place that applies it.

Measured impact

	before	after	saving
OPcache shared memory (the 7 DI files)	71.9 MB	12.3 MB	−59.6 MB
CLI process — include without OPcache	59.26 ms / +29.92 MB	1.10 ms / +0.63 MB	−58 ms, −29 MB
Disk	44.6 MB	7.6 MB	−37 MB
extend() per request	0.664 ms	0.157 ms	−0.5 ms
include per request, warm OPcache	~0 ms	~0 ms	—

Setting expectations honestly: this is not a page-latency optimisation. On a web request with a warm OPcache the gain is about half a millisecond, which is noise. Constant arrays live in shared memory and cost the worker almost nothing to load — measured, a 100 KB literal array costs 0.5 KB of process memory with OPcache on versus 469.8 KB with it off.

What it does buy is resources and process start-up:

• OPcache memory. A compiled file occupies ~1.6× its size in shared memory, so the 44.6 MB on disk were really 71.9 MB of opcache.memory_consumption. With PHP's 128 MB default that is over half the pool spent on DI configuration; freeing it removes a common source of evictions, which in turn cause recompilations and real latency spikes.
• Every CLI process. opcache.enable_cli is off by default, so each bin/magento invocation, cron job and queue consumer currently pays ~58 ms and ~30 MB purely to load its area.
• Disk, images and deploys, 37 MB smaller.

How it was verified

• Unit tests for both the diff and the resolution, plus the full lib/internal/Magento/Framework and setup/src/Magento/Setup suites: 7545 tests, 23595 assertions, no regressions.
• On a real compiled build, array_replace(global, delta) is identical to the complete configuration for all six areas, and both situations above leave the object manager in the same state as today.
• MFTF against a compiled install: AdminCreateSimpleProductTest (14 assertions) and StorefrontCategoryNavigationHighlightingTest (27 assertions).
• Storefront, admin, REST, GraphQL, cron:run and bin/magento exercised on a compiled install.

Note: the build used PHP 8.3, where the LazyTypes chain returns early, so the lazyTypes section of the delta is covered by unit tests but not by a real PHP 8.4 build.

Related Pull Requests

None.

Fixed Issues (if relevant)

1. Fixes Optimize compilation areas #203

Manual testing scenarios (*)

1. Run bin/magento setup:di:compile on a stock install. generated/metadata/global.php is unchanged; every other area is now a small file starting with '_extends' => 'global'.
2. Browse the storefront and the admin, issue a GraphQL query and a REST call — every area must behave as before.
3. Run bin/magento cron:run and start a queue consumer.
4. Keep a generated/ directory compiled before this change and boot the application with the new code: those files have no marker and must load exactly as they do today.
5. In a single process, load one area and then another (the GraphQL resolver-cache tests do this): the second area must fully replace the first, as it does today.

Questions or comments

The reference patch linked in the issue (monogo-m2-optimize-object-manager) proved the idea works in production but was not integrable as-is: it computes the diff at runtime and writes it back into generated/metadata/ on the first request (which breaks on read-only filesystems and races between concurrent workers), it merges a recursive diff with array_replace_recursive — which is not what extend() does and can silently keep stale entries from global — and its diff drops any override to a falsy value ('', 0, false, null, []) through an if (!empty($value)) check. This implementation moves the whole computation to compile time and mirrors extend()'s flat merge exactly.

Worth a follow-up, out of scope here: the compiled plugin-list files (global|frontend|plugin-list.php and friends) are also duplicated per scope, but they hold a different structure and their cache key is already cumulative, so they need their own approach.

[rhoerr]

Interesting PR, thanks for sending it. Can you link to any related issues/modules/discussions?

@jakwinkler I'm curious if you have any thoughts here from your speed suite work

[marcelmtz]

I've tested this, its working good for me. I can see the benefits mentioned for current MageOS release. I think we should be able to proceed with this for MageOS 4 unless the community wants to add anything else.

[rhoerr]

I chatted with Jakub about this earlier. His main comments were 'not sure why core doesn't do this already' and 'has it been submitted upstream?' (not yet). It does help cold start time a bit.

I like the idea, I'm generally in favor of including this for 4.0. I want to review/try it myself, but given Marcel reviewed and tested, don't consider me a blocker.

[jakwinkler]

I've used to work with this one
https://github.com/MonogoPolska/monogo-m2-optimize-object-manager/blob/master/patches/composer/optimize-config-loader.diff

Great PR btw!
This change does bring improvement to Magento, it does load less MB to each Magento request which is a win.
Each request / opcache config laoded is smaller, which is a win.

My research from last year:
On the cold restart of opcache it will save between 30-50ms to load the second large metadata compiled file.
My point: the numbers shown in the table are valid and I had the same results.