For reference, the relevant docs is https://enzymead.github.io/Enzyme.jl/stable/faq/#faq-runtime-activity:

The way Enzyme's runtime activity resolves this issue is to return the original primal variable as the derivative whenever it needs to denote the fact that a variable is a constant. As this issue can only arise with mutable variables, they must be represented in memory via a pointer. All addtional loads and stores will now be modified to first check if the primal pointer is the same as the shadow pointer, and if so, treat it as a constant. Note that this check is not saying that the same arrays contain the same values, but rather the same backing memory represents both the primal and the shadow (e.g. a === b or equivalently pointer(a) == pointer(b)).

Enabling runtime activity does therefore, come with a sharp edge, which is that if the computed derivative of a function is mutable, one must also check to see if the primal and shadow represent the same pointer, and if so the true derivative of the function is actually zero.

Here are some questions and my tentative answers:

How do I detect whether a type is "mutable" in the sense of the previous paragraph? Right now I am using Base.datatype_pointerfree but apparently it doesn't detect Ptr itself (to be solved later).
Is it possible to encounter cases where pointer(a) == pointer(b) but !(a === b)? At least I seem to observe them in my tests.

Details

julia> using Base: unsafe_wrap

julia> a = [1, 2, 3]
3-element Vector{Int64}:
 1
 2
 3

julia> p = pointer(a)
Ptr{Int64} @0x0000000309471820

julia> d = unsafe_wrap(Array, p, 3)
3-element Vector{Int64}:
 1
 2
 3

julia> pointer(a)
Ptr{Int64} @0x0000000309471820

julia> pointer(d)
Ptr{Int64} @0x0000000309471820

julia> a === d
false

Can I perform the aliasing check when I don't compute the primal, e.g. in DI.pushforward as opposed to DI.value_and_pushforward? Apparently not.
Can there be errors when the user provides the shadow memory themselves, as opposed to the shadow being generated inside Enzyme.autodiff? At the moment it feels like in-place operators such as DI.pushforward! should be safe.
Can I perform the aliasing check for high-level Enzyme operators like Enzyme.jacobian? Apparently not, see Safeguard against runtime activity aliasing inside higher-level operators EnzymeAD/Enzyme.jl#2557.
Do we have an MWE for reverse-mode activity uncertainty? The one in the docs is for forward mode only, where the output can be either Duplicated or Const. In reverse mode, does this kind of uncertainty on the output affect shadow aliasing for the input?

Handle runtime activity properly in Enzyme #666

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions