Fix struct/const revision

[timholy]

The aim here is to provide full support for struct/const revision. This first commit just adds simple tests, but it already reveals something that needs to be addressed: when a struct gets revised, methods defined for that struct aren't automatically re-evaluated for the new type. Specifically, in the added tests, after revising the definition of Point we get

julia> methods(StructConst.firstval)
# 1 method for generic function "firstval" from StructConst:
 [1] firstval(p::@world(StructConst.Point, 40598:40739))
     @ /tmp/2wfLvVmwY7/StructConst/src/StructConst.jl:11

and there is no defined method for a currently-valid StructConst.Point.

@Keno, if you have a moment, let me check my understanding of the current situation:

• method applicability is assessed by type-intersection, as usual, but now types (via their bindings) can have a world-age
• the MethodInstances and CodeInstances involving the old type are still valid (their world age is not capped, but of course they require inputs with old types)
• there is no invalidation step needed for changes of the kind tested with the revision of Point (invalidate_code_for_globalref! is not called?)
• we can't count on binding.backedges to list everything that uses this binding
• the right solution is for Revise to:
  • detect that we're about to redefine a struct
  • search the entire system for any methods where that type appears in a signature
  • parse the source files, if necessary
  • redefine the struct
  • re-evaluate the method definitions

Presumably, waiting to do the last step as the last stage of a revision would be wise, as it is possible that more than one struct will be revised at the same time, and one might as well do each evaluation only once.

[Keno]

• method applicability is assessed by type-intersection, as usual, but now types (via their bindings) can have a world-age

I think that's a valid way to say it, but just to clarify, nothing about the type object itself changes, just the name by which you have to access it

• the MethodInstances and CodeInstances involving the old type are still valid (their world age is not capped, but of course they require inputs with old types)

yes

• there is no invalidation step needed for changes of the kind tested with the revision of Point (invalidate_code_for_globalref! is not called?)

I don't understand what you're asking

• we can't count on binding.backedges to list everything that uses this binding

It lists all cross-module edges to get everything, you also need to iterate all methods in the same module as the binding.

• the right solution is for Revise to:

  • detect that we're about to redefine a struct
  • search the entire system for any methods where that type appears in a signature
  • parse the source files, if necessary
  • redefine the struct
  • re-evaluate the method definitions

In the fullness of time, the correct way to implement Revise is to run the toplevel expressions in a non-standard evaluation mode that tracks dependency edges for any evaluated constants. However, that should maybe wait until after the JuliaLowering changes and the interpreter rewrite. I think your proposed strategy is reasonable in the meantime although not fully complete.

[timholy]

there is no invalidation step needed for changes of the kind tested with the revision of Point (invalidate_code_for_globalref! is not called?)

I don't understand what you're asking

I was wondering whether some kind of callback in there could be helpful in short-circuiting the process of determining which methoddefs need to be re-evaluated, but if it's not called then that won't help. From what I can tell, that's not going to work anyway because it only traverses the module containing the type definition. But your comment

It lists all cross-module edges to get everything, you also need to iterate all methods in the same module as the binding.

seems to be what I was looking for.

This seems fairly straightforward, thanks for the excellent groundwork.

[Keno]

I was wondering whether some kind of callback in there could be helpful in short-circuiting the process of determining which methoddefs need to be re-evaluated

No, there's no edges of any kind for signatures. You need to scan the entire system. However, an earlier version of that code did the same whole-system scan, so you can steal some code for whole-system scanning.

[timholy]

I added a cross-module test (the StructConstUser module), expecting this might populate b.bindings, but I get

julia> b = convert(Core.Binding, GlobalRef(StructConst, :Point))
Binding StructConst.Point
   40743:∞ - constant binding to StructConst.Point
   40598:40742 - constant binding to @world(StructConst.Point, 40598:40742)
   1:0 - backdated constant binding to @world(StructConst.Point, 40598:40742)
   1:0 - backdated constant binding to @world(StructConst.Point, 40598:40742)

julia> b.backedges
ERROR: UndefRefError: access to undefined reference
Stacktrace:
 [1] getproperty(x::Core.Binding, f::Symbol)
   @ Base ./Base_compiler.jl:55
 [2] top-level scope
   @ REPL[6]:1

so I don't think I understand what

It lists all cross-module edges

really means. It also doesn't populate if I add

struct PointWrapper
    p::StructConst.Point
end

to that module.

I understand that I have to traverse the whole system, I'm just curious about what b.backedges stores.

[Keno]

There's two kinds of edges that are tracked there. One is explicit import/using. The other is to lowered code of method definitions (but only after the first inference). At no point is an edge ever added for an evaluation of a binding, only for compilation of code that references the binding.

[timholy]

Just to cross-check:

                module StructConstUser
                using StructConst: Fixed, Point
                struct PointWrapper
                    p::Point
                end
                scuf(f::Fixed) = 33 * f.x
                scup(p::Point) = 44 * p.x
                end

yields a backedge for the using StructConst: Fixed, Point statement, but I don't see anything related to scup even if I evaluate it:

julia> e = only(b.backedges)
Binding StructConstUser.Point
   40598:∞ - explicit `using` from StructConst.Point
   1:0 - undefined binding - guard entry

[Keno]

but I don't see anything related to scup even if I evaluate it:

That's because scup doesn't use any bindings other than * and getproperty in its lowered code (as I said, no edges for signatures). However, even if you had like mkscup() = Point(), that binding would be to StructConstUser. To actually get a cross-module edge is not that easy, but you could do @eval mkscup() = $(GlobalRef(StructConst, :Point))(). The most common case to get cross-module edges is macros. Also, not a lot of edges is good, that is the design goal, since we don't want to store them :).

[timholy]

This is getting close, but there's a philosophical question to answer: should Revise try to hew to Base behavior as closely as possible, or should it add/modify behaviors for what might be a nicer interactive experience?

In this case, the issue is the following: when I redefine Point, in Base you can still call previously-defined functions on "old data." However, in this PR, calling functions on old data throws (or is intended to throw) a MethodError. Why might you want that? Because I imagine that a common user-error will be to forget to refresh your data with updated types, and if your whole code pipeline runs (using all the old definitions), I imagine people will bang their heads against walls trying to figure out why their edits aren't having the intended effect. In contrast, if they get a no method foo(::@world(Point, m:n)), they'll pretty quickly learn how you fix it.

I should say that initially I wasn't aiming in this direction, and this current proposal arose from noticing some unexpected subtleties about the order in which Revise's cache files get set up: the order in which you parse the source, lower the code, and rebind the name matters quite a lot, and some of the choices in this version are designed to compensate for the fact that Revise normally doesn't parse a file unless it's been edited. (Now, we'll also have to parse any file with a method whose signature references an updated type.) I think it's possible to hew to the Base behavior, if we decide that's better, but the initial bugs I faced led me to ask what behavior we actually want, and I came to the conclusion that it's likely a better user experience if we invalidate methods that only work on old types.

[Keno]

However, in this PR, calling functions on old data throws (or is intended to throw) a MethodError. Why might you want that?

I think that's fine. Conceptually Revise does both adding new methods and deleting old ones. Base is always "append only".

[timholy]

The collateral damage to tests is something I'm looking into; if one runs all the tests, there appear to be some failures that predate this effort, including an incomplete updating of the ecosystem to JuliaLang/julia#52415. I'll work on that.

But meanwhile, this seems to be working:

tim@diva:~/.julia/dev/Revise$ ~/src/juliaw/julia --startup=no --project -e 'using Pkg; Pkg.test(; test_args=["struct/const revision"], coverage=false)'
     Testing Revise
      Status `/tmp/jl_4EbO8B/Project.toml`
# Pkg output deleted
     Testing Running tests...
Test Summary: | Pass  Total   Time
Revise        |   30     30  10.3s
beginning cleanup

and may be ready for anyone who wants to review it.

[timholy]

@Keno @vtjnash @aviatesk @JeffBezanson here's a fun one:

julia> using Revise
Precompiling Revise finished.
  1 dependency successfully precompiled in 6 seconds. 19 already precompiled.

julia> Revise.track(Core.Compiler)

julia> fieldnames(Core.Compiler.NativeInterpreter)
(:world, :method_table, :inf_cache, :codegen, :inf_params, :opt_params)

#
# edited Compiler/src/types.jl to add a field `dummy` to `NativeInterpreter`
#

julia> 1+1
WARNING: Detected access to binding `Compiler.#NativeInterpreter#476` in a world prior to its definition world.
  Julia 1.12 has introduced more strict world age semantics for global bindings.
  !!! This code may malfunction under Revise.
  !!! This code will error in future versions of Julia.
Hint: Add an appropriate `invokelatest` around the access to this binding.
⋮     # lots more like this
Compiling the compiler. This may take several minutes ...
Base.Compiler ──── 1.72376 seconds
2

julia> fieldnames(Core.Compiler.NativeInterpreter)
(:world, :method_table, :inf_cache, :codegen, :inf_params, :opt_params, :dummy)

julia> function f(::Integer)
           Base.Experimental.@force_compile
           return 1
       end
f (generic function with 1 method)

julia> f(5)
1

[aviatesk]

This is interesting.

So it looks like this PR tracks edges not only from binding to method signature, but also from binding to method body?

I think a similar mechanism will be needed when developing a new language server. If it's already implemented in Revise, I'd like to reuse it.

I haven't had a chance to look at this PR closely yet, but I'll read it later.

[timholy]

Yes, when a type gets rebound, Revise will scan the entire system for methods m::Method where that type appears in m.sig. It then tries to find the expression that defined that method, deletes the old method, and re-evaluates the expression. That way it regenerates methods for the new meaning of the type.

There's a separate bit of novelty for Core.Compiler: this PR also checks whether any of the entry points to type inference have been invalidated, and if so calls bootstrap!

[timholy]

I'll be curious to see whether it improves the efficiency of hacking on inference.

[Keno]

There's a separate bit of novelty for Core.Compiler: this PR also checks whether any of the entry points to type inference have been invalidated, and if so calls bootstrap!

I don't know if I like that. We already have @activate Compiler[:codegen] for this use case.

[timholy]

I didn't know about @activate. Is there a demo of the workflow somewhere? I've seen the docs, and they're good, but I suspect there's a bit more wisdom around effective usage than they convey.

I can back that commit out, if there are other mechanisms to achieve the same aim. Are you saying that Revise.track(Core.Compiler) should be deprecated?

[KristofferC]

I tried this. On:

• Revise (this PR)
• LoweredCodeUtils master
• JuliaInterpreter master
• Julia 1.12.0-beta1

What I did was:

• Load Revise, Tensors.
• Make the following revision to Tensors:

  diff --git a/src/Tensors.jl b/src/Tensors.jl
  index 4cd677b..5f04312 100644
  --- a/src/Tensors.jl
  +++ b/src/Tensors.jl
  @@ -63,6 +63,7 @@ julia> Tensor{1,3,Float64}((1.0, 2.0, 3.0))
   """
   struct Tensor{order, dim, T, M} <: AbstractTensor{order, dim, T}
       data::NTuple{M, T}
  +    x::Int
       Tensor{order, dim, T, M}(data::NTuple) where {order, dim, T, M} = new{order, dim, T, M}(data)
   end

• Press 1 in the REPL
• Press 1 in the REPL again

with the following result:

julia> using Revise, Tensors

# updates Tensors file here

julia> 1
ERROR: UndefVarError: `Broadcasted` not defined in `StaticArrays.StableFlatten`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
 [1] top-level scope
   @ ~/.julia/packages/StaticArrays/LSPcF/src/broadcast.jl:180
Revise evaluation error at /Users/kristoffercarlsson/.julia/packages/StaticArrays/LSPcF/src/broadcast.jl:180

Stacktrace:
  [1] methods_by_execution!(recurse::Any, methodinfo::Revise.CodeTrackingMethodInfo, docexprs::Dict{…}, mod::Module, ex::Expr; mode::Symbol, disablebp::Bool, always_rethrow::Bool, kwargs::@Kwargs{})
    @ Revise ~/JuliaPkgs/Revise.jl/src/lowered.jl:306
  [2] eval_with_signatures(mod::Module, ex::Expr; mode::Symbol, kwargs::@Kwargs{})
    @ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:556
  [3] eval_with_signatures
    @ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:553 [inlined]
  [4] instantiate_sigs!(modexsigs::OrderedCollections.OrderedDict{…}; mode::Symbol, kwargs::@Kwargs{})
    @ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:564
  [5] instantiate_sigs!
    @ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:560 [inlined]
  [6] maybe_extract_sigs_for_meths(meths::Set{Method})
    @ Revise ~/JuliaPkgs/Revise.jl/src/pkgs.jl:162
  [7] (::Revise.var"#107#108"{Bool})()
    @ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:941
  [8] lock(f::Revise.var"#107#108"{Bool}, l::ReentrantLock)
    @ Base ./lock.jl:335
  [9] #revise#104
    @ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:844 [inlined]
 [10] revise()
    @ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:842
 [11] top-level scope
    @ REPL:1

caused by: UndefVarError: `Broadcasted` not defined in `StaticArrays.StableFlatten`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
  [1] lookup_var
    @ ~/.julia/packages/JuliaInterpreter/J7J9G/src/interpret.jl:5 [inlined]
  [2] step_expr!(recurse::Any, frame::JuliaInterpreter.Frame, node::Any, istoplevel::Bool)
    @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/J7J9G/src/interpret.jl:44
  [3] signature(recurse::Any, frame::JuliaInterpreter.Frame, stmt::Any, pc::Int64)
    @ LoweredCodeUtils ~/.julia/packages/LoweredCodeUtils/BIVzf/src/signatures.jl:50
  [4] methoddef!(recurse::Any, signatures::Vector{Any}, frame::JuliaInterpreter.Frame, stmt::Any, pc::Int64; define::Bool)
    @ LoweredCodeUtils ~/.julia/packages/LoweredCodeUtils/BIVzf/src/signatures.jl:595
  [5] methoddef!
    @ ~/.julia/packages/LoweredCodeUtils/BIVzf/src/signatures.jl:534 [inlined]
  [6] methods_by_execution!(recurse::Any, methodinfo::Revise.CodeTrackingMethodInfo, docexprs::Dict{…}, frame::JuliaInterpreter.Frame, isrequired::Vector{…}; mode::Symbol, skip_include::Bool)
    @ Revise ~/JuliaPkgs/Revise.jl/src/lowered.jl:354
  [7] methods_by_execution!
    @ ~/JuliaPkgs/Revise.jl/src/lowered.jl:317 [inlined]
  [8] methods_by_execution!(recurse::Any, methodinfo::Revise.CodeTrackingMethodInfo, docexprs::Dict{…}, mod::Module, ex::Expr; mode::Symbol, disablebp::Bool, always_rethrow::Bool, kwargs::@Kwargs{})
    @ Revise ~/JuliaPkgs/Revise.jl/src/lowered.jl:296
  [9] eval_with_signatures(mod::Module, ex::Expr; mode::Symbol, kwargs::@Kwargs{})
    @ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:556
 [10] eval_with_signatures
    @ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:553 [inlined]
 [11] instantiate_sigs!(modexsigs::OrderedCollections.OrderedDict{…}; mode::Symbol, kwargs::@Kwargs{})
    @ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:564
 [12] instantiate_sigs!
    @ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:560 [inlined]
 [13] maybe_extract_sigs_for_meths(meths::Set{Method})
    @ Revise ~/JuliaPkgs/Revise.jl/src/pkgs.jl:162
 [14] (::Revise.var"#107#108"{Bool})()
    @ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:941
 [15] lock(f::Revise.var"#107#108"{Bool}, l::ReentrantLock)
    @ Base ./lock.jl:335
 [16] #revise#104
    @ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:844 [inlined]
 [17] revise()
    @ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:842
 [18] top-level scope
    @ REPL:1
Some type information was truncated. Use `show(err)` to see complete types.

julia> 1
Compiling the compiler. This may take several minutes ...
Base.Compiler ──── 3.43502 seconds
1

The UndefVarError seems correct:

julia> Tensors.StaticArrays.StableFlatten.Broadcasted
ERROR: UndefVarError: `Broadcasted` not defined in `StaticArrays.StableFlatten`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
 [1] getproperty(x::Module, f::Symbol)
   @ Base ./Base_compiler.jl:48
 [2] top-level scope
   @ REPL[4]:1

so I guess the question is why it is tried to be looked up. And also, why does the Compiler recompile when I press another time 1 with no new modifications?

[aviatesk]

I updated against the latest master branch.

I tested this branch with the simple case, but it seems that for exported bindings, additional edge tracking is needed?
In particular, the world ages of bindings that are usinged need to be updated, it seems.
For example, let's say we have an Example package like this:

module Example
export hello, Hello

struct Hello
    who::String
end

hello(x::Hello) = hello(x.who)

"""
    hello(who::String)

Return "Hello, `who`".
"""
hello(who::String) = "Hello, $who"

end

We get the following error:

julia> using Revise

julia> using Example

julia> hello(Hello("world"))
"Hello, world"

julia> # Apply the following diff
       # ```diff
       # diff --git a/src/Example.jl b/src/Example.jl
       # index 65c7eae..c631814 100644
       # --- a/src/Example.jl
       # +++ b/src/Example.jl
       # @@ -2,10 +2,10 @@ module Example
       #  export hello, Hello
       #  
       #  struct Hello
       # -    who::String
       # +    who2::String
       #  end
       #  
       # -hello(x::Hello) = hello(x.who)
       # +hello(x::Hello) = hello(x.who2 * " (changed)")
       #  
       #  """
       #      hello(who::String)
       # ```

julia> hello(Hello("world"))
ERROR: MethodError: no method matching @world(Example.Hello, 38355:38383)(::String)
The type `@world(Example.Hello, 38355:38383)` exists, but no method is defined for this combination of argument types when trying to construct it.
Stacktrace:
 [1] top-level scope
   @ REPL[5]:1

julia> hello(Example.Hello("world")) # but this works
"Hello, world (changed)"