[vm/ffi] Add support for weak symbols `@Native(weak: true)`

[simolus3]

In the current dart:ffi system, it's possible to deal with different versions of libraries we're binding to by checking for the existence of symbols:

final lib = DynamicLibrary.open('libhelper.so');
if (lib.providesSymbol('my_function_v2')) {
  // Bind to new implementation
} else {
  // Bind to older implementation
}

It looks like no similar functionality exists with the new @Native annotations. While this might not be a big issue when we control the version of the library through native_add_library, that's not always guaranteed and we might be accessing a library from an operating system.

For this, I think the ability to check whether the symbol a @Native annotation points to is accessible would be nice. This could be exposed with nullable functions:

@Native()
external final void Function()? my_function_v2;

But that doesn't really translate to native globals which are already getters. So a better approach may be to:

1. Allow a guarantee that binding to a symbol that doesn't exist only throws when that function/field is first accessed (e.g. @Native(lazy: true) external void my_function_v2();).
2. Add an API similar to Native.addressOf that checks for the existence of a symbol, e.g. bool exists = Native.exists(my_function_v2). This could also work for globals since the call to Native.exists would be lowered in the CFE.

I'm happy to contribute this feature once there's an basic agreement on the API / whether this is something we should support.

[dart-github-bot]

Summary: User requests optional symbol support in dart:ffi's @Native annotations. Currently, checking for native symbol existence isn't possible with @Native, unlike DynamicLibrary.providesSymbol.

[dcharkes]

• Allow a guarantee that binding to a symbol that doesn't exist only throws when that function/field is first accessed (e.g. @Native(lazy: true) external void my_function_v2();).

This might be better controlled by the LinkMode in the CodeAsset. DynamicLoadingBundled / DynamicLoadingSystem is guaranteed to be lazy. If we add support for StaticLinking, DynamicLinkingBundled andDynamicLinkingSystem then these would try to run the linker against the (tree shaken) Dart code. However, so far we've not moved in such direction.

• Add an API similar to Native.addressOf that checks for the existence of a symbol, e.g. bool exists = Native.exists(my_function_v2). This could also work for globals since the call to Native.exists would be lowered in the CFE.

Native.exists SGTM. 👍

cc @mkustermann thoughts?

[dcharkes]

@liamappelbe Would this be useful for the MacOS/iOS APIs as well? Or do you have a different way of guarding against invoking a function that doesn't exist in a version of the OS?

[liamappelbe]

I don't think it will be useful for ObjC, because all the class loading and method invocation goes through the ObjC runtime API (objc_getClass, objc_msgSend etc). For example, objc_getClass returns null if the class doesn't exist.

[simolus3]

I think I need some guidance on the implementation too. I see that resolving symbols ultimately happens through dart::FfiResolve which throws if a lookup error occurs. It looks like we'll probably need a variant of that function that doesn't throw.
Should there be another argument on FfiResolve, Native._ffi_resolver_function and Native._get_ffi_native_resolver to control that behavior?

Alternatively, since dart::ThrowFfiResolveError throws an argument error, we could also introduce a subclass of ArgumentError, e.g. NativeSymbolNotFoundError, that would be thrown when looking up symbols that don't exist. That way, Native.exists(a) could be lowered into:

try {
  Native.addressOf(a);
  return true;
} on NativeSymbolNotFoundError {
  return false;
}

I don't think that's too bad since symbols usually exist and we don't have to introduce a new argument to the resolver (which I suppose would also mean two different CachableIdempotentCall entries?).

[dcharkes]

Alternatively, we could add

static bool FfiSymbolExists(Dart_Handle asset_handle,
                           Dart_Handle symbol_handle) {

DEFINE_NATIVE_ENTRY(Ffi_GetFfiNativeExists, 1, 0) {
  return Pointer::New(reinterpret_cast<intptr_t>(FfiSymbolExists));
}

It would probably be best if FfiSymbolExists throws an argument error if the asset id is not in the mapping or the asset id mapping is wrong (the dylib doesn't exist). This would require some refactoring of FfiResolveInternal to return the errors for symbol lookup separately from dylib opening and asset id errors.

Also, we probably should also add a Native.assetExists, so that you can look up if an asset is available. (The equivalent of doing File(dylibPath).existsSync() that you would use when doing DynamicLibrary.open(dylibPath). This would enable a package having a fallback on a Dart polyfill if the native code is not supplied. cc @mkustermann

If we believe Native.assetsExists(...) and Native.symbolExists(...) would be on the hot path, we might want to cache those two calls with cachable idempotent as well.

cc @mkustermann for thoughts.

[mkustermann]

Conceptually our declarative ffi bindings are modeled after C (basically a C DSL in Dart): Imagine the Dart code wasn't Dart but C code. The idea being that we'll have similar optimizations to what C code would get (e.g. tree shaking, direct calls, etc). The compiler would have the option (we don't do that atm, but we could) to compile the Dart code to compiled code with unresolved symbols and one could statically link in the corresponding C code, making the resulting shared library or executable emit direct calls to the corresponding functions.

So if we made our compiler emit such code, what may happen with those symbols that don't exist is that either we

• get a linker error when we link the two parts together or
• the resulting shared library would have unresolved symbols, which may cause a dlopen() to fail at runtime due to missing symbols

((**) One could add support to tell the dart native compiler the available native symbols when static linking is enabled (since we may know the static libraries we're later going to link the dart code against). That would allow the dart native compiler to tree shake the code for non-existent symbols (i.e. it would evaluate the Native.exists() at dart compile-time). Though this seems very hackish as normally one could compile native code and dart code in parallel and link afterwards, this would require dart code to be compiled after all native libraries)

So it may make sense to tell the compiler that the symbol is special, it may not exist, so it cannot emit direct calls to it.

If the code is checking for a symbol presence, we have to prevent that symbol from being tree shaken (in the case when the native library is compiled & tree shaken). So we should use a declarative way that works with tree shaking and not a string-based lookup approach.

So I may be in favor of

@Native<Void Function()>(symbol: "foo", optional: true) // <-- optional may cause it to throw
external void foo();

@Native<Int8>(symbol: "bar", optional: true) // <-- optional may cause it to throw
external int bar;

if (Native.exists(foo)) { ... } // <-- we only allow referring to declarative symbols and only those that are `optional: true`
if (Native.exists(bar)) { ... } // <-- same

Right now all our resolution is lazy, so lookups are done lazily and errors get thrown if lookup fails, so optional: true wouldn't be necessary with current implementation, but as mentioned above if we did emit direct calls the compiler would need to know that the symbol may not exist and it has to treat it specially (and it may not be practical for the build tool to tell the compiler which symbols are available).

The next thing is that our symbol resolution currently is multi-stage IIRC: It looks in one place and if cannot find it there it falls back to process-wide symbols (@dcharkes may be able to confirm is this is true). We'd probably not want to do this fallback pass for such optional symbols: The developer intend is to say: This symbol should be in a specific library and if it's not in there I do something else - so it shouldn't follow any fallback to process-symbol lookups.

Also, we probably should also add a Native.assetExists, so that you can look up if an asset is available.

If the Dart code works without any of the native assets, that would make all the native symbols optional, which in return would never allow the Dart compiler to emit direct calls to targets (modulo the (**)).

[mkustermann]

In the current dart:ffi system, it's possible to deal with different versions of libraries we're binding to by checking for the existence of symbols:

@simolus3 If you wrote this kind of code in C instead of Dart with FFI, how would the C code look like?

Maybe this is a case where your C code doesn't know what it compiles against, so it would use dlopen(), dlsym(), ... to access those symbols in a dynamic way.

So if that's how the C code would look like, then I would assume you also would use the dynamic FFI api we have (not the declarative ffi api). Though the dynamic API we have - using DynamicLibrary.open() etc doesn't integrate with native assets yet.

Maybe what this is asking for is something like this:

@NativeLib(assetId: ...)
external DynamicLibrary? libfoo;

if (libfoo != null) {
  if (libfoo.providesSymbol('foo')) { ... }
}

That would give developers access to native-asset bundled libraries via the dynamic FFI api. Then a developer would be responsible to ensure the hook is only adding dynamic libraries (and never static ones) and doesn't tree shake it.

[simolus3]

Also, we probably should also add a Native.assetExists, so that you can look up if an asset is available.

I like this idea 👍 It would also allow package authors to include both native-assets based code and older code based on DynamicLibrary lookups, and then use Native.assetExists to check whether the new implementation can be used or whether to fall back to the old one.

If you wrote this kind of code in C instead of Dart with FFI, how would the C code look like?

In my case, I'm talking about these functions for package:sqlite3 (and I need to check whether sqlite3_prepare_v3 is available). SQLite provides a preprocessor definition for the version, so C code could use:

#if SQLITE_VERSION_NUMBER > 3020000
 sqlite3_prepare_v3(...);
#else
 sqlite3_prepare_v2(...);
#endif

Obviously that can't be translated to Dart.

so it would use dlopen(), dlsym(), ... to access those symbols in a dynamic way.

I'm fine with using DynamicLibrary for these optional functions, but:

1. Once the linker is integrated with Dart's tree-shaking system, I need a way to express that I need these functions when they're available (but optional: true would work).
2. There is no way to get a DynamicLibrary for a Dart file using @Native declarations.

[dcharkes]

((**) One could add support to tell the dart native compiler the available native symbols when static linking is enabled (since we may know the static libraries we're later going to link the dart code against). That would allow the dart native compiler to tree shake the code for non-existent symbols (i.e. it would evaluate the Native.exists() at dart compile-time). Though this seems very hackish as normally one could compile native code and dart code in parallel and link afterwards, this would require dart code to be compiled after all native libraries)

This relates to a deeper problem that we have with native API evolution and how to write Dart code that works against different version of a native API. dart-lang/native#1588 dart-lang/native#1590

If we'd know what version of the native API we'd be having at runtime when compiling Dart, we could do something smart. For example use a Dart define and let the AOT tree shaker take care of tree-shaking all calls for the different native API version.

However, if we don't know at compile time what version of the API we will have at runtime, we need checking at runtime. This could either be indirectly through seeing if a symbol exists, or more explicitly via an API call that returns a version. Now, obviously that means we wont be doing static linking!

So we're either (A) static linking and then we could know what version of the native API we have, or (B) we're dynamically loading and we don't know what version of the native API we have.

With native assets, we'd know which assets have (A) static linking and which once have (B) dynamic loading.

and it may not be practical for the build tool to tell the compiler which symbols are available

I don't believe it would be needed to tell which symbols are available. Symbols not being available would only enable us to compile a throw into the code instead of an unresolved symbol

Instead, the Dart compiler needs to know which assets are statically or dynamically linked + a set of defines which should remove the dead code that contains the calls to native code which are not available.

This agrees with the approach if it were C instead of Dart code, you would pass the same define to both builds, or have it be defined by importing a header file, if you have scenario (A). And if you have scenario (B), you're likely using dlsym.

I do agree that trying to support (A) with actual static linking might be impractical. Wiring up the Dart build to get the right defines could be painful.

(@mkustermann we have a bigger issue w.r.t. static linking. We want to tree-shake native assets, the link hooks must run after Dart AOT compilation. If you want to compile Dart code to call external symbols, you must first get the assets list before doing Dart compilation so that you know what assets are statically linked. Which means the link hooks must run before Dart compilation. With us moving asset resolution to embedder callbacks for Flutter, there's no way for the Dart compiler to access which code assets must be statically linked and which ones are dynamically linked. The NativeAssetsApi assumes dynamically loaded. And we can't put whether it's statically linked in the @Native annotation either, because then it would stop working in JIT mode. We'll likely always use dynamic loading for JIT.)

A side question here is how FFIgen configurations should work. If you run FFIgen on your API (with a certain set of defines), it will only output the bindings for a single version of the API. It would be neat if you could run FFIgen multiple times with different sets of defines, and it would automatically spit out the optional: true part.

The next thing is that our symbol resolution currently is multi-stage IIRC: It looks in one place and if cannot find it there it falls back to process-wide symbols (@dcharkes may be able to confirm is this is true). We'd probably not want to do this fallback pass for such optional symbols: The developer intend is to say: This symbol should be in a specific library and if it's not in there I do something else - so it shouldn't follow any fallback to process-symbol lookups.

Yes, we don't want to do process lookup as fallback. @mkustermann we also check GetFfiNativeResolver from Dart_SetFfiNativeResolver, should that be supported?

Maybe what this is asking for is something like this:

@NativeLib(assetId: ...)
external DynamicLibrary? libfoo;

if (libfoo != null) {
if (libfoo.providesSymbol('foo')) { ... }
}

That would make your Dart code imply that you can never use static linking. That's okay. It means you're in scenario (B). Native.assetExists and Native.symbolExists would imply the same thing.

For supporting scenario (B), this solution is simpler than introducing Native.assetExists and Native.symbolExists. 👍

@simolus3 Are you in scenario (A) or (B) ?

If you're in scenario (A), I'd expect your Dart code to be somewhat like

void main() {
  const String sqliteVersion = String.fromEnvironment('SQLITE_VERSION_NUMBER');
  final int = sqliteVersionInt = int.parse(sqliteVersion);

  if (sqliteVersionInt > 3020000) {
    sqlite3_prepare_v3();
  } else {
    sqlite3_prepare_v2();
  }
}

@Native()
external final void Function() sqlite3_prepare_v3;

@Native()
external final void Function() sqlite3_prepare_v2;

And to compile it with flutter build --release --dart-define=SQLITE_VERSION_NUMBER=3090000 for AOT mode and flutter run --debug --dart-define=SQLITE_VERSION_NUMBER=3090000 for JIT mode.

And the hook could then output a dylib in JIT mode (linkingEnabled=false) and a static library in AOT mode.

Passing those Dart defines in manually is not very practical though.