brutalcoding / llama_dart

Flutter / Dart bindings for llama.cpp

llama_dart

Known Issues

If you encounter the error 'Couldn't resolve native function 'llama_context_default_params' on macOS ARM64, please refer to Issue #2 for details and updates. Contributions and insights are welcome.

Description

llama_dart is an experimental Flutter plugin designed to seamlessly integrate the capabilities of llama.cpp into your Flutter or Dart apps. Please note that this package is currently in development and may not be fully functional.

Steps to Run

Important: Native assets support in Flutter MacOS, iOS, and Linux is available on the master channel behind an experimental flag: flutter config --enable-native-assets and flutter create --template=package_ffi [package name]. See this comment for more details.

Here are the steps to run llama_dart:

1. Switch to the master channel: flutter channel master
2. Upgrade Flutter: flutter upgrade
3. Enable native assets: flutter config --enable-native-assets
4. Create a package using the ffi template: flutter create --template=package_ffi llama_dart
5. Navigate to the project folder: cd llama_dart
6. Get dependencies: flutter pub get
7. Run the app: flutter run
8. Generate bindings using Dart SDK (temporary workaround): dart --enable-experiment=native-assets run ffigen --config ffigen.yaml

Please note that step 8 currently requires using the Dart SDK instead of Flutter until the Flutter team completes implementing the feature in the Flutter SDK.

Supported Use Cases

llama_dart aims to support various use cases, including:

• Running the app with flutter run
• Debugging
• Hot reload (1)
• Hot restart (1)
• Release/profile builds
• Building debug/release/profile builds with flutter build
• Running tests with flutter test or dart test
• Adding to an app (e.g., flutter build macos-framework for MacOS)
• Running with the Flutter tester: flutter run -dflutter-tester

Please note that the plugin is still under development, and not all use cases may be fully supported at this time.

Platform Support

• MacOS
• Windows
• Linux
• Android
• iOS

Note: This list represents our intended platform support after addressing the initial issues on MacOS ARM64. Support for other platforms may be added in the future.

Coming Soon

Stay tuned for updates! llama_dart is meant to make it easier to incorporate llama.cpp capabilities into your Flutter or Dart apps, and we're actively working on improving its functionality.

If you encounter any issues or have additional information that could help, please feel free to contribute to the discussion. Your insights and experiences are valuable in addressing this problem effectively.

llama_dart's Issues

Native Function Resolution Error on macOS ARM64

Description:
I encountered an issue while testing the llama_dart project on macOS ARM64 architecture. When running the project, I received the following error:

ArgumentError (Invalid argument(s): Couldn't resolve native function 'llama_context_default_params' in 'package:llama_dart/llama_dart_bindings_generated.dart' : dlsym(RTLD_DEFAULT, llama_context_default_params): symbol not found.)

Background:
I'm working with a cutting-edge experimental feature that the Dart team is likely to port over to Flutter later. This new experimental FFI (Foreign Function Interface) package enables any Flutter/Dart developer to integrate llama_dart into their project by adding it to their pubspec.yaml. Running the app is as simple as typing flutter run in their IDE or terminal. This feature eliminates the need to run cmake to compile binaries for each platform, simplifying the user experience and reducing the effort required to provide platform-specific binaries.

I plan to incorporate this plugin into my other project shady.ai.

Issue Details:
While the plugin successfully compiles, the error mentioned above occurs during runtime. The example app uses the generated bindings to look up a specific symbol, the llama context. This symbol must have a matching address/method in the included binary that was auto-compiled (e.g., llama_dart.dylib). Upon inspecting the binary, I discovered old remnants of the original sum() function, indicating that something isn't working as expected.

I'll be actively working on this issue and will provide updates along the way.

Environment:

• OS: macOS (ARM64)
• Dart Version: [Specify Dart version, if relevant]
• llama_dart Version: 0.0.1

Workaround (if any):
There is no true workaround for this issue that aligns with the intended purpose of the package. The purpose of this FFI package is to simplify integration by seamlessly incorporating C++ source code into the relevant platform projects, eliminating the need for users to compile binaries manually. While you can compile the binary yourself using a similar approach as demonstrated in my previously mentioned project shady.ai, this method deviates from the package's intended workflow. It requires users to engage with low-level tooling such as CMake, which the package aims to abstract away. Ideally, users should not have to think about using CMake or handling binaries directly.

Note to Users:
If you encounter this issue or have additional information that could help resolve it, please feel free to contribute to the discussion. Your insights and experiences are valuable in addressing this problem effectively.