membraneframework / bundlex Goto Github PK

At the moment, bundlex converts local node to a distributed one whenever C node is being started.
This can result in bugs in other applications that do not expect change of local node to a distributed one (e.g. see membraneframework/membrane_core#352).

One of the solutions would be to enforce user to start application as a dristributed one (using --sname or --name flags) whenever there is a C node specified in bundlex.exs.
However, specyfing C node in bundlex.exs does not mean that it has to be spawned. It also requires from user some extra work and awarness of Distributed Erlang.

As reported in membraneframework/shmex#10, build of apps using Bundlex.Loader fail. This is caused because Membrane.delete_attribute/2 was not working but it failed silently before.

mix test
==> bundlex
Compiling 8 files (.ex)
warning: unused alias Output
lib/bundlex/toolchain/llvm_mingw.ex:8

warning: unused alias PathHelper
lib/bundlex/toolchain/llvm_mingw.ex:6

could not compile dependency :example_lib, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile example_lib", update it with "mix deps.update example_lib" or clean it with "mix deps.clean example_lib"
==> example_lib
** (Mix) Bundlex: Unable to find Visual Studio root directory. Please ensure that it is either located in "c:\Program Files (x86)\Microsoft Visual Studio *" or VISUAL_STUDIO_ROOT environment variable pointing to its root is set.
==> example

Everybody lies. Readme as well. In this snippet:

defmodule MyApp.BundlexProject do
  use Bundlex.Project

  def project() do
    [
      nif: nif(Bundlex.platform)
    ]
  end

  defp nif(:linux) do
    [
      my_nif: [
        sources: ["something.c", "linux_specific.c"]
      ]
    ]
  end

  defp nif(_platform) do
    [
      my_nif: [
        sources: ["something.c", "multiplatform.c"]
      ]
    ]
  end
end

it suggests that Bundlex.platform/0 returns atom, while actually it returns tuple of platform atom and platform module

Haven't tested compile with mingw. It's unknown.

@mat-hek Why the default source_base path is set to app name? Wouldn't it be better to used nif name or simply nothing (so that c_src would be a source base)?

In visual_studio.ex there is reference to Program Files (x86), which do not exists on 32bit windows.
We should add some check to see which catalog to use.

This issue has been reported to Erlang/OTP issue tracker as I suspect the problem lies there: https://bugs.erlang.org/browse/ERL-1273. Description of the bug there is much more detailed.

On Linux, while building membrane_element_ffmpeg_h264 one may be nondeterministically welcomed with following error during mix compile:

Bundlex: Building natives: parser, decoder, encoder
Compiling 8 files (.ex)

11:09:25.319 [error] Process #PID<0.250.0> raised an exception
** (ArgumentError) argument error
    :erlang.load_nif(:erlang, :apply)
    lib/membrane_element_ffmpeg_h264/parser_native.ex:1: Membrane.Element.FFmpeg.H264.Parser.Native.Nif.load_nif/0
    (kernel 7.0) code_server.erl:1355: anonymous fn/1 in :code_server.handle_on_load/5

11:09:25.321 [warn]  The on_load function for module Elixir.Membrane.Element.FFmpeg.H264.Parser.Native.Nif returned:
{:badarg, [{:erlang, :load_nif, [:erlang, :apply], []}, {Membrane.Element.FFmpeg.H264.Parser.Native.Nif, :load_nif, 0, [file: 'lib/membrane_element_ffmpeg_h264/parser_native.ex', line: 1]}, {:code_server, :"-handle_on_load/5-fun-0-", 1, [file: 'code_server.erl', ...]}]}

A different but similar problem also occurs on macOS during run-time (e.g. mix test):

Bundlex: Building natives: parser, decoder, encoder
..
10:22:00.050 [error] Process #PID<0.304.0> raised an exception
** (RuntimeError) Bundlex cannot load nif "decoder" of app :membrane_element_ffmpeg_h264
from "/Users/mk/membrane-element-ffmpeg-h264-50430eb677b5b3ab44f17357ae161155a35c2ec3/_build/test/lib/membrane_element_ffmpeg_h264/priv/bundlex/decoder", check bundlex.exs file for information about nifs.
Reason: :load_failed, Failed to load NIF library: 'dlopen(/Users/mk/membrane-element-ffmpeg-h264-50430eb677b5b3ab44f17357ae161155a35c2ec3/_build/test/lib/membrane_element_ffmpeg_h264/priv.so, 2): image not found'    (membrane_element_ffmpeg_h264 0.2.0) lib/membrane_element_ffmpeg_h264/decoder_native.ex:1: Membrane.Element.FFmpeg.H264.Decoder.Native.Nif.load_nif/0
    (kernel 7.0) code_server.erl:1355: anonymous fn/1 in :code_server.handle_on_load/510:22:00.087 [warn]  The on_load function for module Elixir.Membrane.Element.FFmpeg.H264.Decoder.Native.Nif returned:
{%RuntimeError{message: "Bundlex cannot load nif \"decoder\" of app :membrane_element_ffmpeg_h264\nfrom \"/Users/mk/membrane-element-ffmpeg-h264-50430eb677b5b3ab44f17357ae161155a35c2ec3/_build/test/lib/membrane_element_ffmpeg_h264/priv/bundlex/decoder\", check bundlex.exs file for information about nifs.\nReason: :load_failed, Failed to load NIF library: 'dlopen(/Users/mk/membrane-element-ffmpeg-h264-50430eb677b5b3ab44f17357ae161155a35c2ec3/_build/test/lib/membrane_element_ffmpeg_h264/priv.so, 2): image not found'\n"}, [{Membrane.Element.FFmpeg.H264.Decoder.Native.Nif, :load_nif, 0, [file: 'lib/membrane_element_ffmpeg_h264/decoder_native.ex', line: 1]}, {:code_server, :"-handle_on_load/5-fun-0-", 1, [file: 'code_server.erl', line: 1355]}]}  1) test Decode 1 240p frame (Decoder.NativeTest)
     test/decoder/decoder_native_test.exs:6
     ** (UndefinedFunctionError) function Membrane.Element.FFmpeg.H264.Decoder.Native.Nif.unifex_create/0 is undefined (module Membrane.Element.FFmpeg.H264.Decoder.Native.Nif is not available)
     code: assert {:ok, decoder_ref} = Dec.create()
     stacktrace:
       (membrane_element_ffmpeg_h264 0.2.0) Membrane.Element.FFmpeg.H264.Decoder.Native.Nif.unifex_create()
       test/decoder/decoder_native_test.exs:11: (test)
.........Finished in 2.8 seconds
12 tests, 1 failure
Randomized with seed 206326 

It seems that C and C++ versions used for compilation are hardcoded. This is very annoying for end users, because even if you add the flag manually it is overwritten in the compiler script.

There are some corner cases such as the lame element, that cannot be handled easily with current approach to determine linker/compiler flags via pkg-config.

LAME library does not ship with pkg-config file, neither on the most popular Linux distributions such as Debian/Ubuntu nor when installed via brew on Mac OS X.

Current requirement to install the .pc file manually is not nice, error-prone and it is not cross platform.

There should be an option in bundlex to define custom callbacks that will determine the proper compiler/linker flags when needed.

When proper flag is set, the build script is saved after it has been executed. If that fails, build script is not available

Unable to compile on Windows with MSVC and Professional 2019.

How to fix?

It's related to a few other issues because this version msvc changed.

I'm trying to cross-compile membrane_portaudio_plugin to run with nerves on an RPi4. Unfortunately it seems bundlex doesn't handle cross-compilation so the resulting output is x86 instead of aarch64.

I've had a look at lib/bundlex/toolchain/gcc.ex and tried adding -march=armv8-a+crc -mtune=cortex-a72 flags but the outputs are still x86.

Does anyone have any tips on supporting cross-compilation?

Today, running mix under high CPU usage i got the following error:

 iex -S mix
Erlang/OTP 21 [erts-10.0.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]


18:06:10.957 [info]  [Porcelain]: goon executable not found
 
18:06:10.957 [info]  [Porcelain]: falling back to the basic driver.

18:06:10.957 [info]  [Porcelain]: (set `config :porcelain, driver: Porcelain.Driver.Basic` or `config :porcelain, goon_warn_if_missing: false` to disable this warning)
Bundlex: !!! Bulding Bundlex Library: :membrane_common_c
Bundlex: ### Reading project
Bundlex: ### Target platform
Bundlex:   - Building for platform linux
Bundlex: ### Toolchain
Bundlex: ### Resolving NIFs
Bundlex:   - Found Erlang includes: ["/usr/lib/erlang/usr/include"]
Bundlex:   - Ignoring export-only nif :membrane
Bundlex:   - Ignoring export-only nif :membrane_ringbuffer
Bundlex:   - Parsing NIF :membrane_shm_payload
Bundlex:   - Parsing NIF :membrane
Bundlex:   - Ignoring export-only nif :membrane_shm_payload_lib
Bundlex: ### Building
Bundlex:   - Running build script
** (File.Error) could not change mode for "bundlex_tmp.sh": no such file or directory
    (elixir) lib/file.ex:1559: File.chmod!/2
    lib/bundlex/build_script.ex:48: Bundlex.BuildScript.store/3
    lib/bundlex/build_script.ex:56: Bundlex.BuildScript.store_tmp/3
    lib/tasks/compile.bundlex.ex:63: Mix.Tasks.Compile.Bundlex.run/1
    (mix) lib/mix/task.ex:314: Mix.Task.run_task/3
    (mix) lib/mix/tasks/compile.all.ex:63: Mix.Tasks.Compile.All.run_compiler/2
    (mix) lib/mix/tasks/compile.all.ex:47: Mix.Tasks.Compile.All.do_compile/4
    (mix) lib/mix/tasks/compile.all.ex:18: anonymous fn/1 in Mix.Tasks.Compile.All.run/1

Looks like race condition to me, running it again worked OK

When project using Bundlex is placed in directory with path containing Unicode characters (such as ł, Ą or ó), building script will fail. The workaround is to set console code page to utf-8 (with chcp 65001 command) or change used directory

• deprecate :nifs and :cnodes keys in the project specification in favour of single :natives key, and require an :interface key in each native instead
• support specifying multiple interfaces for a single native
• support specifying interfaces for libs and automatically import libs with matching interfaces
• tests

Hi! I'm having trouble add this to an umbrella project. I get this error message when running mix compile in the root directory. If I run it in the specific application directory it works ok but it would be much preferred to be able to compile from the root. Are there plans to support umbrella projects or is there a workaround?

** (Mix) Bundlex: Unable to determine app name, check if :app key is present in return value of project/0 in mix.exs

Description

When bundlex is invoked with empty native list then this function concatenate "\n" to the empty list.

  defp join_commands(commands, :unix) do
    Enum.map_join(commands, " && \\\n", &"(#{&1})") <> "\n"
  end

Since Elixir 1.14 Mix.shell().cmd("\n") throws an error resulting in compile error because it tries to run following script

#!/bin/sh
(
)

somehow in 1.13.4 this was not a problem

How to reproduce?

1. Make sure you are using Elixir 1.14 or newer
2. Initialize a project with empty native list

defmodule Example.BundlexProject do
  use Bundlex.Project

  def project() do
    [
      natives: []
    ]
  end
end

2. Run mix compile
3. Voila!

You recommend in the readme to use

compilers: [:bundlex] ++ Mix.compilers

in mix.exs.

I think

compilers: [:bundlex|Mix.compilers]

is a better way :)

It's a minor thing of course.

Here's a (hopefully) brief description of the problem.

We're trying to start an application that only has IPv6 addresses on its local network. When starting the application with export ELIXIR_ERL_OPTIONS="-proto_dist inet6_tcp", the application can not connect to any CNodes that are started by Bundlex.

We've tracked down the fact that Erlang CNodes do not support IPv6 as referenced in the following paragraph from the Erlang docs (emphasis added):

The addr argument of the listen, accept, and connect callbacks refer to appropriate address structure for currently used protocol. Currently ei only supports IPv4. That is, at this time addr always points to a struct sockaddr_in structure.

I fought with this for a while, and so far the only thing that seems to work is to start the parent application epmdless using the ERL_DIST_PORT variable. This allows me to set the RELEASE_NODE variable to a string including the IPv6 address, which will allow application nodes to connect to each other.

The problem is that Bundlex starts the CNode with the same IPv6 address as the host_name part of the node's name. If we hard-code Bundlex to start the CNode with the host name 127.0.0.1, then the parent application starts epmdless, but uses epmd on localhost to connect to the CNodes (and succeeds).

Compilaton fails with following error:

== Compilation error in file lib/membrane_payload_shm_native.ex ==
** (CompileError) lib/membrane_payload_shm_native.ex:1: inlined function createp/2 undefined
    (stdlib) lists.erl:1338: :lists.foreach/2
    (stdlib) erl_eval.erl:677: :erl_eval.do_apply/6

Error most probably comes from after compile callback in loader.ex. Using defnif instead of defnifp fixes the problem

Add postprocessor key to the project specification that will be a module implementing a Bundlex.Project.Postprocessor behaviour with a single callback process, that will be able to extend natives specifications with custom parameters

Add an option flag to mix compile.bundlex that would generate Compilation Database file in the project directory.

This option may be called --store-compiledb.

Alongside, a --dry option may be handy, to only generate compiledb/build script and not execute it.

... such as ninja or make

We happened to introduce a change recently that seems to break builds on macOS. To mitigate this issue in the future I think we should set up a more comprehensive CI suite.

with Visual Studio 2017 Build Tools this toolchain won't work, the directory structure has been changed. I guess it won't be a priority, so the README might mention that

There are several problems when using bundlex on Windows platform (tested on Win 10 and VSC++ 2015 Build Tools):

• Bundlex.Toolchain.VisualStudio, line 46: mkdir fails when the path exists, so it only works 1 time
• Bundlex.Toolchain.VisualStudio, line 46: this path is not surrounded by " and slashes are not converted to Windows format (actually, slashes may remain in form /path/ as long as quotes are used)
• Bundlex.Toolchain.VisualStudio, line 48: same here

On macOS, the compilation is done in the single stage while Linux with GCC creates intermediate .o files stored right next to .c.
The options are:

• Use one pass on both systems
• use two passes on macOS and Linux + fix paths for .o files (they should probably be stored in build catalogue)
• instead of manual script creation, generate a description for an existing build system (like ninja or make)

At this moment, if some native library is missing user will see in most cases info about missing .pc file which isn't too meaningful.

The task is to enhance info about missing native libraries so that user will see info similar to:

for Ubuntu:

You are missing ffmpeg in your OS.
To install it run `apt install ffmpeg`.

for MacOS:

You are missing ffmpeg in your OS.
To install it run `brew install ffmpeg`.

The whole Visual Studio toolchain is based on a script vcvarsall.bat which sets environment variables required for the compilation (including path to the cl.exe compiler). When commands are executed one by one, the result of running that script is not available in subsequent commands and the compilation fails (unless, of course, that script is invoked manually before mix compile)

Update guide with information about new format of bundlex.exs and deprecated keys