Building against macFUSE and JIT-enabled OTP 25 potentially about erlang-efuse

[tucnak]

My attempt to run a local userfs test against the so-called "patched" version of efuse culminated in a spectacular failure, that is none of the tests actually worked out:

  1) test getattr FS implementation represented to OS (noent error) (UserfsTest)
     test/userfs_test.exs:187
     Expected truthy, got false
     code: assert called(TestFs.userfs_getattr(:mock_state, "/e"))
     stacktrace:
       test/userfs_test.exs:194: (test)

WARNING: Deleting data for module 'Elixir.TestFs' imported from
["/opt/badt/userfs/Elixir.TestFs.17779.coverdata",
 "/opt/badt/userfs/Elixir.TestFs_meck_original.17779.coverdata"]


  2) test mount FS init is called (UserfsTest)
     test/userfs_test.exs:30
     ** (exit) exited in: GenServer.call(#PID<0.1086.0>, :stop, 5000)
         ** (EXIT) time out
     stacktrace:
       (elixir 1.13.4) lib/gen_server.ex:1030: GenServer.call/3
       (userfs 1.0.4) lib/userfs.ex:57: Userfs.umount/1
       test/userfs_test.exs:13: anonymous fn/0 in UserfsTest.__ex_unit_setup_0/1
       (ex_unit 1.13.4) lib/ex_unit/on_exit_handler.ex:143: ExUnit.OnExitHandler.exec_callback/1
       (ex_unit 1.13.4) lib/ex_unit/on_exit_handler.ex:129: ExUnit.OnExitHandler.on_exit_runner_loop/0



  3) test umount returns error for not mounted FS (UserfsTest)
     test/userfs_test.exs:63
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:

       1. per test by setting "@tag timeout: x" (accepts :infinity)
       2. per test module by setting "@moduletag timeout: x" (accepts :infinity)
       3. globally via "ExUnit.start(timeout: x)" configuration
       4. by running "mix test --timeout x" which sets timeout
       5. or by running "mix test --trace" which sets timeout to infinity
          (useful when using IEx.pry/0)

     where "x" is the timeout given as integer in milliseconds (defaults to 60_000).

     stacktrace:
       (elixir 1.13.4) lib/system.ex:1065: System.do_port/3
       (elixir 1.13.4) lib/system.ex:1055: System.do_cmd/3
       test/userfs_test.exs:46: UserfsTest.__ex_unit_setup_1/1
       test/userfs_test.exs:1: UserfsTest.__ex_unit__/2
       (ex_unit 1.13.4) lib/ex_unit/runner.ex:493: ExUnit.Runner.exec_test_setup/2
       (ex_unit 1.13.4) lib/ex_unit/runner.ex:452: anonymous fn/2 in ExUnit.Runner.spawn_test_monitor/4
       (stdlib 4.0.1) timer.erl:235: :timer.tc/1
       (ex_unit 1.13.4) lib/ex_unit/runner.ex:451: anonymous fn/4 in ExUnit.Runner.spawn_test_monitor/4

WARNING: Deleting data for module 'Elixir.TestFs' imported from
["/opt/badt/userfs/Elixir.TestFs.17779.coverdata",
 "/opt/badt/userfs/Elixir.TestFs_meck_original.17779.coverdata"]


  4) test list returns a list of mounted filesystems (UserfsTest)
     test/userfs_test.exs:80
     ** (exit) exited in: GenServer.call(#PID<0.1086.0>, :status, 5000)
         ** (EXIT) time out
     code: assert Userfs.list |> Enum.map(fn({pid,_}) -> pid end) |> Enum.member?(pid1)
     stacktrace:
       (elixir 1.13.4) lib/gen_server.ex:1030: GenServer.call/3
       (userfs 1.0.4) lib/userfs/server.ex:47: Userfs.Server.status/1
       (userfs 1.0.4) lib/userfs.ex:87: anonymous fn/1 in Userfs.list/0
       (elixir 1.13.4) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2
       (userfs 1.0.4) lib/userfs.ex:83: Userfs.list/0
       test/userfs_test.exs:87: (test)

And here the humiliation was a bit too much to continue...

[mwri]

The tests no longer pass for me, patched or not (as noted in #1).

I think your change to c_src/Makefile is reasonable, I've not used it for a great number of years now, and clearly it needs some TLC. It looks like it still works though, I just built and ran it with your changes and everything still works as far as I can see.

[mwri]

Had a quick look. Seems what no longer works is efuse:umount. If you run umount from the shell, it's all good, but the umount call in the lib doesn't work, which breaks the unit tests when it comes to rmdir to clean up the mount point, and it is "busy".

[tucnak]

Oh, interesting. Are you going to be committing my change, and if not, would you be willing to accept a pull request into the matter?

Now, umount is important, could you perhaps provide some pointers as to why it fails?

I'm trying to use your other library upon which efuse is a dependency, userfs— to do a funny thing that seems to me an interesting avenue of research— namely, I wish to model supervision trees in run time along with the states as filesystem hierarchy so it could be manipulated via unix shell tools, such as cat and grep. The idea is to be able and run grep like you would normally do over source code, but in this code over program state:

# search for processes referencing "uuid" and among these processes _only_ find a certain pattern
grep -l uuid | xargs grep pattern

In this model, folder hierarchy would represent a supervision tree, and different files would correspond to various debugging outlets, K=V files for the most part. Thus I would be able to debug my live state without either starting the debugger, or leaving Vim!

What do you think? Would userfs be up to the task?

[mwri]

Of course, I'd be happy to accept PRs, no problem.

I don't know why umount fails, when you call userfs / efuse umount it hits the shell to run umount, so, it should be no different to running umount manually from the shell really... I know this is not nice, generally I'd do anything not to hit the shell like this, so I assume there was no other option available to me.

Since it works fine manually I kinda feel there really can't be a lot wrong... Maybe there's some sort of race condition though now with the sequence of events, running umount, the gen server process, the c interface process shutdown, etc.

Your purpose sounds great, it's super useful to have process state available via the filesystem sometimes isn't it? :)

[tucnak]

Interesting: I no longer get umount failures, but now the following two tests fail:

=== Location: [{efuse_SUITE,'-read_files/1-fun-0-',[74](https://github.com/mwri/erlang-efuse/issues/efuse_suite.src.html#74)},
              {lists,foreach_1,1442},
              {efuse_SUITE,read_files,[72](https://github.com/mwri/erlang-efuse/issues/efuse_suite.src.html#72)},
              {test_server,ts_tc,1782},
              {test_server,run_test_case_eval1,1291},
              {test_server,run_test_case_eval,1223}]
=== === Reason: no match of right hand side value "cat: /tmp/erlang_ct_mount_efuse_erlfs/apps/efuse/descr: No such file or directory\n"
  in function  efuse_SUITE:'-read_files/1-fun-0-'/2 (/opt/badt/efuse/test/efuse_SUITE.erl, line 74)
  in call from lists:foreach_1/2 (lists.erl, line 1442)
  in call from efuse_SUITE:read_files/1 (/opt/badt/efuse/test/efuse_SUITE.erl, line 72)
  in call from test_server:ts_tc/3 (test_server.erl, line 1782)
  in call from test_server:run_test_case_eval1/6 (test_server.erl, line 1291)
  in call from test_server:run_test_case_eval/9 (test_server.erl, line 1223)

=== Location: [{efuse_SUITE,'-read_files/1-fun-0-',[74](https://github.com/mwri/erlang-efuse/issues/efuse_suite.src.html#74)},
              {lists,foreach_1,1442},
              {efuse_SUITE,read_files,[72](https://github.com/mwri/erlang-efuse/issues/efuse_suite.src.html#72)},
              {test_server,ts_tc,1782},
              {test_server,run_test_case_eval1,1291},
              {test_server,run_test_case_eval,1223}]
=== === Reason: no match of right hand side value "cat: /tmp/erlang_ct_mount_efuse_examplefs/file1: No such file or directory\n"
  in function  efuse_SUITE:'-read_files/1-fun-0-'/2 (/opt/badt/efuse/test/efuse_SUITE.erl, line 74)
  in call from lists:foreach_1/2 (lists.erl, line 1442)
  in call from efuse_SUITE:read_files/1 (/opt/badt/efuse/test/efuse_SUITE.erl, line 72)
  in call from test_server:ts_tc/3 (test_server.erl, line 1782)
  in call from test_server:run_test_case_eval1/6 (test_server.erl, line 1291)
  in call from test_server:run_test_case_eval/9 (test_server.erl, line 1223)

Now, these are the most important calls... am I doing something wrong perhaps?

[tucnak]

Your purpose sounds great, it's super useful to have process state available via the filesystem sometimes isn't it? :)

Yeah, that's the thing— process state over the filesystem. I feel like we'd gone in a wrong direction with some of the profiler tools that are so common these days. And now having to learn Elixir/Erlang of all things at a new shop, I have to fall back to what I'm used to— the Unix tools, Vim, et cetera. Bonus points if I could add quickfix capability for run-time failures akin to how it interacts with the output of make/gcc for compile-time failures. I could use this as an opportunity to learn Elixir properly, so the sooner I can get over this hurdle in the form of efuse— Erlang binding that is preventing me from playing with it.

P.S. Sorry for bothering you with this, I know being a FOSS maintainer myself how annoying it can be to address these issues years on when you have moved on by then; it's just that I don't have anyone to ask for advice when it comes to Erlang so if you could point me in a direction of a mailing list, or forum— where schmuck like me can reach out for help— I would really, really appreciate it.

Fuse is an important piece of software, I feel like it's to everyone's benefit.

[mwri]

The no match of right hand side value "cat: /tmp/erlang_ct_mount_efuse_erlfs/apps/efuse/descr: No such file or directory\n" is straight forward to interpret, assuming line 74 is the same as my code, it is ExpectContent = os:cmd("cat "++MountDir++"/"++Filename). You can inspect ExpectContent to see what it is I guess, but whatever it is, the return value of os:cmd("cat "++MountDir++"/"++Filename) is expected to match, and the no match of right hand side value "cat: /tmp/erlang_ct_mount_efuse_erlfs/apps/efuse/descr: No such file or directory\n" error means that the right hand side was "cat: /tmp/erlang_ct_mount_efuse_erlfs/apps/efuse/descr: No such file or directory\n".

It's probably clear enough that whatever ExpectContent is, the result of the unix cat command was that error from the shell, so the /tmp/erlang_ct_mount_efuse_erlfs/apps/efuse/descr file didn't exist.

I imagine it must be because the mount failed or crashed fairly catastrophically, so it's not mounted. The mount above does say "Ok" though, but it could have mounted and then crashed I guess. I'd expect there to be something useful in the ct logs to be honest...