dnlserrano / exavier Goto Github PK
View Code? Open in Web Editor NEWElixir mutation testing library
License: MIT License
Elixir mutation testing library
License: MIT License
Hi!
Thanks for your work on exavier, it looks like a very interesting project! :)
I'm aware that this is mostly still a proof-of-concept, but when I just tried to run it on one of our projects, it failed because it generated invalid code. Specifically, it interpreted the /
in a function reference (i.e. something like &some_function/1
) as a division symbol and tried to mutate it to &some_function * 1
, which is not valid elixir code anymore...
So I guess, I would expect function references to be left alone, or alternatively maybe compilation errors could be caught and respective mutations be discarded?
Exavier
does not infer correctly some modules with dots in their names.
The error message:
MIX_ENV=test mix exavier.test
13:50:30.016 [error] Could not find module Elixir.ExavierCheck.Main inferred from test file test/exavier_check/main_test.exs. You can define your overrides using the :test_files_to_modules option in exavier.
13:50:30.019 [error] GenServer Exavier.Server terminating
** (MatchError) no match of right hand side value: :ok
(exavier 0.3.0) lib/exavier/cover.ex:8: Exavier.Cover.lines_to_mutate/2
(exavier 0.3.0) lib/exavier/server.ex:20: anonymous fn/2 in Exavier.Server.handle_call/3
(elixir 1.13.0) lib/enum.ex:2396: Enum."-reduce/3-lists^foldl/2-0-"/3
(exavier 0.3.0) lib/exavier/server.ex:18: Exavier.Server.handle_call/3
(stdlib 3.16.1) gen_server.erl:721: :gen_server.try_handle_call/4
(stdlib 3.16.1) gen_server.erl:750: :gen_server.handle_msg/6
(stdlib 3.16.1) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
I prepared a repository to reproduce the issue: https://github.com/denispeplin/exavier_check
To reproduce, checkout the repository and run
mix deps.get
MIX_ENV=test mix exavier.test
Hi! Whenever I run MIX_ENV=test mix exavier.test
, this is the output:
..21:47:16.995 [error] GenServer #PID<0.811.0> terminating
** (stop) killed
Last message: {:EXIT, #PID<0.667.0>, :killed}
21:47:16.995 [error] GenServer #PID<0.827.0> terminating
** (stop) killed
Last message: {:EXIT, #PID<0.664.0>, :killed}
21:47:16.995 [error] GenServer Exavier.Server terminating
** (stop) exited in: Task.Supervised.stream(5000)
** (EXIT) time out
(elixir 1.11.3) lib/task/supervised.ex:304: Task.Supervised.stream_reduce/7
(elixir 1.11.3) lib/enum.ex:3473: Enum.reverse/1
(elixir 1.11.3) lib/enum.ex:3066: Enum.to_list/1
(exavier 0.3.0) lib/exavier/server.ex:59: Exavier.Server.handle_call/3
(stdlib 3.14.2) gen_server.erl:715: :gen_server.try_handle_call/4
(stdlib 3.14.2) gen_server.erl:744: :gen_server.handle_msg/6
(stdlib 3.14.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.95.0>): :xmen
** (EXIT from #PID<0.95.0>) exited in: Task.Supervised.stream(5000)
** (EXIT) time out
Can this be linked to #1?
This is my Erlang/Elixir info:
$ elixir --version
Erlang/OTP 23 [erts-11.2.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Elixir 1.11.3 (compiled with Erlang/OTP 23)
Thank you!
When I try running mix exavier.test
, it terminates with an error like this (with a different module named each time):
[error] Task #PID<0.6433.0> started from Exavier.Server terminating
** (RuntimeError) cannot add sync case named MyApp.SomeTest to test suite after the suite starts running
(ex_unit 1.12.1) lib/ex_unit/server.ex:21: ExUnit.Server.add/2
(elixir 1.12.1) src/elixir_module.erl:369: anonymous fn/5 in :elixir_module.expand_callback/6
(elixir 1.12.1) src/elixir_module.erl:368: :elixir_module.expand_callback/6
(stdlib 3.14.2.1) lists.erl:1267: :lists.foldl/3
(elixir 1.12.1) src/elixir_module.erl:150: :elixir_module.compile/5
(stdlib 3.14.2.1) erl_eval.erl:680: :erl_eval.do_apply/6
(elixir 1.12.1) src/elixir.erl:280: :elixir.recur_eval/3
(elixir 1.12.1) src/elixir.erl:265: :elixir.eval_forms/3
(elixir 1.12.1) src/elixir_compiler.erl:46: :elixir_compiler.eval_forms/3
(elixir 1.12.1) src/elixir_lexical.erl:15: :elixir_lexical.run/3
(elixir 1.12.1) src/elixir_compiler.erl:18: :elixir_compiler.quoted/3
(elixir 1.12.1) lib/code.ex:1261: Code.require_file/2
(exavier 0.3.0) lib/exavier/server.ex:49: anonymous fn/5 in Exavier.Server.handle_call/3
(elixir 1.12.1) lib/enum.ex:930: Enum."-each/2-lists^foreach/1-0-"/2
(elixir 1.12.1) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
(elixir 1.12.1) lib/task/supervised.ex:35: Task.Supervised.reply/5
(stdlib 3.14.2.1) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Function: &:erlang.apply/2
Args: [#Function<2.82788263/1 in Exavier.Server.handle_call/3>, [{"test/my_app/some_test.exs", %{file: '/Users/whoever/my_app/apps/my_app/lib/my_app/some.ex', lines_to_mutate: [16, 22, 25, 26], module: MyApp.Some}}]]
Exavier 0.3.0
Elixir 1.12.1
Erlang/OTP 23
More context on this original post over at ElixirForum.com where I first explained it.
Right now I’m not running each test ... do
individually but instead I’m running the whole test module (e.g., HelloWorldTest
). This has one clear disadvantage, which I’ll explain below with an example:
defmodule HelloWorld do
def sum(a, b) do: a + b
def divide(a, b), do: div(a, b)
end
defmodule HelloWorldTest do
test "when testing sum" do
assert HelloWorld.sum(3, 0) == 3
end
test "when testing divide" do
assert HelloWorld.divide(5, 2) == 3
end
end
If I change code to the following:
defmodule HelloWorld do
def sum(a, b) do: a - b # changed from + to - via AOR1
def divide(a, b), do: div(a, b)
end
I will be running the tests for both tests, instead of just running the test for sum/2
(i.e., "when testing sum"
, which was the only one for which the corresponding source code changed). In order to try and maximise the amount of mutations I can catch with running the entire test module, I mutate all in one go. Does that make sense? Maybe it doesn’t… 🤦♂️ AFAIU, finding out what tests I should run per source code change is hard (or at least not trivial? 😅). But I might not be seeing something very obvious.
Hope people can help out with ideas for this one. Maybe traversing tests and annotating to keep track of what operators are present in each and hence are influenced/swing based on mutations performed? 🤔
Hey @dnlserrano! I tried this out on my norm library and I got an argument error from the reporter. It looks like its trying to convert a string to an existing atom and it can't find the atom or something similar. Here's the full output:
keathley in ~/D/norm on improve-docs
λ env MIX_ENV=test mix exavier.test
Compiling 11 files (.ex)
Generated norm app
08:47:04.990 [error] GenServer :exavier_reporter terminating
** (ArgumentError) argument error
:erlang.binary_to_existing_atom("Elixir.Norm.Selection", :utf8)
(exavier) lib/exavier.ex:75: Exavier.string_to_elixir_module/1
(exavier) lib/exavier/reporter.ex:34: Exavier.Reporter.handle_cast/2
(stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
(stdlib) gen_server.erl:711: :gen_server.handle_msg/6
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:test_finished, %ExUnit.Test{case: Norm.SelectionTest, logs: "", module: Norm.SelectionTest, name: :"test generation can generate values", state: nil, tags: %{async: true, case: Norm.SelectionTest, describe: "generation", describe_line: 58, file: "/Users/keathley/Development/norm/test/norm/selection_test.exs", line: 59, module: Norm.SelectionTest, registered: %{}, test: :"test generation can generate values", test_type: :test}, time: 190}}}
State: %Exavier.Reporter{all_failures: [], mutated_modules: %{}}
** (EXIT from #PID<0.92.0>) an exception was raised:
** (ArgumentError) argument error
:erlang.binary_to_existing_atom("Elixir.Norm.Selection", :utf8)
(exavier) lib/exavier.ex:75: Exavier.string_to_elixir_module/1
(exavier) lib/exavier/reporter.ex:34: Exavier.Reporter.handle_cast/2
(stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
(stdlib) gen_server.erl:711: :gen_server.handle_msg/6
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
I'm running on elixir 1.9.1 and OTP 22 if that helps. I've also tested this with the most recent version of exavier on hex and with the master branch. The failure is non-deterministic and I suspect that just reporting whatever test it receives first since they're running async. I suspect that it would be safe to use to_atom
here since this will really only be run in test mode and we don't really have to worry about DOSing the atom table. But I figured I'd leave that up to you.
Hey @dnlserrano, thanks for your work on exavier
! Our team'd like to give it a shot.
We noticed that in case when the function returns negative number, e.g:
def something(_arg1), do: -1
exavier
will mutate this -1
into *1
and here's gonna be an error like so:
** (CompileError): undefined function */1
So we fixed it in our fork in this PR.
Is there a way to skip certain test files altogether? Not all of my test files map one-to-one to a module. For example, I have some test files that only test protocol implementations in my project, so there is no Elixir module to be found. I tried setting the values in the :test_files_to_modules
option to nil
or false
, but that doesn't seem to be supported.
On this line:
The subject
should be ROR1, not AOR1. The test cases are also testing AOR1's replacements.
I'm in the middle of another PR; I wanted to write this down before I forgot.
Howdy!
So I decided to give this a quick try today, and it turns out that having to set configuration in a config.exs
file is a bit of an issue since the code I'm trying to test is a library, and I don't want this config ending up in other people's applications. It would be good if this library configuration could follow the configuration guidelines set out in the documentation here so this problem could be avoided.
For example, I'm trying to test benchee, but we don't even have any mix config in there (for the reasons laid out by the documentation there), and it won't run at the moment without some configuration (custom mappings), so this is a big of a sticky issue for users initial experience. I think either a dotfile that can be .gitignore'd (like Credo does) or some sort of function call somewhere would be a nice place for this configuration to reside.
Thanks for the great start towards a really nice mutation testing library for our community!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.