Not able to run tests. about mal

[kanaka]

@Moniya there are no tests for step0 (I may add some at some point just for completeness). Can you post the output that you get from running make test^ruby^step1? What OS are you on?

[moniyax]

@kanaka I'm on OSX Lion (10.7.5). I experience the same issue running other steps, too. Here's the output when I run make test^ruby^step1:

Testing test^bash^step1, step file: bash/step1_read_print.sh, test file: tests/step1_read_print.mal
Running: ../runtest.py  ../tests/step1_read_print.mal -- bash ../bash/step1_read_print.sh
Started with:
../bash/types.sh: line 8: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]
../bash/types.sh: line 8: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]

Testing read of nil/true/false
TEST: nil -> ['','nil'] -> SUCCESS
TEST: true -> ['','true'] -> SUCCESS
TEST: false -> ['','false'] -> SUCCESS
Testing read of numbers
TEST: 1 -> ['','1'] -> SUCCESS
TEST: 7 -> ['','7'] -> SUCCESS
TEST:   7    -> ['','7'] -> SUCCESS
Testing read of symbols
TEST: + -> ['','+'] -> SUCCESS
TEST: abc -> ['','abc'] -> SUCCESS
TEST:    abc    -> ['','abc'] -> SUCCESS
TEST: abc5 -> ['','abc5'] -> SUCCESS
TEST: abc-def -> ['','abc-def'] -> SUCCESS
Testing read of strings
TEST: "abc" -> ['','"abc"']

Completes with the following after a long pause:

TEST: "abc" -> ['','"abc"'] -> FAIL (line 34):
    Expected : '"abc"\r\n"abc"'
    Got      : None
TEST:    "abc"    -> ['','"abc"']Traceback (most recent call last):
  File "../runtest.py", line 173, in <module>
    r.writeline(form)
  File "../runtest.py", line 84, in writeline
    self.stdin.write(str + "\n")
IOError: [Errno 5] Input/output error
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/anaconda/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "../runtest.py", line 92, in cleanup
    os.killpg(self.p.pid, signal.SIGTERM)
OSError: [Errno 1] Operation not permitted
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/anaconda/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "../runtest.py", line 92, in cleanup
    os.killpg(self.p.pid, signal.SIGTERM)
OSError: [Errno 1] Operation not permitted
make: *** [test^bash^step1] Error 1

[moniyax]

@kanaka I'm excited about your project and can't wait to get on board. Well done. Thanks ;)

[kanaka]

@Moniya Okay, looks like a couple of different problems that are probably related to OS X (I don't have an easy OS X test system so it's not surprising really).

• something weird is going on with the makefile rules such that maybe the default rule is being run instead of the one you specify. What version of make do you have? make --version
• there is a non-fatal complaint at cleanup for the test harness (runtest.py).

Let's see if we can't at least get you a workaround. Try this:

cd ruby
../runtest.py  ../tests/step1_read_print.mal -- ruby ../ruby/step1_read_print.rb

You'll probably see some errors when the test tries to cleanup (and it may leave some orphan child processes around) but maybe that will at least allow you to run tests.

[moniyax]

@kanaka My make is GNU Make 3.81. Tried the workaround. It did work around well. Thanks. :) A shame I'm not good with Make, otherwise I'd gladly contribute a fix.

As an aside, I tried the workaround on the Racket implementation (step 4) several times. It ran well, but sometimes test failures were reported, sometimes not. And the number of failures reported on test runs with failures varied. Is that normal? I don't see the issue with the Ruby implementation.

I have Racket 6.1 though, which I see is more recent than the version here.

[kanaka]

@Moniya I don't see that variation on my system (Ubuntu 14.04) with Racket v5.3.6. I just ran it 10 times in a row with no variation. If you're only seeing that with Racket then it is possible there is something odd with newer Racket. If you see it with other languages, then it's more likely that the use of pseudo-terminals (pty) in runtest.py is being affected by some difference of OS X.

[def-]

Maybe it has to do with your shell using ^ as a special character. ZSH does that for example. Try make "test^ruby^step1".

[moniyax]

@def- Thanks! Using double quotes solved the issue. Also, running the original command on bash gave no issues. You are right. Zsh is the culprit here. I remember having problems running git commands that contained carets in Zsh.

[moniyax]

@kanaka I switched to Racket v5.3.6 and still ran into the same issue. I'll see if I can look into it. I'm not seeing the issue with other languages I've tried: Ruby, Python.

Oh well, PHP is acting up, too. Same issue. The issue is not Racket specific.

[kanaka]

@def- good catch. I didn't realize that was a special character in zsh.

@Moniya it's possible that it's a performance related issue. Racket may be sending output faster than ruby or python and that may be what's triggering the issue for the harness. That's just a wild guess though. I'd be interested in output from a failing case vs a passing case (just the part of the test that failed vs not).

[moniyax]

@kanaka I'll soon post something consistent about the failures that I find interesting and might be helpful in solving the issue. I need to find power for my laptop as the battery has run out .

[moniyax]

@kanaka Output for two test cases when they succeed and when they fail:

TEST: (count (list 1 2 3)) -> ['','3'] -> SUCCESS

TEST: (count (list 1 2 3)) -> ['','3'] -> FAIL (line 16):
    Expected : '(count (list 1 2 3))\r\n3'
    Got      : '(count (list 1 2 3))\r\n(count (list 1 2 3))\r\n3'

TEST: (if (> (count (list 1 2 3)) 3) "yes" "no") -> ['','"no"'] -> SUCCESS

TEST: (if (> (count (list 1 2 3)) 3) "yes" "no") -> ['','"no"'] -> FAIL (line 22):
    Expected : '(if (> (count (list 1 2 3)) 3) "yes" "no")\r\n"no"'
    Got      : '(if (> (count (list 1 2 3)) 3) "yes" "no")\r\n(if (> (count (list 1 2 3)) 3) "yes" "no")\r\n"no"'

All the failures exhibit this pattern: the tested expression being duplicated in the result. Looks like a race issue. Also, a lot more tests were failing when my battery was low and the system culled performance than when it was operating normally.

As earlier stated, the PHP implementation also gave the same issue.

[kanaka]

@Moniya mind testing again with upstream changes to runtest? There is a chance some recent changes may have fixed this. Thanks.

[moniyax]

@kanaka The issue still occurs, but much less frequently (like 1 in 20 tries), and the number of failing test cases are fewer too, just 1 or 2 failures.

[moniyax]

Umm... It's kinda frequent now, but failing cases just 1 or 2.

[kanaka]

So it was infrequent and then suddenly became more frequent? Anything change in the meantime?

Are you still seeing this with the racket and php implementations only? Does it tend to be the same test cases that fail when it does fail?

[moniyax]

I think the failures got less frequent because I just got power on my laptop (MBA 1.7 GHz) and things got a little faster (power supply is often unavailable where I live).

Yes the Racket and PHP implementations only.

Each case fails or passes randomly. I illustrated the nature of the failures in an earlier comment. #30 (comment)

[moniyax]

I'm experiencing a memory-related issue running the Racket implementation in Ubuntu 14.10 on Virtual Box. Should I start another issue or post it here?

[kanaka]

Another issue. Is it perhaps leaving orphan processes around (not cleaning them up properly)?

[moniyax]

O well... That memory issue has stopped happening after yet another restart.

Maybe you can ignore that.
I swear I'm not doing anything crazy with my computer. :)

[moniyax]

P.S: These issues are not in any way hindering my progress on the project. I just deem it useful to bring them to your notice.

[keith-rollin]

FWIW, I also see the error that moniya reports where the input text is duplicated once every 20 tests or so. The issue is random. Running the same test over and over will result in the condition occurring on different test cases. Occasionally, the test runs with no errors.

This is on Mac OS X 10.10.2 with the Swift implementation I'm working on. I'm building against mal b020aa3. I've not done anything to try to track down the source of the problem.

Here's an example from the test for step4:

...
Testing vector equality
TEST: (= [] (list)) -> ['','true'] -> SUCCESS
TEST: (= (list 1 2) [1 2]) -> ['','true'] -> SUCCESS
TEST: (= (list 1) []) -> ['','false'] -> SUCCESS
TEST: (= [] [1]) -> ['','false'] -> FAIL (line 374):
    Expected : '(= [] [1])\r\nfalse'
    Got      : '(= [] [1])\r\n(= [] [1])\r\nfalse'
TEST: (= 0 []) -> ['','false'] -> FAIL (line 376):
    Expected : '(= 0 [])\r\nfalse'
    Got      : '(= 0 [])\r\n(= 0 [])\r\nfalse'
...

[kanaka]

@keith-rollin @Moniya I've pushed a hack which I think should workaround this issue. I now check for the expected output and I also check for the case where the input form is echo'd twice in the output. I'm going to close the issue because I'm pretty confident it will work around it, but can you two confirm that it has resolved it for you? If not, I'll re-open it.

[moniyax]

It worked. Well done. 👍

[kanaka]

@Moniya yay!

[kanaka]

The ^ issue should be fixed by #128

[kanaka]

I think all the problems reflected in this issue have been resolved so I'm closing it.