HaskellKernel times out running tests in a clean repo about l4v

[sorear]

Limiting the number of threads with taskset is counterproductive, user time goes way up. Setting jobs: 1 in ~/.cabal/config, on the other hand, is wildly effective:

user@in-container:/host/r/l4v$ git clean -dfx
Removing misc/regression/__pycache__/
Removing spec/haskell/.cabal-sandbox/
Removing spec/haskell/.stack-work/
Removing spec/haskell/cabal.sandbox.config
Removing spec/haskell/stack.yaml.lock
user@in-container:/host/r/l4v$ time ./run_tests  -j1 HaskellKernel --no-timeouts
Testing for L4V_ARCH=ARM:
Testing Orphanage for ARM
Running 1 test(s)...
  Started HaskellKernel ...         
  Finished HaskellKernel             passed      ( 0:01:52 real,  0:01:41 cpu,  0.74GB)


1/1 tests succeeded.

All tests passed.


real    1m53.019s
user    1m47.143s
sys     0m6.447s

Setting jobs: 64 goes back to the old timings, so I don't think I'm missing a cache here.

[lsf37]

Very strange. Are you using one of the docker images to build from? Which one?

[sorear]

trustworthysystems/l4v-riscv

[lsf37]

Thanks. I still can't reproduce. I've set jobs: 64 in ~/.cabal/config, but am still getting a normal run.

Is the rest of the ~/.cabal/config from the container or mounted from your user home? If from user, would you be Ok posting the rest of it? Maybe there is some other option that conflicts with something.

[heb1001]

I think I saw a problem with the same symptoms. On the first run for a clean build there is some stuff that gets downloaded. There were two downloads causing problems. The first one was: stack —install-ghc build cabal-install The second one was: stack exec — ./stack-path cabal v1-update If you start with a clean build and throttle the network (to maybe 10kB/s) you ought to be able to recreate the timeouts. Harry

…

On Thu, Sep 10, 2020 at 5:00 AM Gerwin Klein ***@***.***> wrote: Thanks. I still can't reproduce. I've set jobs: 64 in ~/.cabal/config, but am still getting a normal run. Is the rest of the ~/.cabal/config from the container or mounted from your user home? If from user, would you be Ok posting the rest of it? Maybe there is some other option that conflicts with something. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#74 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCZLLRJTTH5V3MG3MFJAM3SFBFOPANCNFSM4Q4EFHZA> .

[lsf37]

Ok, so these would be download issues, which we could fix larger timeouts. I don't think there's much we can do about making them faster. Downloading an entire ghc installation + a bunch of cabal packages is relatively large if anything happens with the network.

Maybe we could try to provide that within the docker container, though. That would at least all come from the same source then (docker).

@sorear is that also what you're seeing or do you think your problem is different?

[heb1001]

Perhaps a separate Haskell test with a longer timeout just before/after the Isabelle test to force the download.

…

On Thu, Sep 10, 2020 at 9:31 AM Gerwin Klein ***@***.***> wrote: Ok, so these would be download issues, which we could fix larger timeouts. I don't think there's much we can do about making them faster. Downloading an entire ghc installation + a bunch of cabal packages is relatively large if anything happens with the network. Maybe we could try to provide that within the docker container, though. That would at least all come from the same source then (docker). @sorear <https://github.com/sorear> is that also what you're seeing or do you think your problem is different? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#74 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCZLLTAPJIRD3TRXXRTLC3SFCFGJANCNFSM4Q4EFHZA> .

[sorear]

I believe I am seeing a different issue. When it times out, there are 64 ghc processes running at near-full CPU.

[lsf37]

Reading about the default for jobs on the cabal bug tracker, the main complaint is that a too high number of jobs might run out of memory very quickly. That doesn't seem to be the case here, though, at least not according to the output of time above.

Do lower job numbers like 2, 4 or 8 have the same problem? I could just set them manually to something like 2 for this sandbox -- the Haskell compile time is very low compared to the proofs, so I don't think we'd mind a bit less parallelism.

On the off chance that he's seen something like this before, let me also ping @mbrcknl for this.

[mbrcknl]

The current timeout is based on total cpu time, not real time. I don't think I've ever tried this on a box with 64 CPUs. :-) With that many, you'll get less than 2 minutes real time if they're fully occupied. I guess the single-threaded speed might be slower than on the 4-6 core boxes we're accustomed to, so 30 minutes total CPU time is not enough? That's the only possible explanation I can think of so far.

I suggest changing the cpu-timeout=1800 in spec/tests.xml to timeout=600, and see what happens.

[mbrcknl]

Sorry, I hadn't fully registered your comment that with jobs: 1, it finishes in even less time. In light of that, my previous comment doesn't make sense.

That is indeed very strange. I suppose you've tried building some other random large Haskell packages on this machine, without seeing this issue?

[lsf37]

Since we don't seem to be able to get to the bottom of it, and we do have a workaround (setting jobs: 1 in ~/.cabal/config) I'll close the issue.

If anyone feels like investigating deeper and/or new problems turn up with this, feel free to re-open.