nix scripts shared across projects
This repo contains build code and tools shared between IOHK projects.
Please document any change that might affect project builds in the ChangeLog. For example:
nixpkgs to a different branch.
![iohk-bors[bot] avatar](https://avatars.githubusercontent.com/in/17473?v=4 "iohk-bors[bot]")
Trying to update Plutus to 8.4.4 and all of my .envs (which use shellFor) are broken, which is annoying. Possibly we need to backport something to our branch?
There's also something weird going on since it seems to show it's making a GHC 8.4.3 with packages instead of 8.4.4.
You can check this out on my branch: https://github.com/michaelpj/plutus/tree/imp/bump-versions
Log file:
log.txt
This would (hopefully) give us some improved caching of artifacts in IOHK-nix.
Currently I'm getting a release.nix failure: https://buildkite.com/input-output-hk/cardano-node/builds/664#ab65323b-1cf2-481b-8880-30f954a73649
I’ve fixed this before by modifying the offending nix derivations with the following:
(if system.isWindows
then [ (hsPkgs.Win32) ]
else [ (hsPkgs.unix) ])
However the derivations that are failing already have this conditional. I'm not sure why this is still failing.
Previously, I thought I had to patch out uses of TH to get the windows and static build of my project working, but @angerman said encouraging words, so I tried to make use of it.
Judging from the comments in some file, I need to add the iohk-nix module when I build pkgs.nix, which I tried to do here:
entropia/tip-toi-reveng@37d2eb6
But it does not work out of the box:
~/projekte/tip-toi-reveng $ nix-build -A windows-exe nix
error: The option `packages.remote-iserv.package.license' is used but not defined.
(use '--show-trace' to show detailed location information)
and with full trace:
~/projekte/tip-toi-reveng $ nix-build -A windows-exe nix --show-trace
error: while evaluating 'tttool-exe' at /home/jojo/projekte/tip-toi-reveng/nix/default.nix:4:16, called from /home/jojo/projekte/tip-toi-reveng/nix/default.nix:44:17:
while evaluating the attribute 'tttool.components.exes.tttool' at undefined position:
while evaluating 'buildComp' at /nix/store/vvy3nghh9n50009j8xlld3jgd45damzy-haskell-lib-source/builder/default.nix:77:28, called from undefined position:
while evaluating anonymous function at /nix/store/vvy3nghh9n50009j8xlld3jgd45damzy-haskell-lib-source/builder/comp-builder.nix:3:1, called from /nix/store/vvy3nghh9n50009j8xlld3jgd45damzy-haskell-lib-source/builder/default.nix:77:39:
while evaluating 'fix' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/fixed-points.nix:19:9, called from /nix/store/vvy3nghh9n50009j8xlld3jgd45damzy-haskell-lib-source/builder/comp-builder.nix:195:4:
while evaluating anonymous function at /nix/store/vvy3nghh9n50009j8xlld3jgd45damzy-haskell-lib-source/builder/comp-builder.nix:195:20, called from /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/fixed-points.nix:19:20:
while evaluating 'filterAttrs' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/attrsets.nix:115:23, called from /nix/store/vvy3nghh9n50009j8xlld3jgd45damzy-haskell-lib-source/builder/comp-builder.nix:339:4:
while evaluating anonymous function at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/attrsets.nix:116:29, called from /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/attrsets.nix:116:18:
while evaluating anonymous function at /nix/store/vvy3nghh9n50009j8xlld3jgd45damzy-haskell-lib-source/lib/default.nix:65:39, called from /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/attrsets.nix:116:62:
while evaluating the attribute 'preBuild' at /nix/store/vvy3nghh9n50009j8xlld3jgd45damzy-haskell-lib-source/builder/comp-builder.nix:340:10:
while evaluating anonymous function at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:75:45, called from undefined position:
while evaluating the attribute 'value' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:336:9:
while evaluating the option `packages.tttool.preBuild':
while evaluating the attribute 'isDefined' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:374:5:
while evaluating the attribute 'values' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:362:9:
while evaluating the attribute 'values' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:458:7:
while evaluating anonymous function at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:348:28, called from /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:348:17:
while evaluating 'dischargeProperties' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:416:25, called from /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:349:62:
while evaluating the attribute 'value' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:232:58:
while evaluating the attribute 'config.preBuild' at /nix/store/nqciwbgh34f17lw993h99bizqi8l3a58-source/mingw_w64.nix:78:13:
while evaluating 'assertValidity' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/pkgs/stdenv/generic/check-meta.nix:234:20, called from /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/pkgs/stdenv/generic/make-derivation.nix:249:18:
while evaluating 'checkValidity' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/pkgs/stdenv/generic/check-meta.nix:217:19, called from /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/pkgs/stdenv/generic/check-meta.nix:235:18:
while evaluating 'hasDeniedUnfreeLicense' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/pkgs/stdenv/generic/check-meta.nix:60:28, called from /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/pkgs/stdenv/generic/check-meta.nix:218:8:
while evaluating 'isUnfree' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/pkgs/stdenv/generic/check-meta.nix:45:14, called from /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/pkgs/stdenv/generic/check-meta.nix:63:5:
while evaluating 'toList' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/lists.nix:222:12, called from /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/pkgs/stdenv/generic/check-meta.nix:63:15:
while evaluating the attribute 'meta.license' at /nix/store/vvy3nghh9n50009j8xlld3jgd45damzy-haskell-lib-source/builder/comp-builder.nix:216:5:
while evaluating the attribute 'license' at undefined position:
while evaluating anonymous function at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:75:45, called from undefined position:
while evaluating the attribute 'value' at /nix/store/fm6z30m3jkxg2ycni23aij9zls4nqq3c-source/lib/modules.nix:336:9:
while evaluating the option `packages.remote-iserv.package.license':
The option `packages.remote-iserv.package.license' is used but not defined.
Is it not possible to just plug the iohk-module into a “pure” haskell.nix setup?
Also, is there a reason why this template haskell support is not in haskell.nix proper?
(context: input-output-hk/cardano-ledger-byron#366)
The following warnings are shown when installing in MinGW:
❯ curl -sL https://github.com/input-output-hk/iohk-nix/releases/latest/download/msys2.libblst.pkg.tar.zstd > libblst.pkg.tar.zstd
❯ pacman --noconfirm -U libblst.pkg.tar.zstd
loading packages...
warning: mingw-w64-x86_64-blst-03b5124-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...
Packages (1) mingw-w64-x86_64-blst-03b5124-1
:: Proceed with installation? [Y/n]
(1/1) checking keys in keyring [#################################################] 100%
(1/1) checking package integrity [#################################################] 100%
(1/1) loading package files [#################################################] 100%
(1/1) checking for file conflicts [#################################################] 100%
(1/1) checking available disk space [#################################################] 100%
:: Processing package changes...
(1/1) reinstalling mingw-w64-x86_64-blst [#################################################] 100%
warning: directory permissions differ on /mingw64/opt/cardano/
filesystem: 755 package: 555
warning: directory permissions differ on /mingw64/opt/cardano/bin/
filesystem: 755 package: 555
warning: directory permissions differ on /mingw64/opt/cardano/include/
filesystem: 755 package: 555
warning: directory permissions differ on /mingw64/opt/cardano/lib/
filesystem: 755 package: 555
warning: directory permissions differ on /mingw64/opt/cardano/lib/pkgconfig/
filesystem: 755 package: 555It's out now.
Not sure what the blockers are here - do we need to still use our fork and make a new patched branch, or is enough stuff upstream now?
PS is: IntersectMBO/cardano-base#28
Build log at: https://buildkite.com/input-output-hk/cardano-base/builds/63#c9b36d79-a00e-4f68-9b9c-62aa6b53203e
Error:
"required": {
"error": "attribute 'lts-13.26' missing, at /nix/store/avrzla5qcblrh9myl554s9bw24a7p737-source/default.nix:91:19"
No idea how to fix this.
iohk-nix/jormungandr-lib/versions.json
Line 275 in 77a51e8
There is wrong nonexistent revision of jormungandr 0.8.11: 3ee87e845813fd151114a19be678b5a65798a649. It should be 241e01e6a2b13dc66380b043b8fe0fa1917f3469.
Failure is on PR : input-output-hk/cardano-shell#202
Build kite log : https://buildkite.com/input-output-hk/cardano-shell/builds/697#2680a4c8-a22f-4866-9aa8-a0a71a561445
Process exited with code: ExitFailure 1
Logs have been written to: /var/lib/buildkite-agent/builds/buildkite-packet-3/input-output-hk/cardano-shell/.stack-work/logs/libsystemd-journal-1.4.4.log
Configuring libsystemd-journal-1.4.4...
Cabal-simple_mPHDZzAJ_2.4.1.0_ghc-8.6.5: The program 'pkg-config' version >=0.9.0 is required but it could not be found.
Another issue trying to update Plutus to 8.4.4. For some reason it's getting conflicting versions of template-haskell. I wonder if this is related to the oddness in #27 where we seemed to unexpectedly have a GHC 8.4.3 in there.
You can see this on my branch: https://github.com/michaelpj/plutus/tree/imp/bump-versions
Run nix build -f default.nix localPackages.plutus-tx-plugin
Log file:
log.txt
Hi,
there's a typo, isnt there?
Should be topology.json.
iohk-nix/cardano-lib/default.nix
Line 47 in b426fb9
Today I extracted the Byron Proxy (using git filter-branch) into its own repo here. It currently builds using stack (but the cabal build is still busted).
This new repo needs:
Once all that is done, I will raise a PR removing byron-proxy from ouroboros-network.
During the haskell.nix migration of plutus (IntersectMBO/plutus#1825), at least the following functions went MIA:
getPackgesstylishHaskellhaskellPackagesThe cardano-repo-tool has a featrue where it can update the git hashes in the cabal.project file, from the stack.yaml file.
It would be nice to have this as a CI job. Basically:
cardano-repo-tool update-cabal-project in the top level directory.cabal.project file changes,
As seen in https://hydra.iohk.io/build/975598 / https://hydra.iohk.io/build/975527, pretty-show fails to build:
Preprocessing library for pretty-show-1.9.5..
happy: /nix/store/42skizzfd23sb0fiz0f5dhi411cb5jbr-happy-1.19.11-exe-happy/share/x86_64-linux-ghc-8.6.5/happy-1.19.11-JKLIZ4vRYXC4477d9ZYQwf-happy/HappyTemplate-arrays-coerce: openFile: does not exist (No such file or directory)
builder for '/nix/store/2546dybwh3cdfpjf1rh10ml90280q796-pretty-show-1.9.5-lib-pretty-show.drv' failed with exit code 1
Local reproduction provides the following:
/nix/store/42skizzfd23sb0fiz0f5dhi411cb5jbr-happy-1.19.11-exe-happy/share/x86_64-linux-ghc-8.6.5/happy-1.19.11-JKLIZ4vRYXC4477d9ZYQwf-happy/HappyTemplate-arrays-coerce does not indeed exist/nix/store/42skizzfd23sb0fiz0f5dhi411cb5jbr-happy-1.19.11-exe-happy/share/x86_64-linux-ghc-8.6.5/happy-1.19.11/HappyTemplate-arrays-coerce however existsSo this hints at that happy was, perhaps, somehow overridden, where it should have been?
(since the GHC packages shoudn't get the disambiguating Cabal suffix? JKLIZ4vRYXC4477d9ZYQwf in this case)
EDIT: on a second thought, it seems that happy is irrepairably schizophrenically broken, as it appears to embode two incompatible worldviews (Weltanschauungs as our german colleagues might appreciate):
If we run the very useful command nix eval --json .#lib-srcs, we get:
[1 copied (127.4 MiB), 21.5 MiB DL]{"blst":"https://github.com/supranational/blst/archive/03b5124029979755c752eec45f3c29674b558446.tar.gz","secp256k1":"https://github.com/bitcoin-core/secp256k1/archive/5dcc6f8dbdb1850570919fc9942d22f728dbc0af.tar.gz","sodium":"https://github.com/input-output-hk/libsodium/archive/dbb48cce5429cb6585c9034f002568964f1ce567.tar.gz"}
in particular, we use commit 5dcc6. There was a recent release of version 0.3.1. We should update to that latest version, which is also what is currently used in nixpkgs.
Also, adding instructions on how to do such an update would be extremely helpful 🙏
Minor issue but, in cardano-node (but following the instructions from this repo)
$ nix-build -A scripts.shelley_selfnode.node -o shelley-selfnode
$ ./shelley-selfnode
Starting cardano-node run: /nix/store/xk3a37drb548hd1xinaayc7h8z6daba7-cardano-node-1.11.0-exe-cardano-node/bin/cardano-node run
--config state-node-shelley_selfnode/config.yaml
--database-path state-node-shelley_selfnode/db-shelley_selfnode
--socket-path state-node-shelley_selfnode/node.socket
--topology /nix/store/azj9m8id82ywn9q4qgjz9ba73vcf92p4-topology.yaml
--host-addr 127.0.0.1
--port 3001
--shelley-vrf-key /nix/store/288myg0mxb4bslaklqi5rzqvad6xj0zk-node-vrf.skey
--shelley-kes-key /nix/store/w6dq8h1cx9f2h88him85cbv653sv5fiy-node-kes.skey
--shelley-operational-certificate /nix/store/h0d7673y9pf3pas4wnj84jpjxgivfbg9-node.opcert
..or, once again, in a single line:
/nix/store/xk3a37drb548hd1xinaayc7h8z6daba7-cardano-node-1.11.0-exe-cardano-node/bin/cardano-node run --config state-node-shelley_selfnode/config.yaml --database-path state-node-shelley_selfnode/db-shelley_selfnode --socket-path state-node-shelley_selfnode/node.socket --topology /nix/store/azj9m8id82ywn9q4qgjz9ba73vcf92p4-topology.yaml --host-addr 127.0.0.1 --port 3001 --shelley-vrf-key /nix/store/288myg0mxb4bslaklqi5rzqvad6xj0zk-node-vrf.skey --shelley-kes-key /nix/store/w6dq8h1cx9f2h88him85cbv653sv5fiy-node-kes.skey --shelley-operational-certificate /nix/store/h0d7673y9pf3pas4wnj84jpjxgivfbg9-node.opcert
Wiping all data in state-node-shelley_selfnode
date: illegal option -- -
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]date command is different on macOS. Works after manually replacing date with gdate in the generated shelley-selfnode.
Hi!
I'd like to know if it's possible to completely dissociate from the default Hackage server and provide a custom server instance as source of snapshots and package data when using haskell.nix.
The check-hydra script fails sometimes, and succeeds when retried.
The message is:
warning: `--gc-roots-dir' not specified
restarting hydra-eval-jobs after job 'native.benchmarks.cardano-wallet-core.db.x86_64-linux' because heap size is at 1317535744 bytes
warning: `--gc-roots-dir' not specified
restarting hydra-eval-jobs after job 'native.benchmarks.cardano-wallet-http-bridge.restore.x86_64-linux' because heap size is at 1292369920 bytes
warning: `--gc-roots-dir' not specified
restarting hydra-eval-jobs after job 'native.cardano-wallet-http-bridge.x86_64-linux' because heap size is at 1401421824 bytes
warning: `--gc-roots-dir' not specified
error: getting status of '/nix/store/rar9fy1a12axkdjadksqbgj7b8sz9cf6-hackage-exprs-source': No such file or directory
ERROR: Failed to evaluate release.nix
It looks like something got GC'd while the check-hydra script was running.
Eval-time dependencies aren't GC rooted correctly, because nix assumes evals are fast and GCs don't happen often. But with the auto-gc, an eval can trigger a GC, and break itself.
The check-hydra script should ensure that eval-time dependencies are GC rooted. It should also not print confusing warning messages.
Some pretty weird cache misses in the current tip of https://github.com/input-output-hk/cardano-node/tree/serge/mainnet-ci (commit 8b6f615506f2de42c339f16aa1f5fe6fbcd0c808):
Caches filled on Hydra: https://hydra.iohk.io/build/1377499 (note the .drv: /nix/store/9ihjzh425p2s3wrk06767amclc9ab8an-vm-test-run-chairmans-cluster-test.drv)
The attrbute evaluates locally to the same .drv:
$ nix-instantiate -A nixosTests.chairmansCluster
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/9ihjzh425p2s3wrk06767amclc9ab8an-vm-test-run-chairmans-cluster-test.drv
$ nix-build -A nixosTests.chairmansCluster
these derivations will be built:
/nix/store/cqad1a89vy2lqsv963vj087clm8ax876-ghc-8.4.4.drv
/nix/store/x4qg7x969vb7lwiay7idcmll820vwlz9-default-Setup.drv
/nix/store/j2gxy6rw6rmfrrf8mb59wpvckllv7xpj-bytestring-0.10.8.2-lib-bytestring.drv
/nix/store/md70yw3ghhgla0ihkga015fbhcgy9ylf-containers-0.5.11.0-lib-containers.drv
...
don't know how to build these paths:
/nix/store/0kah289565r372gnyw5r426h418hxjiw-yaml-0.11.0.0-lib-yaml
/nix/store/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz-ghc-8.4.4
/nix/store/9m173vkvd6r8i706nykyvxfhlqjlhzb8-ghc-8.4.4-doc
/nix/store/bhnwnvrv94q1lm072v68wswyc1gkw438-nonempty-containers-0.1.1.0-lib-nonempty-containers
/nix/store/dihzx42kyjh51hgbv582775qnij2mgi2-conduit-1.3.1.1-lib-conduit
/nix/store/iywqv861qhkrii08ypk3ysgbfaqzrn4p-libyaml-0.1.1.0-lib-libyaml
/nix/store/vxhvbfjqzpf1gzi2vikczzsc8bl95pd1-mono-traversable-1.0.11.0-lib-mono-traversable
..and note the failure to build the paths.
$ curl https://hydra.iohk.io/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
$ nix show-config | grep 'substituters '
extra-substituters =
substituters = https://cache.nixos.org/ https://hydra.iohk.io/ https://mantis-hydra.aws.iohkdev.io/ https://cache.nixos.org/
trusted-substituters = https://cache.nixos.org/ https://hydra.iohk.io/ https://mantis-hydra.aws.iohkdev.io/
The regen.nix derivation that we use in Plutus and Cardano could live here, since it just creates a script.
(We could probably also have it be an actual derivation, if we took an appropriately filtered source directory as an argument)
AFAICT it's using https://github.com/input-output-hk/libsodium/tree/66f017f16633f2060db25e17c170c2afa0f2a8a1 which is ancient.
Hi, this is a post-to-the-right-project move of input-output-hk/stack2nix#170 -- I had a stack project that used proto-lens-protoc, and there are both haskell packages and top-level nix packages for this.
I frankly don't care if the solution is hacky, but what's the simplest (to do now, to keep working later) method of getting my proto's sorted for my package? The custom setup used /just/ generates haskell from the .proto files. So any invocation of the protoc compiler that results in the files being in a place that's accessible (and not in my src/ directory, where version control's an issue) for compilation is fine by me.
Create a generator for the log used by new node. This will generate log configs for byron-proxy, cardano-node, cardano-explorer, etc...
If you build cardano-node with nix-build with the default config file then the output is cluttered with cardano.node.BlockFetchDecision. Can these be disabled? Karl has said we don't need them anyway (and they're hidden by default with Cabal).
[nixos:cardano.node.BlockFetchDecision:Info:46] [2020-04-22 14:40:19.88 UTC] [String "ConnectionId {localAddress = 192.168.0.19:43215, remoteAddress = 3.125.193.96:3001}",String "FetchDecision declined",String "FetchDeclinePeerBusy 582000 501000 1002000"]
[nixos:cardano.node.ChainDB:Info:36] [2020-04-22 14:40:19.88 UTC] Valid candidate (Point 429304, 14d0dcfe1999dcca)
[nixos:cardano.node.ChainDB:Notice:36] [2020-04-22 14:40:19.88 UTC] Chain extended, new tip: (Point 429304, 14d0dcfe1999dcca)
[nixos:cardano.node.BlockFetchDecision:Info:46] [2020-04-22 14:40:19.88 UTC] [String "ConnectionId {localAddress = 192.168.0.19:43215, remoteAddress = 3.125.193.96:3001}",String "FetchDecision declined",String "FetchDeclinePeerBusy 578000 501000 1002000"]
[nixos:cardano.node.ChainDB:Info:36] [2020-04-22 14:40:19.88 UTC] Valid candidate (Point 429305, 7329d30a4fafd741)
[nixos:cardano.node.ChainDB:Notice:36] [2020-04-22 14:40:19.88 UTC] Chain extended, new tip: (Point 429305, 7329d30a4fafd741)
[nixos:cardano.node.BlockFetchDecision:Info:46] [2020-04-22 14:40:19.88 UTC] [String "ConnectionId {localAddress = 192.168.0.19:43215, remoteAddress = 3.125.193.96:3001}",String "FetchDecision declined",String "FetchDeclinePeerBusy 576000 501000 1002000"]
[nixos:cardano.node.ChainDB:Info:36] [2020-04-22 14:40:19.88 UTC] Valid candidate (Point 429306, 25b073cf456c3563)
[nixos:cardano.node.ChainDB:Notice:36] [2020-04-22 14:40:19.88 UTC] Chain extended, new tip: (Point 429306, 25b073cf456c3563)
[nixos:cardano.node.BlockFetchDecision:Info:46] [2020-04-22 14:40:19.88 UTC] [String "ConnectionId {localAddress = 192.168.0.19:43215, remoteAddress = 3.125.193.96:3001}",String "FetchDecision declined",String "FetchDeclinePeerBusy 574000 501000 1002000"]
[nixos:cardano.node.ChainDB:Info:36] [2020-04-22 14:40:19.88 UTC] Valid candidate (Point 429307, 206eab5cec085f16)
[nixos:cardano.node.ChainDB:Notice:36] [2020-04-22 14:40:19.88 UTC] Chain extended, new tip: (Point 429307, 206eab5cec085f16)
[nixos:cardano.node.ChainDB:Info:36] [2020-04-22 14:40:19.88 UTC] Valid candidate (Point 429308, e1f55d834c4e70b3)
[nixos:cardano.node.ChainDB:Notice:36] [2020-04-22 14:40:19.88 UTC] Chain extended, new tip: (Point 429308, e1f55d834c4e70b3)
[nixos:cardano.node.ChainDB:Info:36] [2020-04-22 14:40:19.88 UTC] Valid candidate (Point 429309, 8cb863deade65cf5)
[nixos:cardano.node.ChainDB:Notice:36] [2020-04-22 14:40:19.88 UTC] Chain extended, new tip: (Point 429309, 8cb863deade65cf5)
[nixos:cardano.node.BlockFetchDecision:Info:46] [2020-04-22 14:40:19.88 UTC] [String "ConnectionId {localAddress = 192.168.0.19:43215, remoteAddress = 3.125.193.96:3001}",String "FetchDecision declined",String "FetchDeclinePeerBusy 572000 501000 1002000"]
[nixos:cardano.node.BlockFetchDecision:Info:46] [2020-04-22 14:40:19.88 UTC] [String "ConnectionId {localAddress = 192.168.0.19:43215, remoteAddress = 3.125.193.96:3001}",String "FetchDecision declined",String "FetchDeclinePeerBusy 570000 501000 1002000"]
[nixos:cardano.node.ChainDB:Info:36] [2020-04-22 14:40:19.88 UTC] Valid candidate (Point 429310, 4ec336951aca09a7)
[nixos:cardano.node.ChainDB:Notice:36] [2020-04-22 14:40:19.88 UTC] Chain extended, new tip: (Point 429310, 4ec336951aca09a7)
[nixos:cardano.node.ChainDB:Info:36] [2020-04-22 14:40:19.88 UTC] Valid candidate (Point 429311, ce4150a73278bb2e)
[nixos:cardano.node.ChainDB:Notice:36] [2020-04-22 14:40:19.88 UTC] Chain extended, new tip: (Point 429311, ce4150a73278bb2e)
Updating this file is one of the most painful things I have to do in getting my changes through CI.
Would be really nice to improve this.
Not sure why not, possibly my fault. @disassembler
Id like to the run a node with the latest plutus builtins.
I dont intimately understand the dependency structure her but from what i can tell, cardano-api is fine with relaxing the plutus constraints
waalge/cardano-api@5167ab6
but no such luck with the cardano node
IntersectMBO/cardano-node@abe0445
Its been a long fight to this point. I had a similar issue that was resolved with a pointer that something was up with the iohknix dep. https://github.com/input-output-hk/cardano-haskell-packages/issues/334
And there is an ominous note in the node flake
https://github.com/waalge/cardano-node/blob/af43012b81f0c0370dc136298d641c7228bbb58f/flake.nix#L105
suggesting this is delicate.
would really appreciate some pointers as to nix cannot see blst here. The nix code is involved to say the least
if you run justStaticExecutables on something that the haddock-hydra.nix overlay has touched, then the build will fail:
Installing library in /nix/store/1my1336f7g8fz4yvkwq0pmzrbh62lfj2-meadow-0.1.0.0/lib/ghc-8.6.4/x86_64-linux-ghc-8.6.4/meadow-0.1.0.0-IpbvTtm5O0h7URJLPn0lq
Installing executable meadow-exe in /nix/store/1my1336f7g8fz4yvkwq0pmzrbh62lfj2-meadow-0.1.0.0/bin
Warning: The directory
/nix/store/1my1336f7g8fz4yvkwq0pmzrbh62lfj2-meadow-0.1.0.0/bin is not in the
system search path.
Registering library for meadow-0.1.0.0..
mkdir: cannot create directory ‘/nix-support’: Permission denied
thats due to justStaticExecutables removing out from outputs and now $out is unset
https://github.com/input-output-hk/iohk-nix/blob/master/overlays/haddock-hydra.nix
will need to be modified to somehow detect that justStaticExecutables has been ran and stop trying to create docs
one option is to just test if $doc is set
a temporarily work-around is to either:
meadow-static = static (pkgs.haskell.lib.overrideCabal haskellPackages.meadow (drv: { postInstall = ""; }));
or to exclude the package from getting haddock at the filterOverrides = { haddock = localButNot area
When starting a new repository, or adding CI support for an existing one, there seem to be no reference template repository one can use to make sure the latest CI scripts and best practices are being used.
https://hydra.iohk.io/build/930934/nixlog/1
async-2.2.2: dependency "hashable-1.2.7.0-A3DnuZniYSIDZYmh6JlprE" doesn't exist (ignoring)
async-2.2.2: dependency "stm-2.5.0.0-2oKC33YEmw5FLGhjz9knXu" doesn't exist (ignoring)
ghc-pkg: Couldn't open database /nix/store/b3i610hn7dz0aqykrc63ig8497msrbsn-streaming-commons-0.2.1.1-lib-streaming-commons-config/package.conf.d for modification: /nix/store/b3i610hn7dz0aqykrc63ig8497msrbsn-streaming-commons-0.2.1.1-lib-streaming-commons-config/package.conf.d/package.cache: openBinaryFile: resource exhausted (Too many open files in system)
builder for '/nix/store/v0qnaai3shba5riysl06ycyx2in07ryc-streaming-commons-0.2.1.1-lib-streaming-commons-config.drv' failed with exit code 1
We can have nix code that builds the rust repos
without the rust repos having proper build infrastructure
by adding a rust overlay into iohk-nix.
We should have release.nix build any pkgs in this overlay.
At the moment if you want to use a newer/different nixpkgs or haskell.nix you have to either:
iohk-nix, wait for that to be merged, update the pin in your repo.
NIX_PATH override.
nix invocations, not .nix files.I propose we add a third way: let them be passed as arguments to default.nix, and only load them from the pin files if they aren't provided that way.
I think this is complementary to the NIX_PATH method, depending what you want. As a bonus, we could get rid of the horrible application thing in favour of applications just passing their own specific pin.
We need to make sure the latest stackage.nix and hackage.nix are available on the iohk-nix master branch.
I suggest that we automatically update this repo daily after Haskell.nix has done its nightly update. Automatically updating the Haskell.nix pin could case trouble (especially because there are no test cases here), so let's update that regularly, but not automatically.
hackageSourceJSON and stackageSourceJSON arguments in iohk-nix/haskell.nixhackage-src.json and stackage-src.json into the pins directory.After ee5f5b0#diff-206b9ce276ab5971a2489d75eb1b12999d4bf3843b7988cbe8d687cfde61dea0R16 you might have experienced breakage like we did.
error: value is a function while a set was expected
at «none»:0: (source not available)
To fix do the following...
In your flake.nix inputs use iohk-nix repo as a flake if you didn't already
- iohk-nix = { url = "github:input-output-hk/iohk-nix"; flake = false; };
+ iohk-nix.url = "github:input-output-hk/iohk-nix";And change the way you add the crypto overlay
- (import "${iohk-nix}/overlays/crypto")
+ inputs.iohk-nix.overlays.cryptoIt would be nice not to have our own fork of nixpkgs.
As far as I know, there are a few things preventing us from using upstream:
haskell.nix eventually so we don't depend on the nixpkgs versions any more, which would solve this.haskell.nixAm I missing anything?
MSYS2 now suggests using UCRT64 instead of MINGW64 (ref: https://www.msys2.org/news/#2022-10-29-changing-the-default-environment-from-mingw64-to-ucrt64). Could we build libsecp256k1, libsodium and libblst for that one instead?
Hi,
I've encountered an issue when running nix-shell on my Mac (and only there not on my other nixos box):
error: undefined variable 'file' at /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/commit-id.nix:33:49
looking at that function (https://github.com/input-output-hk/iohk-nix/blob/master/commit-id.nix#L31-L36), I'm not sure where file is supposed to come from. Could this be a bug?
Thanks,
Torsten
full trace below:
$ nix-shell --show-trace
trace: Using latest index state for hasktorch!
trace: Using index-state: 2020-08-13T00:00:00Z for hasktorch
trace: Using latest index state for hasktorch!
trace: Using index-state: 2020-08-13T00:00:00Z for hasktorch
trace: Shell for codegen examples experimental hasktorch libtorch-ffi libtorch-ffi-helper pipes-text
trace: Shell for codegen examples experimental hasktorch libtorch-ffi libtorch-ffi-helper pipes-text
error: while evaluating the attribute 'NIX_GHC_LIBDIR' of the derivation 'hasktorch-dev-shell' at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/shell-for.nix:99:5:
while evaluating the attribute 'buildCommand' of the derivation 'ghc-shell-for-packages-ghc-8.8.4-env' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/pkgs/build-support/trivial-builders.nix:7:14:
while evaluating the attribute 'buildCommand' of the derivation 'hasktorch-dev-shell-config' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/pkgs/build-support/trivial-builders.nix:7:14:
while evaluating anonymous function at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/make-config-files.nix:50:66, called from /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/make-config-files.nix:50:5:
while evaluating the attribute 'depends' at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/shell-for.nix:64:5:
while evaluating anonymous function at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/shell-for.nix:17:16, called from /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/shell-for.nix:28:24:
while evaluating 'foldComponents' at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/lib/default.nix:43:31, called from /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/shell-for.nix:17:20:
while evaluating 'libComp' at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/lib/default.nix:47:17, called from /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/lib/default.nix:53:8:
while evaluating the attribute 'library' at undefined position:
while evaluating 'buildComp' at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/hspkg-builder.nix:93:28, called from undefined position:
while evaluating 'makeOverridable' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/customisation.nix:67:24, called from /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/hspkg-builder.nix:93:39:
while evaluating 'self' at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/comp-builder.nix:4:1, called from /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/customisation.nix:69:16:
while evaluating 'filterAttrs' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/attrsets.nix:124:23, called from /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/comp-builder.nix:373:6:
while evaluating anonymous function at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/attrsets.nix:125:29, called from /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/attrsets.nix:125:18:
while evaluating anonymous function at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/lib/default.nix:105:39, called from /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/attrsets.nix:125:62:
while evaluating the attribute 'postInstall' at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/builder/comp-builder.nix:376:12:
while evaluating the attribute 'postInstall' at undefined position:
while evaluating anonymous function at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:84:45, called from undefined position:
while evaluating the attribute 'value' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:379:9:
while evaluating the option `packages.codegen.components.library.postInstall':
while evaluating the attribute 'mergedValue' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:411:5:
while evaluating the attribute 'values' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:405:9:
while evaluating anonymous function at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:401:19, called from /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:401:14:
while evaluating the attribute 'value._type' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:503:73:
while evaluating the attribute 'value.content' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:564:14:
while evaluating the attribute 'default' at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/modules/plan.nix:213:7:
while evaluating 'getDefaultOrNull' at /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/modules/plan.nix:24:27, called from /nix/store/5qscx67hmz4885fmk79d0hfqq13j136v-haskell.nix-src/modules/plan.nix:213:17:
while evaluating the attribute 'postInstall' at undefined position:
while evaluating anonymous function at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:84:45, called from undefined position:
while evaluating the attribute 'value' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:379:9:
while evaluating the option `packages.codegen.postInstall':
while evaluating the attribute 'mergedValue' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:411:5:
while evaluating the attribute 'values' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:405:9:
while evaluating the attribute 'values' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:505:7:
while evaluating anonymous function at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:391:28, called from /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:391:17:
while evaluating 'dischargeProperties' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:464:25, called from /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:392:62:
while evaluating the attribute 'value' at /nix/store/aggz1knma32dpgsxzdmw52bw43ch8bzh-nixpkgs-2003-src/lib/modules.nix:277:44:
while evaluating 'commitIdFromGitRepoOrZero' at /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/default.nix:69:33, called from /Users/tscholak/Projects/thirdParty/hasktorch/nix/haskell.nix:152:10:
while evaluating anonymous function at /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/commit-id.nix:57:3, called from /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/default.nix:72:33:
while evaluating 'readCommitFromFile' at /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/commit-id.nix:11:30, called from /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/commit-id.nix:60:12:
while evaluating 'readCommitFromRefFile' at /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/commit-id.nix:22:33, called from /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/commit-id.nix:15:13:
while evaluating 'readCommitFromFile' at /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/commit-id.nix:11:30, called from /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/commit-id.nix:29:13:
while evaluating 'readCommitFromPackedRefsFile' at /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/commit-id.nix:31:40, called from /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/commit-id.nix:19:15:
undefined variable 'file' at /nix/store/jh9h7m9cppy969b87yhfw4kzm9ibrbc9-iohk-nix-src/commit-id.nix:33:49
Just found this - it says it's an improved version of the one in nixpkgs, we should upstream it if so.
The stack-rebuild job is the longest in CI -- and by far.
As @rvl suggested in, we should look into replicating the setup in https://github.com/input-output-hk/iohk-nix/tree/master/skeleton/.buildkite, to enable caching of Stack builds -- and this should make the build very fast.
We only really need the stack-rebuild job because of coveralls, which we currently do [1,2] via http://hackage.haskell.org/package/stack-hpc-coveralls -- which can't work well in Nix context.
It is true that Stack users are dependent on maintaining the Stack-buildability, but that maintenance can be done asynchronously -- adding fixes after the fact. And one would think that such issues (that is Stack failing, with Nix builds succeeding) should happen quite rarely, anyway -- after all, Stack is used as source for Nix definitions, so it's implicitly always used, in some sense.
So if plan A fails to satisfy our performance goals, we should look into:
stack rebuild job non-mandatory.--
Many of the gotchas in gotcha 2 are signalled by "attribute missing" errors. We could significantly improve the UX there (and hopefully not need separate documentation!) by throwing a custom error in those places.
For example, consider the stackage resolver. At the moment if you try and use a resolver that isn't in the current stackage.nix, you get an unhelpful error, but we can easily change that:
let
# The Stackage release referenced in the stack config
- pkg-def = stackage.${stack-pkgs.resolver};
+ pkg-def = stackage.${stack-pkgs.resolver} or throw
+ "This version of stackage.nix does not know about the Stackage resolver ${stack-pkgs.resolver}, you may need to update haskell.nix to one that includes a newer stackage.nix.";
# The compiler referenced in the stack config
compiler = (stack-pkgs.extras hackage).compiler or (pkg-def hackage).compiler;
patchesModule = ghcHackagePatches.${compiler.nix-name} or {};(I'll open a PR for this one if people like the idea.)
I think there's a pattern here: any time we access an attribute that we don't know statically will be present, we should do or throw <msg>. Otherwise we're doing the equivalent of fromJust $ lookup name attrset.
At least I think we could do this for some of the common cases in the gotchas. And if a user reports an "attribute missing" error and it's not them screwing up, then we should probably add a custom error.
I added a proof-of-concept for this in: IntersectMBO/cardano-node@5c06f2b
Would be nice to generalize this so it could be used in all the projects.
Todo:
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.