Coder Social home page Coder Social logo

linuxkit-nix's Introduction

You should look at the darwin.builder, which is now part of nixpkgs.

linuxkit-nix was started in 2017 when QEMU did not support macOS' Hypervisor.framework API. This meant that QEMU had to use full emulation with no hardware acceleration. Not ideal for building large software.

QEMU also had some issues on macOS with userspace networking.

At the time, LinuxKit was the easiest way to spin up a VM for builds, because it spun up HyperKit for hardware accelerated virtualisation and VPNKit for userspace networking - both used in Docker for Mac.

Theoretically the underlying technology was stable but it was bit tricky to get everything working well together. There were bootstrapping issues. For example, we had to be careful when referencing linux-x86_64 packages because we were on darwin-x86_64 and it could only fetch from Hydra - it couldn't even build a custom shell script for the Linux VM until we got that initial VM running.

This project also had issues with daemons, permissions and race conditions.

In 2018, QEMU got experimental support for Hypervisor.framework and that got promoted to stable in 2019. QEMU is now fast and since nixpkgs has great support for building and running QEMU virtual machines, there's little need for this project.


LinuxKit Nix - Linux on Mac Nix builder

hydra jobset

LinuxKit Nix makes it easy to build Linux binaries from a macOS machine using Nix. It's installing a VM using the native virtualization (Hypervisor.Framework) so it's quite liteweight compared to installing VirtualBox. The project also comes with an installation script that configures Nix to use the VM as a remote builder automatically.

Requirements

This project depends on Nix and a nixpkgs channel >= 18.03.

Installation

Fetch it from the NixOS binary cache:

nix-env -i /nix/store/jgq3savsyyrpsxvjlrz41nx09z7r0lch-linuxkit-builder
nix-linuxkit-configure

It'll write to:

  • ~/.cache/nix-linuxkit-builder/, in particular ~/.cache/nix-linuxkit-builder/nix-state/console-ring is interesting
  • ~root/.ssh/ for the SSH config
  • /etc/nix/machines
  • ~/Library/LaunchAgents/org.nix-community.linuxkit-builder.plist

Once installed the daemon should automatically start and stay running.

Debugging

To see if the daemon is running execute the following command and look at the first column. If it has a number (PID) it's running, if it's - then it's stopped:

launchctl list | grep linuxkit

You can force start it with:

launchctl start org.nix-community.linuxkit-builder

You can force stop it with:

launchctl stop org.nix-community.linuxkit-builder

If after you stop it you may want to check for processes, like:

pgrep vpnkit
pgrep linuxkit
pgrep hyperkit

If something goes wrong and it didn't stop properly, you can try:

pkill -F ~/.cache/nix-linuxkit-builder/nix-state/hyperkit.pid hyperkit

Troubleshooting

cannot build on 'ssh://nix-linuxkit': cannot connect to 'nix-linuxkit' ...

When runninng nix-linuxkit-configure, an SSH config is created at /var/root/.ssh/nix-linuxkit-ssh-config. Copy the contents of that SSH config into your regular SSH config located at ~/.ssh/config.

error: 'x86_64-linux' is require to build ...

Check the /etc/nix/nix.conf file for a builders option. It should either be set to @/etc/nix/machines or not set at all for LinuxKit Nix to work properly.

Another solution is to set export NIX_REMOTE_SYSTEMS=/etc/nix/machines before running the nix or nixops command.

cannot build on 'ssh://nix-linuxkit': cannot connect ...: Operation timed out

Something is wrong with LinuxKit. See the debugging section to try things out.

Leave an issue at https://github.com/nix-community/linuxkit-nix/issues

Uninstalling

# Remove configuration
rm -rf ~/.cache/nix-linuxkit-builder/

# Remove build machine
# (edit manually if you have other configuration here)
sudo rm -f /etc/nix/machines

# Remove LaunchAgent
launchctl stop org.nix-community.linuxkit-builder
rm -f ~/Library/LaunchAgents/org.nix-community.linuxkit-builder.plist

# Remove SSH config
# (edit manually if you have other configuration here)
sudo rm -rf /var/root/.ssh

# Uninstall Nix package
nix-env -e linuxkit-builder

linuxkit-nix's People

Contributors

andir avatar copumpkin avatar evanrelf avatar graham-at-target avatar grahamc avatar imalsogreg avatar nicknovitski avatar puffnfresh avatar zimbatm avatar zowoq avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

linuxkit-nix's Issues

Can't start the service after stopping it

I installed linuxkit-nix and it worked fine to run a few builds. Then I wanted to stop the service while I was not using it, as it took up 7gb of RAM, and start it back later. Unfortunately, after starting it again, it crashes after a minute or two.

$ launchctl stop org.nix-community.linuxkit-builder
$ launchctl list | grep linuxkit                                                      
-	0	org.nix-community.linuxkit-builder
$ launchctl start org.nix-community.linuxkit-builder
$ echo $?
0
$ date
Sun Jun 30 12:28:56 PDT 2019
$ launchctl list | grep linuxkit
17022	0	org.nix-community.linuxkit-builder
... wait a little while
$ date
Sun Jun 30 12:29:56 PDT 2019
$ launchctl list | grep linuxkit
-	0	org.nix-community.linuxkit-builder
Click to show nix-info
$ nix-shell -p nix-info --run "nix-info -m"         
 - system: `"x86_64-darwin"`
 - host os: `Darwin 18.5.0, macOS 10.14.4`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.2.2`
 - channels(lev): `""`
 - channels(root): `"nixpkgs-19.09pre184256.5b7e3718ee3"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

I can't find any relevant logs anywhere. How do I check why it is crashing?

Fails to create the launch agent

From the installer as followed in the README:

cp: cannot create regular file '/Users/zimbatm/Library/LaunchAgents/org.nix-community.linuxkit-builder.plist': No such file or directory

It's assuming that Library/LaunchAgents exists

catalina issue

I installed from the latest README instructions, when running the service I get the following

vpnkit: [ERROR] Vmnet.Server.listen: read EOF so closing connection
vpnkit: [INFO] Vmnet.Server.listen returning Ok()
FATA[0083] Cannot run hyperkit: exit status 2
linuxkit's ending state: ExitStatus(ExitStatus(256))
Bye!

I did this by running /nix/store/z2qfwj97dh2yc43x42pbrp7i1qmhbrp4-shellchecked-nix-linuxkit-builder directly which is what the plist entry was doing.

I assume this is a Catalina problem, since most things are recently :)

Digital Ocean: Host key verification failed.

I'm having ssh-related issues with a nixops deployment to digital ocean.

The initial ssh into the machine goes fine:

machine> waiting for SSH........................                                                                                                                
machine> running nixos-infectWarning: Permanently added '165.22.236.211' (ECDSA) to the list of known hosts.   

The installer seems to be running ok until we get to the reboot:

machine> waiting for the machine to finish rebooting....[down]..................................................................................................
[up]

Then, the address is added to known_hosts again?

Warning: Permanently added '165.22.236.211' (ED25519) to the list of known hosts.

Build info is output:

these derivations will be built:
  /nix/store/018fxn4x26vgq2iy56nvdbdg0pk7zik2-vconsole.conf.drv
  ....

And our third? ssh connection fails.

cannot build on 'ssh://[email protected]': cannot connect to '[email protected]': Host key verification failed.

(Nixops then errors with error: a 'x86_64-linux' with features {} is required to build '/nix/store/9j73g9irkbd904xs6m1kjhzjkcinziz6-append-initrd-secrets.drv', but I am a 'x86_64-darwin' with features {benchmark, big-parallel, nixos-test}).

Any idea what could be causing this issue with the ssh connection?

I've posted here instead of https://github.com/NixOS/nixops, as I suspect I need to somehow configure digital ocean and linuxkit to trust each others' keys, but I'm not clear on what exactly needs to be done.

One thing I did try was adding the client.pub key to digital ocean, but the result was exactly the same.

Let me know if you need more info for debugging.

Fails when /etc/nix missing

Had to sudo mkdir /etc/nix before nix-linuxkit-configure would work.

Now it completed but it won't work, the *kit processes go away after a short time.

how to run garbage collection on linuxkit-builder?

I've been using linuxkit for a while and it has been working great, however I've noticed that the following file /Users/myusername/.cache/nix-linuxkit-builder/nix-disk has grown to over 80 GB. Running nix-collect-garbage doesn't reduce the size of this file. How could I reduces it's size? Should I just nuke the whole thing? Thanks!

rust binaries are `-`-separated?

$ /nix/store/9ri2i7r780aghp3bh6nr8sqhi9qqigqz-shellchecked-nix-linuxkit-builder
# ...
/nix/store/9ri2i7r780aghp3bh6nr8sqhi9qqigqz-shellchecked-nix-linuxkit-builder: line 39: /nix/store/bmz6kwcwb1gqawppvllng7l6r6jsf9n3-rust_nix-linuxkit-runner-0.1.0/bin/nix_linuxkit_runner: No such file or directory
$ ls /nix/store/bmz6kwcwb1gqawppvllng7l6r6jsf9n3-rust_nix-linuxkit-runner-0.1.0/bin
nix-linuxkit-runner   nix_linuxkit_runner.d

The commit message of f4b7b25 suggests this was a necessary due to nixpkgs changes. I can believe that, but in that case I don't understand why I'm getting --separated binary.

I'm on darwin, using the installation and setup instructions in the readme.

/tmp is unwritable

Iโ€™m having some trouble building halide, which tries to mkdtemp with a template of /tmp/XXXXXX . In stage-1.sh, it appears that /tmp is created as unwritable to anyone but root. Is there a workaround, or would it be reasonable to make /tmp world-readable? Thanks.

Find a better name

@grahamc mentioned that developers tend to assume that it's related to Docker due to the name.

Looking for a brain dump of ideas, one comment per name:

linuxkit-builder exits soon after starting

Hi there, trying out linuxkit-nix for the first time, and it all seems well except that the linuxkit-builder process exits shortly after starting. This is on MacOS 10.15.7. Not really sure how to debug, can't find any relevant logs. Would be grateful for any pointers, thanks!

/etc/nix/machines: No such file or directory when trying to install linuxkit-nix

This is because on macOS Catalina and above, the root partition is no longer writable.

/Users/siraben/.cache/nix-linuxkit-builder/gcroot
Setting up /var/root//.ssh/nix-linuxkit-ssh-config...
Password:
Adding the SSH configuration (/var/root//.ssh/nix-linuxkit-ssh-config) to /var/root//.ssh/config...
Adding the Nix Machines configuration (/etc/nix/machines) to /etc/nix/machines...
tee: /etc/nix/machines: No such file or directory

ssh://nix-linuxkit x86_64-linux /Users/siraben/.cache/nix-linuxkit-builder/keys/client 1 1 kvm,big-parallel

Doesn't work on my system

How can I debug the following?

$ nix-store -r /nix/store/1f5zgx8qykz2fxzhqphmsfp6cvpnfc94-linuxkit-builder
     
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/1f5zgx8qykz2fxzhqphmsfp6cvpnfc94-linuxkit-builder
$ nix-env -i /nix/store/1f5zgx8qykz2fxzhqphmsfp6cvpnfc94-linuxkit-builder
replacing old 'linuxkit-builder'
installing 'linuxkit-builder'
$ nix-linuxkit-configure
     
$HOME/.cache/nix-linuxkit-builder/gcroot
Setting up /var/root//.ssh/nix-linuxkit-ssh-config...
Password:
Ok, try it out!

    nix-build $HOME/.cache/nix-linuxkit-builder/example.nix

If this doesn't work right away, maybe wait a 10+ seconds and try again.
$ nix-build $HOME/.cache/nix-linuxkit-builder/example.nix
these derivations will be built:
  /nix/store/b2gyqwbg8nx1qlhdn2v3gj0872j7j6ar-hello-1535529157.drv
error: a 'x86_64-linux' is required to build '/nix/store/b2gyqwbg8nx1qlhdn2v3gj0872j7j6ar-hello-1535529157.drv', but I am a 'x86_64-darwin'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.