direktiv / vorteil Goto Github PK

View Code? Open in Web Editor NEW

174.0 14.0 13.0 1.73 MB

turn your applications and containers into micro virtual machines

Home Page: https://direktiv.io

License: Apache License 2.0

Go 100.00%

linux micro-virtual-machines containers virtual-machine cloud

vorteil's Introduction

vorteil.io

turn your applications and containers into micro virtual machines

Vorteil is an operating system for running cloud applications on micro virtual machines. It takes only the files you need and runs them on a custom Linux kernel without any unnecessary background services: there's no ssh, no shell, and no login; just a toml file that Vorteil's init process (vinitd) uses to configure the system and launch your apps.

The tools that build Vorteil images have been optimized for speed, and can take your project from loose files to a running virtual machine in as little as one second.

Installation

The Vorteil binary runs anywhere, just download the archive for your system, extract it, and put the executable on your PATH.

Linux

wget https://github.com/vorteil/vorteil/releases/latest/download/vorteil_linux-x86.tar.gz
tar -xzf vorteil_linux-x86.tar.gz
./vorteil version

Windows

wget https://github.com/vorteil/vorteil/releases/latest/download/vorteil_windows-x86.zip -UseBasicParsing -OutFile .\vorteil_windows-x86.zip
Expand-Archive -Path .\vorteil_windows-x86.zip -DestinationPath .\
.\vorteil.exe version

Mac

curl -LJO https://github.com/vorteil/vorteil/releases/latest/download/vorteil_darwin-x86.dmg
hdiutil attach vorteil_darwin-x86.dmg
cp /Volumes/Vorteil\ CLI/vorteil ./vorteil
hdiutil detach /Volumes/Vorteil\ CLI
./vorteil version

Dependencies

To use the vorteil run command you'll need VirtualBox, QEMU, firecracker or Hyper-V installed on your system and reachable on the PATH.

If you're using Windows, it's recommended that you enable developer mode as well, so that the tools can use Unix-style symlinks.

Using Vorteil

Hello World

vorteil run https://apps.vorteil.io/file/vorteil/helloworld

This command downloads our Hello World package from apps.vorteil.io and runs it using one of your local hypervisors.

Our Hello World app is a simple web server that should be reachable via NAT on port 8888 (http://localhost:8888/). If that port was unavailable you should notice a yellow warning message in the logs telling you which port it bound instead.

Convert from a Container

An easy way to build Vorteil micro virtual machines is to build them from container images. Vorteil can convert all OCI compliant images from remote repositories or local docker and containerd runtimes. The following command would convert the hello-world application from docker hub.

# converts hello-world from docker hub into /tmp/hello
vorteil projects convert-container hello-world /tmp/hello

# converts hello-world from local docker into /tmp/hellolocal
vorteil projects convert-container local.docker/hello-world /tmp/hellolocal

# run it
vorteil run /tmp/hellolocal

Deploying an Image

Vorteil can deploy the images to different cloud providers. Please read the documentation for examples.

Modify an Existing Package

wget -O cockroachdb.vorteil https://apps.vorteil.io/file/vorteil/cockroachdb
vorteil unpack cockroachdb.vorteil cockroachdb
ls cockroachdb

These commands download a Vorteil package and use the vorteil unpack command to extract its contents into a structured Vorteil project directory. Think of this directory like it will become the root directory on your app's virtual machine. You can modify the app's file-system contents by changing the files in this directory.

There are two types of special files in the project directory: .vorteilproject and files that end with the .vcfg file extension. These are used by Vorteil to understand how to build the virtual machine and what it should do when it's running.

Make whatever changes you need to make, then repackage them.

vorteil pack cockroachdb -o cockroachdb-modified.vorteil

This vorteil pack command packs up your project directory into a package file again.

vorteil run cockroachdb-modified.vorteil

Try out your modified package by using the vorteil run command on it.

Building From Source

These tools are 100% written in Go, which means compiling them is the same as compiling most simple Go programs.

go build github.com/vorteil/vorteil/cmd/vorteil

Code of Conduct

We have adopted the Contributor Covenant code of conduct.

Contributing

Any feedback and contributions are welcome. Read our contributing guidelines for details.

License

Distributed under the Apache 2.0 License. See LICENSE for more information.

vorteil's People

Contributors

Stargazers

Watchers

Forkers

jalfvort beesitech elserhumano cutff 123lab askmetoo micwarr jawar1mx dawnadvent jensg-st wesley-yang nuclearpidgeon kisun-bit

vorteil's Issues

calling cli with a random string crashes the app

If I call

vorteil randmostring

the app crashes with:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1508f53]

goroutine 1 [running]:
main.main()
	/home/jensg/go/src/github.com/vorteil/vorteil/cmd/vorteil/main.go:31 +0xb3

add vcfg flags / logic to provision command

The provision command should support vcfg flags in the same way that build and run do.

if running qemu is not permitted it crashes


Error: unable to start QEMU as socket wasn't created in time

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x225154a]

goroutine 114 [running]:

github.com/vorteil/vorteil/pkg/virtualizers/qemu.(*Virtualizer).Start.func1(0xc000724000, 0xc001fc65e0)

	/home/travis/gopath/src/github.com/vorteil/vorteil/pkg/virtualizers/qemu/virtualizer_notwindows.go:75 +0x58a

created by github.com/vorteil/vorteil/pkg/virtualizers/qemu.(*Virtualizer).Start

	/home/travis/gopath/src/github.com/vorteil/vorteil/pkg/virtualizers/qemu/virtualizer_notwindows.go:38 +0x1d9

Makefile:84: recipe for target 'test' failed```

vorteil images tree ./helloworld.vmdk --vpartition panics

Problem description

The following command as the title explains panics.

Expected behaviour

List a tree of the vpartition on a disk

Actual behaviour

.\vorteil.exe images tree ..\..\trentis\vorteil\helloworld.vmdk --vpartition
/
panic: runtime error: index out of range [-1]

goroutine 1 [running]:
main.glob..func16(0x2da6780, 0xc00009a3c0, 0x1, 0x2)
        C:/Users/trent/Documents/github.com/vorteil/vorteil/cmd/vorteil/images.go:1186 +0x822
github.com/spf13/cobra.(*Command).execute(0x2da6780, 0xc00009a3a0, 0x2, 0x2, 0x2da6780, 0xc00009a3a0)
        C:/Users/trent/go/pkg/mod/github.com/spf13/[email protected]/command.go:846 +0x2a4
github.com/spf13/cobra.(*Command).ExecuteC(0x2da42c0, 0x0, 0x16, 0xc000102058)
        C:/Users/trent/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x350
github.com/spf13/cobra.(*Command).Execute(...)
        C:/Users/trent/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main()
        C:/Users/trent/Documents/github.com/vorteil/vorteil/cmd/vorteil/main.go:39 +0x39
PS C:\Users\trent\Documents\github.com\vorteil\vorteil>

Steps to reproduce

Run this command on any VMDK

System details

Please include details about the system that the issue can be reproduced on, including:

windows 10
0.51

Bars are still displaying although --json set, debug always on

Even if --json is the logging module still renders the progress bars which should not happen when json output is requested

change unpack behavior

the command cli unpack helloworld.vorteil tries to unpack in the current directory. This behavior needs to be changed to he following:

2 mandatory args (project file and target directory)
If the target directory does not exist, it is getting created

Missing the 'projects new' subcommand

Summary

We should implement the projects new subcommand. This would make the process of creating a new project far easier, as it should generate a .vorteilproject and default.vcfg file.

Priority

High

Virtualizer, adding fail with Graphql error

Problem description

Not able to Virtualizer from Vorteil Studio, getting following error

GraphQL ERROR: Syntax Error GraphQL request (1:109) Unterminated string. 1: mutation newHyperVVirtualizer { newHyperVVirtualizer(name:"HyperV", headless: true, switch: "Default Switch ^ 2: ") }

Expected behaviour

Virtualization platform should be added.

Actual behaviour

Getting graphql error and not able to add.

Steps to reproduce

List of steps that must be taken to reproduce the issue.

System details

Please include details about the system that the issue can be reproduced on, including:

operating system / version ``````
CLI version ($ vorteil version) client: 4.0.2-37d914d (26 Aug 20 00:04 +0000) api: 0.1.x server: 4.0.2-37d914d (26 Aug 20 00:04 UTC) api: 0.1.1 compiler: 3.0.0 kernel: 1.0.x package: 3.x
hypervisors being used (firecracker, Virtualbox etc.) Hyper-V
cloud environment (if applicable)

firecracker not starting if no memory is configured in vcfg

Problem description

If there no memory is set in vcfg a VM with platform "firecracker" does not start with erro:

Error starting virtual machine: machine needs a nonzero amount of memory

Expected behaviour

Setting default memory for the VM

Steps to reproduce

Create vcfg without ram set.

System details

Please include details about the system that the issue can be reproduced on, including:

Linux
firecracker
local

Size calculation is wrong

Problem description

The calculated size of the disk is wrong with the new DNS implementation.

Expected behaviour

Disk starts without providing disk-size.

Actual behaviour

Startup stops because of disk-space

import-shared-objects creates wqrong path

Problem description

E.g. using the command on envoy create a lib/i386-linux-gnu folder which is not correct.

Expected behaviour

Shoud add libs under /lib/x86_64-linux-gnu

Actual behaviour

creates libs under /lib/i386-linux-gnu

Steps to reproduce

Run command on envoy binary

Supported environmental variables in logging configuration

Summary

Support the ability to use the pre-defined vorteil environmental variables in the logging statements (or possibly throughout the configuration file). See the below configuration for logging to an ELK stack locally:

[[logging]]
  config = ["Name=tcp" , "Host=$EXT_IP0", "Port=10100", "Format=json", "tls=Off"]
  type = "system"

[[logging]]
  config = ["Name=tcp" , "Host=$EXT_IP0", "Port=10100", "Format=json", "tls=Off"]
  type = "kernel"

Priority

High importance, but not critical

'vorteil pack' unexpected behaviour

Problem description

vorteil pack command in project directory creates wrong file.

Expected behaviour

Running vorteil pack in a directory should create a .vorteil package with the parent's folder name, e.g. myapp.vorteil.

Actual behaviour

If I'm in a project directory and run vorteil pack it creates a ..vorteil package. I'm assuming the first dot is the current directory.

Steps to reproduce

Run vorteil pack in a vorteil project directory

firecracker kernel-args have loglevel=9

Summary

loglevel=9 is a bit high for firecracker, 4 gives us memory errors etc so we should use 4 instead

Container converter not working on Mac.

$ ./vorteil projects convert-container golang:alpine test/dl
convert image: library/golang:alpine
registry docker.io, url https://registry-1.docker.io
downloading manifest file
downloading 5 layers
layer ed8968b2:   0s [==============================================================================] 276.0 KiB / 276.0 KiB
layer a92cc7c5: done [==============================================================================] 155.0 b / 155.0 b
layer e73272ec:   0s [==============================================================================] 126.0 b / 126.0 b
layer e871e8e8:   0s [==============================================================================] 102.3 MiB / 102.3 MiB
layer df20fa93: done [==============================================================================] 2.7 MiB / 2.7 MiB
all layers downloaded
unpack layer /var/folders/ph/j82f_zjd3bz91wn8fkgbwxyc0000gn/T/vtest026237546/df20fa93.tar into test/dl
unpigz not found, falling back to go gzip
lchown test/dl/bin: operation not permitted

Hostname ending with -

Problem description

Sometime hostname ends with a -

Expected behaviour

Avoid - at the end of hostnames

Steps to reproduce

Random occurence

Add descriptive help text for the --record flag

Summary

The --record flag on the run command does not have a useful description explaining what it does/how it is used.

Example:

$ vcli run ~/Documents/Projects/Postgres --help
...
--record string
...

os.Exit prevents deferred closes, which can suppress error messages in CLI commands

Problem description

We use os.Exit in many places in our CLI command code in order to return a fail exit status. We DO still want to return this failed exit status, but it's important that our deferred cleanup code is allowed to run and os.Exit prevents that. We need to rethink how we handle errors.

Timeout for google cloud too short for large images

Problem description

Although it is not the main use case, I tried to upload a 5GB image to GCP and it timed-out. The image got created so it all worked but the CLI returned with an error message.

Creating Image  ⠦                                                                                
timed out waiting for image creation

Container converter not respecting -v flag or -d flag.

It must be logging directly to logrus in some places, instead of to a CLI-provided elog.Logger.

Custom types defined for VCFG struct aren't unmarshalling correctly from JSON.

The strings aren't being un-escaped.

VCFG commands are not getting recognized on command line

Running a command like:

vorteil run /myproject --program.1.args="/newvalue" --vm.kernel=99.99.1

returns:

Error: unknown flag: --program.1.args

firecracker hostname is empty if not set in vcfg

Problem description

firecracker hostname is empty if not set in vcfg

Expected behaviour

The hostname should be set to a default if not configured in vcfg.

Actual behaviour

Environment variable "HOSTNAME" is empty

vorteil provisioners help not correct

Problem description

Provisioner command does not show the correct syntax

Expected behaviour

Command vorteil provisioners new amazon-ec2 shows the help and the syntax of the command.

It should show:

Usage:
  vorteil provisioners new amazon-ec2 [flags] file

but it shows

Usage:
  vorteil provisioners new amazon-ec2 [flags]

It is not obvious that the command needs a path to a file.

Steps to reproduce

Run vorteil provisioners new amazon-ec2

System details

N/A

vorteil run crashes app

running vorteil run /path/to/app results in crash:

[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x12c7cb1]

goroutine 1 [running]:
github.com/vorteil/vorteil/pkg/vimg.NewBuilder(0x19fd960, 0xc000120000, 0xc000415ac8, 0x0, 0x0, 0x0)
	/home/jensg/go/src/github.com/vorteil/vorteil/pkg/vimg/builder.go:88 +0x1f1
github.com/vorteil/vorteil/pkg/vdisk.Build(0x19fd960, 0xc000120000, 0x19e45a0, 0xc00058e050, 0xc000415c18, 0x0, 0x0)
	/home/jensg/go/src/github.com/vorteil/vorteil/pkg/vdisk/build.go:60 +0x26e
main.runQEMU(0x19fdde0, 0xc00072ee00, 0xc00000cf00, 0x0, 0x0)
	/home/jensg/go/src/github.com/vorteil/vorteil/cmd/vorteil/run.go:267 +0x5a1
main.glob..func29(0x2237620, 0xc00033dfa0, 0x1, 0x1)
	/home/jensg/go/src/github.com/vorteil/vorteil/cmd/vorteil/cli.go:1996 +0x6e5
github.com/spf13/cobra.(*Command).execute(0x2237620, 0xc00033df80, 0x1, 0x1, 0x2237620, 0xc00033df80)
	/home/jensg/go/pkg/mod/github.com/spf13/[email protected]/command.go:846 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0x2231ce0, 0xc00010e058, 0x0, 0x0)
	/home/jensg/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
	/home/jensg/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main()
	/home/jensg/go/src/github.com/vorteil/vorteil/cmd/vorteil/main.go:29 +0x36```

Provisioning Large Disks to GCP gives incorrect size to image Boot size

Problem description

If you provision a vorteil project to google and set vcfg field disk-size = "10 GiB". Google will estimate this image has a Boot Disk of 2GB which will cause all spawned instances to panic.

Steps to reproduce

Edit the helloworld package with disk-size = "10 GiB" and provision that to google.

System details

kernel: 20.10.5

Firecracker visualizer logic earlier than kernel manager logic

It looks like the Firecracker visualizer is running earlier than the kernel manager or whatever merges the default kernel values. Here is an example of what kind of problem this causes:

Scenario 1 - Not Kernel defined :

VCFG

....
[vm]
  cpus = 1
  ram = "91 MiB"
  inodes = 100
  disk-size = "+10 MiB"

.$ /cli run /tmp/helloworld/ --platform firecracker
Scanning inputs  ⠋                                                                                
Writing image done [=================================] 32.0 MiB / 32.0 MiB
Kernel 'firecracker-' VMLinux does not exist

visualizers logic thinks kernel is = "" , but it should be whatever the default / latest kernel is.

Scenario 2 - Old Kernel defined :

VCFG

....
[vm]
  cpus = 1
  ram = "91 MiB"
  inodes = 100
  kernel = "20.8.3"
  disk-size = "+10 MiB"

$ ./cli run /tmp/helloworld/ --platform firecracker
Scanning inputs  ⠋                                                                                
Requested kernel '20.8.3' is too old for this compiler. Using latest kernel instead.
Writing image done [==============================] 32.0 MiB / 32.0 MiB
Kernel 'firecracker-20.8.3' VMLinux does not exist

visualizers logic thinks kernel is = "20.8.3" (from vcfg) , but it should be whatever the default / latest kernel is (kernel manger is setting this).

App finish very fast, VM does not stop

Problem description

If an app finishes immediately and netlink does not have chance to register the VM stays online although the app has finished.

Expected behaviour

If an app finishes, the VM should shutdown

Actual behaviour

In the rare case of an app finishing immediately, the VM stays online.

Steps to reproduce

Use a simple go app as the only application in vcfg:

package main

func main() {

}

Change of firecracker url

Summary

Change of firecracker download URL from "https://storage.googleapis.com/" to our downloads DNS entry downloads.vorteil.io

cli prints warning about wrong kernel versions

Problem description

Tools are throwing a warning about an unusual kernel version.

Expected behaviour

No error message if the kernel has not changed

Actual behaviour

Warning:

detected an unusual remote kernel file update for source 'https://downloads.vorteil.io/kernels' on kernel '20.10.5'

vdisk.Build without GetKernel and GetLatestKernel provided

Problem description

vdisk.Build library function crashes with a null pointer.

Expected behaviour

A call like the following should either succeed or fail and return an error:

vdisk.Build(context.Background(), f, &vdisk.BuildArgs{
	PackageReader: pkgReader,
	Format:        vdisk.RAWFormat,
	KernelOptions: vdisk.KernelOptions{},
	Logger:        &elog.CLI{},
})

Actual behaviour

Because the certain variables are not set it crashes. These variables need to be like this before the call of the function:

	vkern.Global 
	vimg.GetKernel 
	vimg.GetLatestKernel

There are two options. Either checking if the values are nil in the build phase and return errors or create default values if the functions are not provided.

Steps to reproduce

Simple app at:

package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"github.com/vorteil/vorteil/pkg/elog"
	"github.com/vorteil/vorteil/pkg/vdisk"
	"github.com/vorteil/vorteil/pkg/vkern"
	"github.com/vorteil/vorteil/pkg/vpkg"
	"github.com/vorteil/vorteil/pkg/vproj"
)

func main() {

	proj, err := vproj.LoadProject("/home/jensg/vorteil-projects/hw2")
	if err != nil {
		log.Fatalf("ERR %v", err)
	}

	ptgt, err := proj.Target("default")
	if err != nil {
		log.Fatalf("ERR %v", err)
	}

	pkgb, err := ptgt.NewBuilder()
	if err != nil {
		log.Fatalf("ERR %v", err)
	}

	pkgReader, err := vpkg.ReaderFromBuilder(pkgb)
	if err != nil {
		log.Fatalf("ERR %v", err)
	}
	defer pkgReader.Close()

	f, err := os.Create("/tmp/demo")
	if err != nil {
		log.Fatalf("ERR %v", err)
	}
	defer f.Close()

	// ksrc, err := vkern.CLI(vkern.CLIArgs{
	// 	Directory:          "/tmp/jens",
	// 	DropPath:           "/tmp/jens",
	// 	RemoteRepositories: []string{"https://downloads.vorteil.io/kernels"},
	// }, nil)
	// if err != nil {
	// 	log.Fatalf("ERR %v", err)
	// }

	// vkern.Global = ksrc
	// vimg.GetKernel = ksrc.Get
	// vimg.GetLatestKernel = constructGetLastestKernelsFunc(ksrc)

	fmt.Printf("before %v\n", pkgReader)
	err = vdisk.Build(context.Background(), f, &vdisk.BuildArgs{
		PackageReader: pkgReader,
		Format:        vdisk.RAWFormat,
		KernelOptions: vdisk.KernelOptions{},
		Logger:        &elog.CLI{},
	})
	fmt.Printf("after")
	if err != nil {
		log.Fatalf("ERR %v", err)
	}

}

func constructGetLastestKernelsFunc(ksrc vkern.Manager) func(ctx context.Context) (vkern.CalVer, error) {
	return func(ctx context.Context) (vkern.CalVer, error) {
		s, err := ksrc.Latest()
		if err != nil {
			return vkern.CalVer(""), err
		}
		k, err := vkern.Parse(s)- hypervisors being used (firecracker, Virtualbox etc.)
- cloud environment (if applicable)
		if err != nil {
			return vkern.CalVer(""), err
		}
		return k, nil
	}
}

System details

Please include details about the system that the issue can be reproduced on, including:

Linux
CLI 0.5.1

The 'run' command could try to run an existing disk.

It'd have to extract information from the disk to know how to configure it. And it would have to validate some things and ignore others (wrong disk format for platform for one example, vcfg flags would be difficult but not impossible to support, as another).

Save vorteil machine image to disk after completed run

Summary

Allow vorteil to save the virtual machine disk image once the machine has completed / powered-down.

Priority

Critical as it allows for the image analysis tools to work

Make constructGetLastestKernelsFunc public

Summary

Making constructGetLastestKernelsFunc a public function. In most use cases people will probably copy and paste the function from the library source code which makes it error prone. It looks like external users want to use this as default.

vkern.CLIArgs crashes with non-existent directories

Problem description

If vkern.CLIArgs point to non-existent directories it crashes with a NPE

Expected behaviour

The following code snippet should throw an error or create /tmp/doesntexist"

	ksrc, err := vkern.CLI(vkern.CLIArgs{
		Directory:          "/tmp/doesntexist",
		DropPath:           "/tmp/doesntexist",
		RemoteRepositories: []string{"https://downloads.vorteil.io/kernels"},
	}, nil)

Actual behaviour

NPE with:

goroutine 1 [running]:
github.com/vorteil/vorteil/pkg/vkern.(*CLIRemoteManager).forceUpdate(0xc0001eac60, 0x9f0e60, 0xc0000240c8, 0x0, 0x0)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vkern/cli.go:141 +0xf8
github.com/vorteil/vorteil/pkg/vkern.(*CLIRemoteManager).update(0xc0001eac60, 0x9f0e60, 0xc0000240c8, 0x0, 0xc000139910)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vkern/cli.go:130 +0x74
github.com/vorteil/vorteil/pkg/vkern.(*CLIRemoteManager).List(0xc0001eac60, 0x9f0e60, 0xc0000240c8, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vkern/cli.go:382 +0x5a
github.com/vorteil/vorteil/pkg/vkern.(*CompoundManager).List(0xc0001f21c0, 0x9f0e60, 0xc0000240c8, 0x0, 0x93598f, 0x3, 0x0, 0x0)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vkern/manager-compund.go:61 +0xbe
github.com/vorteil/vorteil/pkg/vkern.(*CompoundManager).Get(0xc0001f21c0, 0x9f0e60, 0xc0000240c8, 0xc0001ab620, 0x7, 0xc000139c08, 0x7f6d87, 0xc0001a49c0)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vkern/manager-compund.go:30 +0x43
github.com/vorteil/vorteil/pkg/vimg.(*Builder).loadKernel(0xc00012ef00, 0x9f0e60, 0xc0000240c8, 0x93598f, 0x3)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vimg/os.go:49 +0x7e
github.com/vorteil/vorteil/pkg/vimg.(*Builder).calculateMinimumOSPartitionSize(0xc00012ef00, 0x9f0e60, 0xc0000240c8, 0x0, 0x0)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vimg/os.go:65 +0x6a
github.com/vorteil/vorteil/pkg/vimg.(*Builder).calculateMinimumSize(0xc00012ef00, 0x9f0e60, 0xc0000240c8, 0x0, 0x0)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vimg/builder.go:139 +0x4b
github.com/vorteil/vorteil/pkg/vimg.NewBuilder(0x9f0e60, 0xc0000240c8, 0xc000139d50, 0x7f420886b4e0, 0xc0001ec450, 0xc00019ac00)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vimg/builder.go:105 +0x1fe
github.com/vorteil/vorteil/pkg/vdisk.CreateBuilder(...)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vdisk/build.go:67
github.com/vorteil/vorteil/pkg/vdisk.build(0x9f0e60, 0xc0000240c8, 0x9ec740, 0xc000011098, 0xc00000c960, 0xc0001e6c00, 0x0, 0x0)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vdisk/build.go:78 +0x20b
github.com/vorteil/vorteil/pkg/vdisk.Build(0x9f0e60, 0xc0000240c8, 0x9ec740, 0xc000011098, 0xc0001e6c00, 0x0, 0x0)
	/home/jensg/go/pkg/mod/github.com/vorteil/[email protected]/pkg/vdisk/build.go:121 +0x109

System details

Please include details about the system that the issue can be reproduced on, including:

Linux
0.5.1

firecracker crashes if dhcp not running

Scanning inputs ⠋
Writing image done [==============================================================================] 652.0 MiB / 652.0 MiB
Error: Run 'sudo vorteil firecracker-setup' for the listener
^Cpanic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x188 pc=0x21f1b06]

goroutine 16 [running]:
github.com/firecracker-microvm/firecracker-go-sdk.(*Machine).Shutdown(0x0, 0x0, 0x0, 0x0, 0x0)
/home/jensg/go/pkg/mod/github.com/firecracker-microvm/[email protected]/machine.go:410 +0x26
github.com/vorteil/vorteil/pkg/virtualizers/firecracker.(*Virtualizer).Stop(0xc000956dc0, 0x297b368, 0x0)
/home/jensg/go/src/github.com/vorteil/vorteil/pkg/virtualizers/firecracker/virtualizer_linux.go:528 +0xbc
main.run.func2(0x2d36900, 0xc000956dc0, 0xc0004a8770)
/home/jensg/go/src/github.com/vorteil/vorteil/cmd/vorteil/run.go:257 +0x35
created by main.run
/home/jensg/go/src/github.com/vorteil/vorteil/cmd/vorteil/run.go:256 +0x7bb

Typo in the documentation

Problem description

Just a little typo in the documentation.

Expected behaviour

Typo is gone.

Actual behaviour

At the documentation page
https://docs.vorteil.io/docs/guides/configure-machines/build-from-os/
at step 6 there is a [[netowrk]] instead of [[network]]

Sparsifying Dynamic-VHDs

Problem description

In the branch https://github.com/vorteil/vorteil/tree/aws-rework. The dynamic-vhd's that get built are broken. Potentially something to do with the RegionisHole function and File seeking.

Steps to reproduce

List of steps that must be taken to reproduce the issue.

Build a disk using dynamic-vhd format from this branch
Use the disk attached to compare the hex

vhd that works

./cli version missing

Improve cloud custom userdata.

Summary

Make automatic retrieval of cloud custom userdata a bit smarter. (Needs discussion)

Improve Vm shutdown

Summary

Improving shut-down mechanism.

Priority

Some applications have different requirements. Suggestions:

Configurable timeout for shutdown
Send signal to configured applications first and then, after a timeout, send to the rest of the processes
Configure signal per app (?)

Set defaults during build

This is the list of defaults we should use during build:

Cwd = /
Stdout = /dev/vtty
Stderr = /dev/vtty
Privilege = root
MTU = 1500, 1460 if GCP
DNS = 8.8.4.4
MaxFDS = 1024
User = root

add sysctl in vcfg reference

sysctl is missing in the vcfg reference section on docs.vorteil.io

system level logging does not report utilisation information

Problem description

When the following configuration is applied:

[[logging]]
  config = ["Name=tcp", "Host=35.201.9.144", "Port=3000", "Format=json"]
  type = "system"

[[logging]]
  config = ["Name=tcp", "Host=35.201.9.144", "Port=3000", "Format=json"]
  type = "kernel"

Kernel messages are received, however, the system messages with the CPU, memory, disk and network information is not received.

Expected behaviour

Expect to see these types of messages

[{"date":1570599209.000048,"cpu_p":13.000000,"user_p":12.000000,"system_p":1.000000,"cpu0.p_cpu":13.000000,"cpu0.p_user":12.000000,"cpu0.p_system":1.000000},{"date":1570599210.000038,"cpu_p":99.000000,"user_p":93.000000,"system_p":6.000000,"cpu0.p_cpu":99.000000,"cpu0.p_user":93.000000,"cpu0.p_system":6.000000},{"date":1570599211.000217,"cpu_p":89.000000,"user_p":87.000000,"system_p":2.000000,"cpu0.p_cpu":89.000000,"cpu0.p_user":87.000000,"cpu0.p_system":2.000000},{"date":1570599212.000099,"cpu_p":0.000000,"user_p":0.000000,"system_p":0.000000,"cpu0.p_cpu":0.000000,"cpu0.p_user":0.000000,"cpu0.p_system":0.000000},{"date":1570599213.000217,"cpu_p":0.000000,"user_p":0.000000,"system_p":0.000000,"cpu0.p_cpu":0.000000,"cpu0.p_user":0.000000,"cpu0......

Actual behaviour

Only kernel messages are received.

Steps to reproduce

Configure the vcfg file with:

[[logging]]
  config = ["Name=tcp", "Host=35.201.9.144", "Port=3000", "Format=json"]
  type = "system"

[[logging]]
  config = ["Name=tcp", "Host=35.201.9.144", "Port=3000", "Format=json"]
  type = "kernel"

System details

Please include details about the system that the issue can be reproduced on, including:

Version: 0.5.6
Ref: 764564e
Released: Wed, 28 Oct 2020 01:00:09 +0000
Verified on AWS & GCP

Virtualizer shouldn't make a copy of a disk image.

Save time by not duplicating the entire disk.

make .vorteilproject optional

Summary

vorteil run should be able to run witout a .vorteilproject file and run with default.vcfg if it does not exist.

Priority

Nice to have

If kernel does not exist with firecracker it crashes

Writing image done [==============================================================================] 46.0 MiB / 46.0 MiB
Error: Kernel '20.9.5' VMLinux does not exist
^Cpanic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x188 pc=0x21aa4b6]

goroutine 27 [running]:
github.com/firecracker-microvm/firecracker-go-sdk.(*Machine).Shutdown(0x0, 0x0, 0x0, 0x0, 0x0)
	/home/jensg/go/pkg/mod/github.com/firecracker-microvm/[email protected]/machine.go:410 +0x26
github.com/vorteil/vorteil/pkg/virtualizers/firecracker.(*Virtualizer).Stop(0xc00014e840, 0x0, 0x0)
	/tmp/vorteil/pkg/virtualizers/firecracker/virtualizer_linux.go:542 +0xbc
main.run.func2(0x2d3cd80, 0xc00014e840, 0xc0001ab690)
	/tmp/vorteil/cmd/vorteil/run.go:385 +0x35
created by main.run
	/tmp/vorteil/cmd/vorteil/run.go:384 +0x7bb

Support packing/unpacking from a valid Vorteil repository URL

Summary

The unpack/pack commands should be able to work if the 'PACKABLE' argument is a valid URL that returns a Vorteil package file (eg. https://apps.vorteil.io/file/vorteil/helloworld).

Priority

High - examples/guides in documentation require it.

CNCF-Runtime discussion/presentation

Hello Vorteil team,

I'm one of the co-chairs of the CNCF SIG-Runtime. I'm reaching out and I think it would be great for you to present/discuss the project at one of our meetings. For example, a general overview of the project or/and a demo.

Feel free to add it to our agenda or reach out to me (raravena80 at gmail.com)

Thanks!

Container Converter Hangs When Downloading images with high layer count

Problem description

When trying to container convert images with a large layer count, the convert-container command seems to hang.

Expected behaviour

convert-container command should download all layers and extract them to the output dir.

Actual behaviour

All visible layers are downloaded and then the process hangs.

Steps to reproduce

I've been able to successfully reproduce the issue on these two images:

Example :

$ ./cli projects convert-container tensorflow/syntaxnet ./tensor
convert image: tensorflow/syntaxnet:latest
registry docker.io, url https://registry-1.docker.io
downloading manifest file
downloading 25 layers
layer cce4cca5:   0s [==============================================================================] 215.0 b / 215.0 b
layer e61641c8:   0s [==============================================================================] 579.5 KiB / 579.5 KiB
layer 7d27bd3d: done [==============================================================================] 40.6 MiB / 40.6 MiB
layer 2726297b: done [==============================================================================] 17.7 MiB / 17.7 MiB
layer 6d827a3e: done [==============================================================================] 49.1 MiB / 49.1 MiB
layer 68262275: done [==============================================================================] 242.0 b / 242.0 b
layer c03b117f:   0s [==============================================================================] 124.3 MiB / 124.3 MiB
layer 821a1547: done [==============================================================================] 282.3 KiB / 282.3 KiB
layer 74cc3387:   0s [==============================================================================] 144.7 MiB / 144.7 MiB
layer a8424c51: done [==============================================================================] 19.2 MiB / 19.2 MiB

At this point it hangs, I left it for about 5 minutes

System details

Ubuntu 20

direktiv / vorteil Goto Github PK

vorteil's Introduction

vorteil.io

turn your applications and containers into micro virtual machines

Installation

Linux

Windows

Mac

Dependencies

Using Vorteil

Hello World

Convert from a Container

Deploying an Image

Modify an Existing Package

Building From Source

Code of Conduct

Contributing

License

See Also

vorteil's People

Contributors

Stargazers

Watchers

Forkers

vorteil's Issues

Problem description

Expected behaviour

Actual behaviour

Steps to reproduce

System details

Summary

Priority

Problem description

Expected behaviour

Actual behaviour

Steps to reproduce

System details

Problem description

Expected behaviour

Steps to reproduce

System details

Problem description

Expected behaviour

Actual behaviour

Problem description

Expected behaviour

Actual behaviour

Steps to reproduce

Summary

Priority

Problem description

Expected behaviour

Actual behaviour

Steps to reproduce

Summary

Problem description

Expected behaviour

Steps to reproduce

Summary

Problem description

Problem description

Problem description

Expected behaviour

Actual behaviour

Problem description

Expected behaviour

Steps to reproduce

System details

Problem description

Steps to reproduce

System details

Scenario 1 - Not Kernel defined :

Scenario 2 - Old Kernel defined :

Problem description

Expected behaviour

Actual behaviour

Steps to reproduce

Summary

Problem description

Expected behaviour