viniciusbds / navio

Navio is a program to create and manage linux containers. This project exists only for study purposes, so feel free to contribute :)

Navio

Navio is a simple tool for creating and managing linux containers.

A container is a set of processes isolated by Linux namespaces. We use, for example, PID namespace to isolate the PID number space, UTS to isolate the hostname and MOUNT to isolate the filesystem mount points.

All containers have their own rootfs (a isolated "mini operating system") associated, so that a change (for example, an installation of any library or a creation of a file) in a container does not affect others ones.

It is also possible use Cgroups to limit the amount of resources that each container can use.

Why?

For study purposes only. If you want to use real containers use Docker or Containerd.

Available Default Images

Image	version	size
alpine	v3.11	2.7M
busybox	v4.0	1.5M
ubuntu	v20.04	90M

These are the default images, but you can easily create your own images with the packages and files you find necessary. For example, with the following Naviofile:

FROM ubuntu    

ADD . /mydir

RUN apt update && apt upgrade -y && apt install python -y

cd /path/to/Naviofile
sudo navio build . --t python-image

we created an image with python installed, so the containers that use this image will already have python installed by default, as well the files from the current directory that will be copied to the /mydir directory in the container.

Demo

Namespaces

what the processes can see

• UTS - isolate hostname and domainname

• PID - isolate the PID number space

• MNT - isolate filesystem mount points

Cgroups

what the processes can use

• Memory

• CPU

• Process numbers

Available Commands

How to use

To create a basic container just type:

sudo navio create ubuntu bash

But you can use some flags, ex:

--name ubuone to define the name of container

--pids 25 to limit the maximum number of processes that the container can create

--cpus 0-3 to limit the cpu cores that the container can use. For example, in this case (0-3) if you computer have at least 4 cores, this flag will permit that the process of container use only 4 cores, no more

--cpus-share 100

--memory 1G to limit the maximum memory RAM that the container can use

Limiting the container to use only 4 cpu cores

Building new Images

To build a image you need first create a Naviofile with the base image and the commands.

Limitations

• The network is not being isolated and is only working on the ubuntu image.
• The Navio does not allow containers to run in the background.
• The Navio does not limit the use of I/O

Requirements

• linux. Navio's doesn't support other operational system :(
• golang environment
• make
• wget
• mysql configured with the root password == root
• some of commands (ex.: navio build, navio create, navio rmi and navio exec) must be executed with sudo privilegies.

How to install

1°: we assume that your root mysql database user and password is 'root', feel free to change it locally.

If you just want use, is very simples

 git clone https://github.com/viniciusbds/navio.git
 cd navio
 ./install.sh

If you want compile the code before install

 git clone https://github.com/viniciusbds/navio.git
 cd navio
 make
 ./install.sh

To run all unit tests, type

 cd /path/to/project/navio
 sudo make unit-tests

To uninstall

 cd navio
 ./uninstall.sh

Example Commands

$ navio images

$ sudo navio create ubuntu bash --name mycontainer`

$ navio containers

ID	   NAME	   	   IMAGE  	COMMAND  	STATUS

14806622   mycontainer     ubuntu  	bash  		Stopped

$ sudo navio exec 14806622 bash 

$ navio rename 14806622 new_name

$ sudo navio create busybox sh

$ sudo navio create alpine /bin/sh --name levezin

Contributing

You can contribute to the project in any way you want, either by fixing bugs, implementing new features, improving the documentation or proposing new features through issues

See Contributting for more details

References

• Containers from Scratch • Liz Rice

• Containers from Scratch

• Building a container with less than 100 lines in Go

• Linux Namespaces

• Namespaces

• Understanding Containerization By Recreating Docker

• Doqueru kun

• Icon made by Freepik from www.flaticon.com

navio's Issues

Allow containers to run in background

What would you like to be added:

Once we are inside the shell (via navio run ubuntu /bin/bash, for example), there should be some way to get out without killing the process

Use a progressbar in the i/o operations

What would you like to be added:

we need use the progressbar in all I/O operations.

Problem on the directory to download images

Problem on the directory to download images

What happened:
The navio pull command download the image in the directory where we run the navio binary.

What you expected to happen:

Is better define a default directory to put these images

Add `golangci-lint`

Description

golangci-lint is a fast Go linters runner. It runs linters in parallel, uses caching, supports yaml config, has integrations with all major IDE, and has dozens of linters included.

References

https://golangci-lint.run/

go modules

Aprender e usar os go modules da maneira mais eficiente para o projeto

Using Go Modules

Fix golangci-lint Errors

run golangci-lint
  Running [/home/runner/golangci-lint-1.32.2-linux-amd64/golangci-lint run --out-format=github-actions --enable goimports] in [] ...
  Error: `naiveSelf` is unused (deadcode)
  Error: `imgTag` is unused (deadcode)
  Error: Error return value of `insForm.Exec` is not checked (errcheck)
  Error: Error return value of `insForm.Exec` is not checked (errcheck)
  Error: Error return value of `insForm.Exec` is not checked (errcheck)
  Error: Error return value of `delForm.Exec` is not checked (errcheck)
  Error: Error return value of `Insert` is not checked (errcheck)
  Error: Error return value of `Remove` is not checked (errcheck)
  Error: Error return value of `Pull` is not checked (errcheck)
  Error: Error return value of `Pull` is not checked (errcheck)
  Error: Error return value of `Pull` is not checked (errcheck)
  Error: Error return value of `RemoveAll` is not checked (errcheck)
  Error: Error return value of `removeContainer` is not checked (errcheck)
  Error: Error return value of `removeContainerDB` is not checked (errcheck)
  Error: Error return value of `CreateContainer` is not checked (errcheck)
  Error: Error return value of `Remove` is not checked (errcheck)
  Error: Error return value of `CreateContainer` is not checked (errcheck)
  Error: Error return value of `CreateContainer` is not checked (errcheck)
  Error: Error return value of `images.PrepareRootFS` is not checked (errcheck)
  Error: Error return value of `updateStatus` is not checked (errcheck)
  Error: Error return value of `updateStatus` is not checked (errcheck)
  Error: Error return value of `os.Mkdir` is not checked (errcheck)
  Error: Error return value of `os.Mkdir` is not checked (errcheck)
  Error: Error return value of `os.Mkdir` is not checked (errcheck)
  Error: Error return value of `Remove` is not checked (errcheck)
  Error: Error return value of `rootCmd.MarkFlagRequired` is not checked (errcheck)
  Error: Error return value of `images.Untar` is not checked (errcheck)
  Error: Error return value of `io.Copy` is not checked (errcheck)
  Error: Error return value of `containers.CreateContainer` is not checked (errcheck)
  Error: Error return value of `containers.Exec` is not checked (errcheck)
  Error: Error return value of `io.Tar` is not checked (errcheck)
  Error: Error return value of `images.Insert` is not checked (errcheck)
  Error: Error return value of `rootCmd.MarkFlagRequired` is not checked (errcheck)
  Error: Error return value of `containers.CreateContainer` is not checked (errcheck)
  Error: Error return value of `containers.Exec` is not checked (errcheck)
  Error: Error return value of `containers.UpdateName` is not checked (errcheck)
  Error: File is not `goimports`-ed (goimports)
  Error: S1008: should use 'return len(strings.TrimSpace(imageName)) == 0' instead of 'if len(strings.TrimSpace(imageName)) == 0 { return true }; return false' (gosimple)
  Error: SA9003: empty branch (staticcheck)
  Error: SA9003: empty branch (staticcheck)
  Error: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
  Error: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
  
  Error: issues found
  Ran golangci-lint in 8253ms

Develop the kill command

https://github.com/viniciusbds/navio/blob/master/cmd/kill.go

Develop the pause command

https://github.com/viniciusbds/navio/blob/master/cmd/pause.go

Adicionar os pre requisitos

• golang
• cobra

Fix database configuration

Chgroups não está funcionando corretamente

nesse exemplo o número máximo de processos foi definido para 20 e isso não foi respeitado

Develop the restart command

https://github.com/viniciusbds/navio/blob/master/cmd/restart.go

Documentation of exported functions

Document all the source code that contains the [TODO]: Document. Doesn't need document the unexported functions

e.x:

// Execute executes the root command.
// [TODO]: Document this function
func Execute() {
	if err := rootCmd.Execute(); err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
}

Progress

• assert/assert.go
• cmd/root.go
• images/images.go
• src/containers/containers.go
• src/logger/logger.go
• src/util/util.go

Networking error

When we run navio run ubuntu /bin/bash the container is create perfectly

But there is some bug on the process, because this error appears when we type: apt-get update

Err:1 http://archive.ubuntu.com/ubuntu focal InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu focal-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

Testing progress

• assert
• images
• logger
• utilities
• container

The build command is not calulating the size of a image

What happened:
The build command is not calulating the size of a image

What you expected to happen:
The image size must be calulated

Navio commands

• navio ps : mostra todos os containeres, indicando id, nome, status ...

• navio build . --t [image-name] : gera uma imagem com base no arquivo Naviofile

• navio run [image-name] [command] [args...]

• navio pull image [image-name]

• navio get images #10

• navio remove image [image-name] #10

• navio describe image [image-name]

Add more images

• Ubuntu Fossa 20.04 LTS
• Alpine Linux
• Busybox
• Arch Linux
• Debian

And others ...

Criar um diagrama com as entidades do sistema

What would you like to be added:
Adicionar um diagrama com todas as entidades do sistema. Ex.: Container, Imagem, etc ...
Why is this needed:
Fará com que novos contribuidores entendam melhor a organização do projeto, para que possam contribuir com o mesmo

Criar um agente de log com registro nivelado

Criar um agente de log com registro nivelado.

O registro nivelado é um meio de categorizar seus registros para que você possa procurar seus registros para tipos específicos de eventos. Os níveis que vamos colocar no nosso agente de log são:

---

O nível info, que representa eventos do tipo informação que informam o usuário de uma ação, como Program started ou Email sent. Isso nos ajuda a depurar e rastrear partes do nosso programa para ver se o comportamento esperado está acontecendo.

O nível warning. Esses tipos de eventos identificam quando algo inesperado está acontecendo e que não é um erro, como Email failed to send, retrying. Eles nos ajudam a ver partes do nosso programa que não estão rodando tão bem quanto esperávamos que estivessem.

O nível error , que significa que o programa encontrou um problema, como File not found. Frequentemente, isso resultará na falha da operação do programa.

fonte: https://www.digitalocean.com/community/tutorials/understanding-package-visibility-in-go-pt#visibilidade-dentro-de-m%C3%A9todos

MOver configuração do banco de dados para Docker

What would you like to be added:

Why is this needed:

Generate a random name for container without the --name flag

What is happing:

Currently, if the user doesn't insert the --name containerName flag we use XPTO as the containerName

see https://github.com/viniciusbds/navio/blob/master/cmd/run.go#L43

if containerName == "" {
	// TODO: generate a random container name
	containerName = "XPTO"
}

What you expected to happen:

The cmd run commannd must generate a random name. ex: container1928327421, instead XPTO

Develop the unpause command

https://github.com/viniciusbds/navio/blob/master/cmd/unpause.go

Develop the rename command

https://github.com/viniciusbds/navio/blob/master/cmd/rename.go

Debug the vscode golang project using root privilege

O vscode não consegue testar o seguinte código de teste go.

package logger

import (
	"testing"

	"github.com/viniciusbds/navio/src/container"
)

func TestLog(t *testing.T) {
	result := container.CreateContainer([]string{"run", "ls", ""})
	expected := "oi"

	if result != expected {
		t.Errorf("[ERROR] expected %s, result %s", expected, result)
	}
}

O motivo é que para executar o código do pacote github.com/viniciusbds/navio/src/container é necessário que o usuário tenha permissões de super usuário.

Mensagem de erro

Running tool: /usr/local/go/bin/go test -timeout 30s -coverprofile=/tmp/vscode-go9DQZ83/go-code-cover github.com/viniciusbds/navio/src/logger

ERROR fork/exec ./navio: operation not permitted
FAIL	github.com/viniciusbds/navio/src/logger	0.001s
FAIL
Error: Tests failed.

Navio get images

Develop a command that list all available images in the /images directory

This command is util after we use the pull command, to see that we have downloaded the image

"Vendorizar a aplicação ou usar Go packages? "

Escolher qual é a melhor opção

https://github.com/Masterminds/glide

Error on ShowDownloadedImages

What happened:

When we move the navio bin to a new directory, a error on navio get images appears: the directory doesn't exists

vinicius_barbosa_silva@experiments:/usr/local/bin$ ./navio get images
[ERROR] 12:33AM --> open ./images: no such file or directory
[ERROR] 12:33AM --> invalid argument

What you expected to happen:

Check if the dir exists and if not is the case, create it

Configure Cgroups

What would you like to be added:

Configure Cgroups to allow the user to limit resources using parameters

• Memory

• CPU

• I/O

• Process numbers

Develop the rm command

https://github.com/viniciusbds/navio/blob/master/cmd/rm.go

Improve the images package

Refactore it.

l.Log("INFO", fmt.Sprintf("Downloading %s  from %s ...", file, url))
	wgetCmd := exec.Command("wget", url)
	mkdirCmd := exec.Command("mkdir", "-p", imagePath)
	tarCmd := exec.Command("tar", "-C", imagePath, "-xf", file)
	rmFileCmd := exec.Command("rm", file)

	util.Must(wgetCmd.Run())
	util.Must(mkdirCmd.Run())
	util.Must(tarCmd.Run())
	util.Must(rmFileCmd.Run())

Maybe is better use the os package directly ex.:

os.Mkdir(imagePath)

instead of

exec.Command("mkdir", "-p", imagePath)

Bug on container creation

What happened

When we interrupt the create command (with a CTRL + C) while the Navio is downloading a new image, we enter a state of inconsistency. Only a part of the image is downloaded and the system breaks when execute a new container that use the inconsistent image.

example (How to reproduce it)

What you expected to happen:

The command should remove the inconsistent image and download it again

Finish the BUILD command to support

Implements DAO pattern