shreyanshk / boottorrent

P2P Network booting with BitTorrent

BootTorrent

BootTorrent allows distributed P2P BitTorrent based network booting of various Operating systems.

The advantage of BootTorrent over standard network booting is that the bottleneck caused by a central server serving the image is reduced and BootTorrent reduces the linear scaling factor in total time to bring a cluster online fully functional by utilizing Peer-to-Peer protocols. The nodes share the image data among themselves, hence, improved boot times.

+--------------------------------------------------------------------+
|                                     +----------------------------+ |
|  Fig 1: A network of computers      |     +----+  FE  +----+     | |
|    sharing data together after      |     |Node|<---->|Node|     | |
|    receiving it from server.        |     +----+      +----+     | |
|                                     |       ^           ^        | |
|                                     |    FE |           | FE     | |
|  +------+ 1 Gbps  +------+ 100 Mbps |       v           v        | |
|  |Server|-------->|Switch|--------->|     +----+  FE  +----+     | |
|  +------+  Link   +------+ Link(FE) |     |Node|<---->|Node|     | |
|                                     |     +----+      +----+     | |
|                                     +----------------------------+ |
+--------------------------------------------------------------------+

Cases where BootTorrent may be useful would be:

• If the clients in your network are not getting enough bandwidth individually and have significant bandwidth being left unused, which causes increased boot times. With BootTorrent you can repurpose the remaining bandwidth to help clients mutually share it among themselves.
• If your server (such as a laptop) can only connect to your cluster of computers via a comparatively slow link (such as WiFi or Fast-ethernet) then BootTorrent can help you mitigate the low bandwidth issues of network link.
• If you have large number of computers at your disposal and you're simply looking to deploy any given system image(s) (that may have been hand-crafted according to your needs) as painlessly as possible. BootTorrent can help you deploy it in three easy steps to the whole network.
• If your current network boot server is unable to meet your requirements and deliver much needed performance, consider giving BootTorrent a try. Its distributed architecture will reduce the dependence on server, which means improved boot performance.

For more details on use cases please refer to Use cases list and visit the documentation.

We have data to back our claims. Check out the performance improvements here:

[Images & Data courtesy of SL-Lab: http://sl-lab.it/dokuwiki/lib/exe/fetch.php/tesi:tesi_bruschi.pdf]

The above images were created from tests done at the University of Milan (ITALY) during the development of the original "boottorrent" project.

WARNING: STATUS - ALPHA: DO NOT USE IN PRODUCTION!

• Free software: GNU General Public License v3
• Documentation: https://boottorrent.readthedocs.io.

Features

• Boot Linux Kernel with Kexec.
• Includes Qemu to run any type of Operating system on client machine.
• Ability to run a binary executable.
• Client side user input to choose an OS.
• Leverage BitTorrent to transfer files over the network.

boottorrent's Issues

qemu method starts without focus

the qemu method starts without focus so that when the ph2 boot is finished the user must click the mouse to gain keyboard control

images in the README

maybe linking to the graph on this (http://sl-lab.it/dokuwiki/doku.php/tesi:boottorrent_en#graphs) section of the original thesis would be even more explicative

python, why?

given all the problems we are facing with python (versions, virtualenv, dependencies, etc.) I'd like you to consider abandoning it for a more sane (bash maybe?) language

(from clean install) error in starting boottorrent

• untouched files, from commit 70630c1
• inside virtualenv

(venv) bt@boottorrent:~/boottorrent/envtest$ sudo boottorrent start
Traceback (most recent call last):
File "/usr/local/bin/boottorrent", line 11, in
load_entry_point('boottorrent==0.1.0', 'console_scripts', 'boottorrent')()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 480, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2691, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2322, in load
return self.resolve()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2328, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/usr/local/lib/python2.7/dist-packages/boottorrent/init.py", line 9, in
from .boottorrent import BootTorrent # noqa
File "/usr/local/lib/python2.7/dist-packages/boottorrent/boottorrent.py", line 117
print(line, end="")
^
SyntaxError: invalid syntax

The problem is sudo of course, but I tried to activate virtualenv after a 'sudo su -' (changing to the venv dir), same error

yet another install error

$ sudo python3 setup.py install
[sudo] password for bt:
Traceback (most recent call last):
File "setup.py", line 9, in
readme = readme_file.read()
File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 728: ordinal not in range(128)

killing qemu "shuts down" the whole machine

I've noticed that killing the qemu process "shuts down" (actually, the screen goes blank instantly, keyboard is useless and the only option is the power switch) the machine.

It may be more useful if, e.g., the tui could re-gain control?

referring to the usecases file in the README file...

I'm not sure about the usefulness of the usecases file for external users, I see it as an internal document for the development team, maybe here a brief summary of the hypothetical contexts can be described, what do you think?

TUI default item not always selected

in tui the default starting item (if it's not the first one in the list) is not automatically selected so it's a bit confusing for the user

python code is not documented!

I cannot evaluate the code, I'd like to understand the workings

bsdtar

is there a reason to use bsdtar instead of the "standard" tar?

verbose autodiscovery ?

I'm running boottorent without hefur, it's taking ages... can the autodiscovery be made more verbose?

timeout should be visible

if boot timeout is enabled a "countdown timer" should be visible in the TUI

running xorg in ph2 with qemu method: may be made optional?

X is not always needed for a qemu method ph2 boot (e.g. when booting OSes such as FreeDOS or a console-mode-only Linux), could it be made optional?

(see also #36)

httpd running in ph1 boot (slitaz)

is it useful for something?
can it be removed?

python version concerns

I'm concerned about your choice of python 3.6, it's a version that's not available on Debian stable (in fact it's only in testing and unstable), many servers use stable...
Can you motivate?

Maybe the virtualenv tool is able to download an entirely new and recent version of python?

sequence diagram

in the https://boottorrent.readthedocs.io/en/latest/internals.html doc page, the sequence diagram is not very readable, if (as suggested) I open it in a new page this is the result:

tried "-accel=kvm" but it only shows a "_" cursor

(moreover other consoles are no longer reachable, I have to switch off and then on again the machine)

how can I debug it?

rename envtest/Boottorrent.yaml

since it's an example I would rename it to Boottorrent.yaml.example

so that, once configured a local Boottorrent.yaml, every time a user 'git pull's the repo for updates git would not complain

error in "make initrd"

$ make initrd
tui.go:5:2: cannot find package "github.com/jroimartin/gocui" in any of:
/usr/lib/go-1.10/src/github.com/jroimartin/gocui (from $GOROOT)
/home/.../apps/GOPATH/src/github.com/jroimartin/gocui (from $GOPATH)
tui.go:6:2: cannot find package "gopkg.in/yaml.v2" in any of:
/usr/lib/go-1.10/src/gopkg.in/yaml.v2 (from $GOROOT)
/home/.../apps/GOPATH/src/gopkg.in/yaml.v2 (from $GOPATH)
make: *** [Makefile:4: initrd] Error 1

in "quick start" a small demo/test should be included

in the https://boottorrent.readthedocs.io/en/latest/quickstart.html#configure-the-linux-kernel page the user is told to copy everything, it may be useful to offer a small test O.S. (FreeDos? Minix? Slitaz itself?) to be booted for a quick and dirty test

kvm by default in qemu?

would you suggest/put a "-accel kvm" in args for qemu in the various config.yaml?

pip install failed

$ pip install -e boottorrent
Directory 'boottorrent' is not installable. File 'setup.py' not found.

starting boottorrent and netcard config

when starting boottorrent the network card should be already configured statically otherwise dnsmasq could fail

aria2 parameters configurable

it would be useful to have some configurability (at the yaml level) of the aria2 daemon running on the nodes, e.g., to change verbosity

dafault values in config files for missing variables?

I don't know if it would be useful or misleading for the user, how about setting default values for variables in config files (yaml)? I mean: implementing a hard-coded default value if the variable is not found in the yaml.

It happened to me that I upgraded boottorrent but left Boottorrent.yaml at previous version (without console_log_level for aria2) and I got an error.
Is it better getting an error or a default value?

in general, look at commit logs when I commit something

I usually commit notes, so that if you look at my commits you can have directions

hefur is not a standard package (Debian/Devuan at least)

are there alternatives?

requirements in the README file

these requirements are for the server only, correct?
and not even all of them if the bootimage is already created?
it may be useful to distinguish the requirements for every phase

could not contact transmission

on the booting node, see:

transmission daemon should be inactive

or the boottorrent-activated transmission daemon (I'm referring to the /etc/init.d/... one) should work on a different port
(maybe a note in the doc?)

tmux/screen?

in the phase1 boot could it be useful to have a tmux/screen session opened to have a shell in parallel with the boottorrent procedure?

in "quick start", file not found, it searches in the wrong dir?

I'm following the instructions in https://boottorrent.readthedocs.io/en/latest/quickstart.html#booting-the-kernel and I get this:

$ boottorrent start
Traceback (most recent call last):
File "/usr/local/bin/boottorrent", line 11, in
load_entry_point('boottorrent==0.1.0', 'console_scripts', 'boottorrent')()
File "/usr/local/lib/python3.6/dist-packages/click-6.7-py3.6.egg/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/click-6.7-py3.6.egg/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/dist-packages/click-6.7-py3.6.egg/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/dist-packages/click-6.7-py3.6.egg/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/dist-packages/click-6.7-py3.6.egg/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/boottorrent-0.1.0-py3.6.egg/boottorrent/cli.py", line 43, in start
bt.start()
File "/usr/local/lib/python3.6/dist-packages/boottorrent-0.1.0-py3.6.egg/boottorrent/boottorrent.py", line 72, in start
self.generate_client_config()
File "/usr/local/lib/python3.6/dist-packages/boottorrent-0.1.0-py3.6.egg/boottorrent/boottorrent.py", line 276, in generate_client_config
osconfig = open(f"{self.wd}/oss/{os}/config.yaml", "r").read()
FileNotFoundError: [Errno 2] No such file or directory: '/home/bt/boottorrent/bootprova/oss/a/config.yaml'

I have created the:
bootprova/oss
bootprova/oss/README.rst
bootprova/oss/atrent-test
bootprova/oss/atrent-test/config.yaml
bootprova/oss/atrent-test/initramfs-linux-lts.img
bootprova/oss/atrent-test/vmlinuz-linux-lts

And I'm launching 'boottorrent start' while in dir 'bootprova', here are the config files:

$ cat Boottorrent.yaml
boottorrent:
version: 0
timeout: 30
seed_time: 0
default_os: atrent-test
host_ip: 192.168.1.1
display_oss: atrent-test

- testos

- freedos

dnsmasq:
enable_dhcp: True
enable_tftp: True
interface: eth0
bind_interfaces: True
dhcp_range: "192.168.1.50,192.168.1.150,12h"
#user: shreyansh

hefur:
enable: True
port: 10001

transmission:
rpc_port: 9091

aria2:
bt_enable_lpd: True
check_integrity: True
enable_dht: False
enable_dht6: False
enable_peer_exchange: True

$ cat oss/atrent-test/config.yaml
dispname: atrent-test
method: kexec
kernel: vmlinuz-linux-lts
initrd: initramfs-linux-lts.img
cmdline: break

[not an issue, just a reminder] The main value of Free Software...

... is the knowledge that you (as the author) share.
You should make yourself very clear, your documentation perfect and your code easily readable and understandable.
That's the first priority, according to me of course.

please look at my comment in the README.rst

a good intro for someone who does not know the goal of the project is still missing

Slim SliTaz base image

Image generation files at phase1bootstrap/slitaz.

general note for the README file

[from now on I'll also use issues to track notes/comments I make in various files, so that we can track down the process more easily and transparently]

The README is way better now!

The next thing to do is to summarize there (as an introduction) the different "usecases" (contexts&actors) so that a "would be" user may identify if and how this project can be of use for him

At the moment it's still not clear (i.e., not documented) the role of the 'boottorrent' executable (.py), I understand (but I had to read the source code!) that it's a "launcher" of various components (e.g., the dnsmasq), it must be stated and documented in detail. See also #7

a way to use "self" IP address in Boottorrent.yaml

in the Boottorrent.yaml config file instead of putting an explicit IP address for the 'host_ip' variable it would be useful to have a "self" (or something else) keyword to use the current IP address of the netcard

missing assets in pip package

• boottorrent version: 0.1.0
• Python version: 3.6
• Operating System: Debian stable/testing

Description

I was trying to init a new boottorrent project but the command ended with an error since assets are not included in package

What I Did

I installed dependences then installed boottorrent via pip, please see my deployment notes for details, if needed

after boottorrent installation I did:

$ boottorrent init boottest
Traceback (most recent call last):
  File "/usr/local/bin/boottorrent", line 11, in <module>
    load_entry_point('boottorrent==0.1.0', 'console_scripts', 'boottorrent')()
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/boottorrent/cli.py", line 24, in init
    copy_tree(base, nfolder)
  File "/usr/lib/python3.6/distutils/dir_util.py", line 127, in copy_tree
    "cannot copy tree '%s': not a directory" % src)
distutils.errors.DistutilsFileError: cannot copy tree '/usr/local/lib/python3.6/dist-packages/boottorrent/assets/skel': not a directory

qemu-iso method not starting

I'm testing the iso boot without success, after the torrent is finished a single underscore cursor is displayed and nothing happens, could it be that when using qemu-iso method I've seen in tui.go that you launch
qemu := exec.Command("/usr/bin/qemu-system-x86_64")
but this may not work on 32bit machines?

also, what are the qemu memory parameters? qemu defaults to 128M

qemu launch should be more confgurable?

I mean, the actual executable to be launched (whether qemu-system-x86_64 or qemu-system-i386 or any other architecture emu) could be chosen by config.yaml?

'boottorrent' is not a command

see https://github.com/shreyanshk/boottorrent#get-started

there is no file named 'boottorrent' anywhere

maybe a missing dependency?

I'm trying boottorrent on a clean Debian (Devuan actually) machine, after cloning and 'make initrd' I tried:

$ python3 setup.py install
Traceback (most recent call last):
File "setup.py", line 6, in
from setuptools import setup, find_packages
ModuleNotFoundError: No module named 'setuptools'

Then by installing python3-setuptools it (almost) works since the command should be prepended with sudo?

alpine (in the example envtest) does not boot

error is "could not read the bot disk"

system architecture description is badly needed

you should describe (maybe with a diagram) the various components of the system you're thinking and working on, with a "map" (a matrix) of "what should run on which component"