bastillebsd / bastille Goto Github PK

The template sub-command errors trying to execute service hooks due to unnecessary quotes in the script. This is easily fixed and will be patched in the next release.

Issue looks like:

--- template.sh	2019-11-12 23:11:09.473457000 +0000
+++ template.sh 2019-11-12 23:11:01.577429000 +0000
@@ -162,7 +162,7 @@
     if [ -s "${bastille_template_SERVICE}" ]; then
         echo -e "${COLOR_GREEN}Managing services.${COLOR_RESET}"
         while read _service; do
-            jexec -l ${_jail} /usr/sbin/service "${_service}"
+            jexec -l ${_jail} /usr/sbin/service ${_service}
         done < "${bastille_template_SERVICE}"
     fi

Unfortunately I did not see the notice on how to setup ZFS (since it wasn't visible in the documentation) but after I found the README in GitHub I realised I hadn't set it up properly.

I assume it would be a manual process, but how would you go about switching to ZFS after-the-fact?

Leaving this here for any takers (or for future @cedwards )...

Bastille needs a man page. The tool provides help output, but a proper man page would be a nice addition.

Hello there !

I try to mount a folder form the host to a jail.

I have added this line to the jails fstab at /usr/local/bastille/jails/sonarr/fstab

/data/mediaserver /usr/local/bastille/jails/sonarr/root/mnt/mediaserver nullfs rw 0 0

The jail start with no error but the mount is not present.

Installed bastille from latest pkg

sudo bastille -v
0.5.20191128

host and jail are on 12.1-RELEASE-p1

jail.conf

interface = bce1;
host.hostname = sonarr;
exec.consolelog = /usr/local/bastille/logs/sonarr_console.log;
path = /usr/local/bastille/jails/sonarr/root;
ip6 = disable;
securelevel = 2;
devfs_ruleset = 4;
enforce_statfs = 2;
exec.start = '/bin/sh /etc/rc';
exec.stop = '/bin/sh /etc/rc.shutdown';
exec.clean;
mount.devfs;
mount.fstab = /usr/local/bastille/jails/sonarr/fstab;

sonarr {
	ip4.addr = 192.168.1.113;
}

Requested via IRC, the ability to bootstrap, create, and otherwise manage HardenedBSD-based jails on a HardenedBSD system.

Initially I expect this should only require patching the bootstrap and create functions.

bastille.conf contains a bastille_zfs_mountpoint option to change where the jails are mounted, but it's not actually used anywhere in the code and therefore has no effect.

subsequent to #129 a snapshot and restore feature (based on zfs) would be nice to have

Hi @cedwards
Can you briefly explain how bastille compares to iocage? (and to Docker for that matter)

Thanks!

In the installation notes and on the documentation page there is no mention to copy the jib or jng files from /usr/share/examples/jails/{jib,jng} files to /usr/sbin/{jib,jng}.

This needs to be done before people can use VNET enabled jails.

Missed a test; easy fix.

Update /usr/local/share/bastille/create.sh:

--- create.sh
+++ create.sh
@@ -211,6 +211,9 @@
 12.0-RELEASE|12.0-release)
     RELEASE="12.0-RELEASE"
     ;;
+12.1-RELEASE|12.1-release)
+    RELEASE="12.0-RELEASE"
+    ;;
 11-stable-LAST|11-STABLE-last|11-stable-last|11-STABLE-LAST)
     RELEASE="11-stable-LAST"
     ;;

This fix and an upgrade to the entire bootstrap/create release verification pending 0.5.x release.

I have been creating some test templates, one is a node.js app if I do npm insall -g appname in CMD, and then add it as a service it will start fine from restart of container or console.

If I add SERVICE and the app start it can't start as CMD runs after SERVICE.

Would it be better to change the order of template creation so SERVICE runs last?

This issue / question was raised in IRC and noting here...

In 0.5.x a user is unable to add bootstrap archives and re-bootstrap. The system reports "already bootstrapped" and is done.

It would be nice to have functionality that would allow later adding other distfiles such as lib32, ports, src, etc.

When starting or stopping a VNET jail we get the error no IP address found for -

root@test01-head:~/projects/bastillebsd/bastille # bastille start vnetjail3
[vnetjail3]:
e0a_bastille3
e0b_bastille3
vnetjail3: created
no IP address found for -

And a stop
root@test01-head:~/projects/bastillebsd/bastille # bastille stop vnetjail5
no IP address found for -
[vnetjail5]:
vnetjail5: removed

This is from pfctl which can not find the IP address. from the command $(jls -j ${_jail} ip4.addr).

when you remove old jails it should archive the logs (ie /usr/local/bastille/logs) with a timestamp in case you remove and redeploy a jail of the same name

Hi,

jail_enable="NO" disables jails booting universally.

Is there a parameter in bastillebsd that enables certain jails to disable starting on host boot?

Did I miss something?

Thanks!
/z

The Bastille template system (template.sh) parses template files and executes functionality reflected by sub-commands. *I would like the template system to "eat it's own dog food" by executing template hooks using sub-commands directly (where applicable).

• Submitting this issue as part of my mental TODO list.

The bootstrap sub-command uses the curses interface to display the download progress. Fix it so that it displays the proper header and title.

I think it could be useful to put every jail into separate ZFS tanks.

That should make it much easier to migrate jails (e.g. moving it to a new host) and give the ability to zfs snapshot single jails (if you don't want a full export).

feature request:
adding force option to destroy.sh

Hi @cedwards

First off, thanks for awesome work!

I am considering Bastille for building a production jail host.
I need to delegate a physical network interface to a jail, exactly like in man ifconfig :

   vnet jail
             Move the interface to the jail(8), specified by name or JID.  If
             the jail has a virtual network stack, the interface will
             disappear from the current environment and become visible to the
             jail.

Use case: for security I need to separate management interface (the one that has SSH listening) from public interface (that serves live traffic). Management interface should only be accessible to the jail host, while public interface should be accessible from a jail. At this moment I don't need any integration with pf/ipfw as firewall rules are going to be enforced by AWS security groups.

Is it currently possible to achieve with Bastille?

Thanks.

There needs to be a system to manage system resources as that bastille would be used to build process dense hosts. The system needs to manage items such as CPU time, system memory, network bandwidth, and disk quotas.

FEATURE REQUEST

I would be nice to change UPSTREAM_URL (bootstrap.sh) as config parameter in bastille.conf.
This would give Bastille the functionality for offline installation.

I have a jail which is extremely slow to start with bastille (only has Nginx in it). I have other jails which work fine (and run instantly) within bastille.

I was debugging using sh -x and it hangs on this line:
https://github.com/BastilleBSD/bastille/blob/master/usr/local/share/bastille/start.sh#L76

If I run jail -f /usr/local/bastille/jails/www/jail.conf -c www manually outside bastille it runs instantly.

$ bastille -v
0.4.20190714

$ bastille create b1 12.0-RELEASE 10.1.0.1
$ bastille start b1
[b1]:
jail: /usr/local/bastille/jails/b1/jail.conf line 1: ;: syntax error

$ head -1 /usr/local/bastille/jails/b1/jail.conf
interface = ;

Changing that line to

interface = lo1;

solves the issue.

I discovered that the RC script needs some love. Hopefully I can fix this in the next patch.

Until that time be aware that the auto start/stop may not work as expected.

I am testing Bastille v0.6.20200202 and I like it a lot so far. On comparing my notes for creating jails with what bastille create does, I have noticed the following:

1. Bastille does not set strict permissions on a jail's directory, e.g.:

chmod 700 ${bastille_prefix}/jails/${myjail}

2. Bastille does not disable the adjkerntz job from crontab, as suggested in the FreeBSD Handbook.

3. Bastille does not create a dummy fstab in the jail:

touch ${myjail}/root/etc/fstab

but some programs may expect that file to exist (e.g., see FreeBSD Mastery: Jails, Ch. 2).

Just pointing these to your attention.

Implement a check if you created ip address exists.
Otherwise you will run into errors like:

jail: jailname: /sbin/ifconfig bastille0 inet 10.0.0.1 netmask 255.255.255.255 alias: failed

Could be done by a simple grep on ifconfig (ifconfig | grep ipadress), but maybe there's a more reliable solution

VNET interface is not correctly specified in the configuration files when creating the first jail with bastille.
The problem doesn't occur if other jails already exist.

In create.sh's generate_vnet_jail_conf(),
the variable uniq_epair seems to be empty when there's no jail (jail_list is empty).

Thank you for developing this awesome tool!
I have just started exploring bastille but already like its simple design and handy features.
I'm a big fan of the VNET so I'm going to playing around it with bastille.

Details

The version I used

I use the commit 644b531.

$ sudo bastille -v
0.6.20200202

The first jail

Vnet interface names are incomplete in jail.conf and rc.conf.

$ sudo bastille list jail
$ sudo bastille create --vnet vm1 12.1-RELEASE 192.168.1.240 lagg0
Valid: (192.168.1.240).
Valid: (lagg0).

NAME: vm1.
IP: 192.168.1.240.
INTERFACE: lagg0.
RELEASE: 12.1-RELEASE.

syslogd_flags: -s -> -ss
sendmail_enable: NO -> NONE
cron_flags:  -> -J 60
ifconfig_e0b__name:  -> vnet0
ifconfig_vnet0:  -> inet 192.168.1.240

$ fgrep vnet.interface /vm/bastille/jails/vm1/jail.conf 
  vnet.interface = e0b_;

$ fgrep ifconfig_e0b /vm/bastille/jails/vm1/root/etc/rc.conf
ifconfig_e0b__name="vnet0"

The second jail

No problem is found.

$ sudo bastille list jail
vm1
$ sudo bastille create --vnet vm2 12.1-RELEASE 192.168.1.241 lagg0
Valid: (192.168.1.241).
Valid: (lagg0).

NAME: vm2.
IP: 192.168.1.241.
INTERFACE: lagg0.
RELEASE: 12.1-RELEASE.

syslogd_flags: -s -> -ss
sendmail_enable: NO -> NONE
cron_flags:  -> -J 60
ifconfig_e0b_bastille0_name:  -> vnet0
ifconfig_vnet0:  -> inet 192.168.1.241

$ fgrep vnet.interface /vm/bastille/jails/vm2/jail.conf 
  vnet.interface = e0b_bastille0;

$ fgrep ifconfig_e0b /vm/bastille/jails/vm2/root/etc/rc.conf
ifconfig_e0b_bastille0_name="vnet0"

Would it be possible for each jail to have it's own MAC address ?

Currently, all jails on the same physical interface have the same MAC address.

root@jailbeast:/usr/local/bastille/templates # bastille template md-mail postfix
[md-mail]:
Copying files...
Copy complete.
Installing packages.
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/Latest/pkg.txz: No address record
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/Latest/pkg.txz: No address record
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/Latest/pkg.txz: No address record
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.
Updating services.
sendmail_enable: NONE -> NO
sendmail_submit_enable: YES -> NO
sendmail_outbound_enable: YES -> NO
sendmail_msp_queue_enable: YES -> NO
postfix_enable: -> YES
dovecot_enable: -> YES
Executing final command(s).
chsh: WARNING: shell '/usr/local/bin/zsh' does not exist
pwd_mkdb: warning, unknown root shell
chsh: user information updated
Template Complete.

[rmd-mail]:
Copying files...
Copy complete.
Installing packages.
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...

Hi,

pkg fetch is unbelievably slow even in gigabyte network (~300kb/s) even at this point of writing similar to https://forums.freebsd.org/threads/very-slow-pkg-install.70708/ despite changing the upstream pkg repos to alternatives including pkg0.bme.freebsd.org (EU mirror).

Fetching gsfonts-8.11_8.txz: 100%    4 MiB 169.4kB/s    00:25    
Fetching ghostscript9-agpl-base-9.50.txz: 100%   12 MiB 168.9kB/s    01:13 

However, ports is relatively faster than pkg in the same host and jails, fyi.

Cheers,
/z

Add a double check if bastille0 exists in start.sh.

(not actual code)

if [ ! ifconfig | grep $bastille_jail_interface> /dev/null] than
echo "Interface bastille0 not found. See Docs/Network Requirements" # or something like that

error message if bastille0 doesn't exist:

ifconfig: interface bastille0 does not exist
jail: esfrasltj1: /sbin/ifconfig bastille0 inet 10.0.0.1 netmask 255.255.255.255 alias: failed
jls: jail "jail_test_lo1" not found

Bastille calls "bastille bootstrap" for templates listed in an INCLUDE file. Pretty sure it should be "bastille template" instead?

Bastille 0.6 can't handle IPv6 jails out of the box

bastille -v 
0.6.20200202

Errors when creating IPv6 jails

bastille create v6jail1 12.1-RELEASE ::2
Invalid: (::2).

Error when adding ip6.addr manually to jail.conf

bastille start v6jail1
Error: IP address () already in use.

Suggestion

create.sh

validate_ip() {
    ip4=${IP}
    if expr "$ip4" : '[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*$' >/dev/null; then
    ip6=${IP}
    if expr "$ip6" : 'TODO$' >/dev/null; then

ip6 disable only if ipv6 is not set:

177c177,179
< ip6 = disable;
---
> if  [ -z "$ip6" ]
>     ip6 = disable;
> fi

And change the for loop to add IPv6

start.sh (1st thoughts):

        ipv4=$(grep 'ip4.addr' "${bastille_jailsdir}/${_jail}/jail.conf" | awk '{print $3}' | sed 's/\;//g')
        ipv6=$(grep 'ip6.addr' "${bastille_jailsdir}/${_jail}/jail.conf" | awk '{print $3}' | sed 's/\;//g')
        if [ifconfig | grep -w "$ipv4"] | [ifconfig | grep -w "$ipv6"] >/dev/null; then
            echo -e "${COLOR_RED}Error: IP address ($ipv4$ipv6) already in use.${COLOR_RESET}"
            exit 1
        fi

/usr/sbin/sysrc -f "${bastille_jail_rc_conf}" ip6=enable

Then there's rdr.sh and stop.sh as well (should be similar to the 2 above)...

Add check if zfs tank "bastille" exists during "service start".
Otherwise bootstrap will fail with the following error (parent does not exist)

bastille bootstrap 12.1-RELEASE
cannot create 'zroot/bastille/backups': parent does not exist

Any suggestions where to put it? bastille(.sh)?

if I run the template to a jail more than once instead of the rc.conf getting left alone if something is laready set, it make the value = NONO or YESYES instead of correctly working from the entries inside SYSRC file

First of all I want to say thanks for the great work you are doing. Proper container orchestration is something that should have been done long time ago.

I have a question regarding template re-use. For example I want to base my template on top of another template. Think of "FROM" directive in Dockerfile. Is this possible at the moment?

The update to 0.4.20190623 is packaged as such but internally the version in Bastille still reports a 0.3.x. This is easy to fix, but reporting here for visibility.

Is your feature request related to a problem? Please describe.

bastille destroy currently doesn't delete any active RDR rules so these will end up still active in PF

Describe the solution you'd like

Clean up RDR rules on destroy

Describe alternatives you've considered

N/A

Additional context

Patch attached:

diff --git a/usr/local/share/bastille/destroy.sh b/usr/local/share/bastille/destroy.sh
index c0c1fdd..dc35fed 100644
--- a/usr/local/share/bastille/destroy.sh
+++ b/usr/local/share/bastille/destroy.sh
@@ -80,6 +80,12 @@ destroy_jail() {
             echo -e "${COLOR_GREEN}Note: jail console logs archived.${COLOR_RESET}"
             echo -e "${COLOR_GREEN}${bastille_jail_log}-$(date +%F)${COLOR_RESET}"
         fi
+
+        ## clear any active rdr rules
+        if [ ! -z "$(pfctl -a "rdr/${TARGET}" -Psn 2>/dev/null)" ]; then
+            echo -e "${COLOR_GREEN}Clearing RDR rules:${COLOR_RESET}"
+            pfctl -a "rdr/${TARGET}" -Fn
+        fi
         echo
     fi
 }

(I wrote the original RDR functionality but unfortunately didn't think about this at the time)

Paul

$ bastille -v
0.4.20190714

$ bastille template b1 j0
[b1]:
Template Complete.

The template j0 doesn't exist; it would be nice if bastille either complained (ENOENT) or was silent to indicate non-existence. :)

Solution could be iterating through jails/*/jail.conf and grepping the information.
Might be diffed to jls (-a), showing if jail is active or inactive.

Bastille should, in theory, work on FreeBSD and downstream variants that support jails. This issue is to track verification on the following platforms (and any missing):

• FreeBSD
• HardenedBSD
• OPNsense
• DragonflyBSD
• FreeNAS
• XigmaNAS
• NomadBSD
• FuryBSD
• GhostBSD
• TrueOS

Hello !

I created a new jail "sonarr-devel" to test the new version of this PVR.

I wanted to stop the "classic" sonarr jail but had this behavior :

zutsuu% sudo bastille stop sonarr
Password:
[: sonarr: unexpected operator
[: sonarr: unexpected operator

[sonarr-devel]:
sonarr-devel: removed
zutsuu% sudo bastille list
 JID             IP Address      Hostname                      Path
 jackett         192.168.1.112   jackett                       /usr/local/bastille/jails/jackett/root
 lidarr          192.168.1.115   lidarr                        /usr/local/bastille/jails/lidarr/root
 minecraft       192.168.1.120   minecraft                     /usr/local/bastille/jails/minecraft/root
 radarr          192.168.1.114   radarr                        /usr/local/bastille/jails/radarr/root
 sonarr          192.168.1.113   sonarr                        /usr/local/bastille/jails/sonarr/root
 transmission    192.168.1.111   transmission                  /usr/local/bastille/jails/transmission/root

Here I tried to stop the sonarr jail but instead the sonarr-devel jail got shut down

zutsuu% sudo bastille stop sonarr
[sonarr]:
sonarr: removed

Running bastille stop a second time shut down the sonarr jail, the sonarr-devel is already shut down

zutsuu% sudo bastille start sonarr-devel
[sonarr-devel]:
sonarr-devel: created

Starting seems to work fine tho

zutsuu% sudo bastille destroy sonarr
Jail running.
See 'bastille stop sonarr'.

Can't destroy the sonarr jail because sonarr-devel is running

In ZFS filesystem the command
bastille bootstrap 12.1-RELEASE update
fails to update release with error message

Insecure permissions on /usr/local/bastille  
Try: chmod 0750 /usr/local/bastille

In the Getting Started guide there's a pf.conf which should be copied to /etc/pf.conf.
There are the lines

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if)

which enable NATing from the jails table on the external interface.
This table is never populated, though. One needs to run

pfctl -t jails -T add $some_jail_ip

to have a jail access the external network.

Could this be added to "start.sh"? Or alternatively drop the table line and nat the whole jail subnet?

writing a test issue

update man page and documention for limits

I was able to bastille bootstrap 12.1-RELEASE, but on trying to then create a new jail with that release:

# bastille create mail 12.1-RELEASE 192.168.1.2 lo1
Unknown Release.
Usage: bastille create name release ip | interface.

This looks to be because bootstrapping and creating validate the release name in separate code paths. But doesn't it make sense that if I was able to bootstrap the release in the first place that creating a jail from that release should always work?

I think the release name validation logic should be moved into only occurring on bootstrap, and create should never validate the name at all, just that the requested release is in fact present on disk

(Note: I'm aware that the specific issue of 12.1 not being create-able is addressed in 9bc8837 but the overall issue stands)

i like what's going on with this project so far. i've been thinking about how a user could expand it a little bit to make deployments of bastille containers a little faster. for example i spend a lot of time in different ruby environments, so i use rbenv to manage that part, but rbenv has a few dependencies, and getting a base container up and running it take a little while. i could script the outputs from pkg info -l <pkg_name> for the packages i want to an overlay, but then i'd have to do that for all the packages. is there a better way to do this?