arizvisa / lolfuzz3

navs: lol.

General

│00:42 <@navs> lol.

Installation

First clone the repository for some particular project.

$ git clone https://github.com/arizvisa/lolfuzz3 some-software-name.lol

Python is used for some of the scripts that are required for building. These scripts also require some external python modules which are packaged within the requirements.txt in the root of the repository. This can be used with pip in order to install all of the required modules.

$ python -m pip install -r requirements.txt

To install them local to the current user, one can simply provide the --user option to pip.

$ python -m pip install --user -r requirements.txt

HashiCorp Packer

This software depends on HashiCorp's Packer. This software is written in Go and is used to automate the building of virtual machine templates. For more information, you can visit the "What is Packer?" section at http://packer.io/intro/index.html

To install it, you can either visit http://packer.io/downloads.html to download it, or compile from source as documented here The repository containing HashiCorp's Packer is hosted at GitHub under hashicorp/packer.

The following will also install using go get assuming you have go installed and in your path. The latest version of packer at the time of writing this document is tag v1.4.4.

$ go get -u -v github.com/hashicorp/packer
...
$ git -C $GOPATH/src/github.com/hashicorp/packer checkout v1.4.4
$ go install github.com/hashicorp/packer/...

After downloading or compiling packer, ensure the packer binary somewhere in your path. You can run packer version to confirm it is working.

Post-Processor Plugin for HashiCorp Packer (packer-post-processor-vagrant-vmware-ovf)

To automatically output the resulting templates as a deploying OVF template, one can choose to install this plugin. The plugin itself is hosted on GitHub and can be found under the repository frapposelli/packer-post-processor-vagrant-vmware-ovf.

GNU Make (and some posix tools)

The makefiles within this software use specific features that are only available in GNU's flavor of make which is hosted at ftp://ftp.gnu.org/pub/make. Some other posix utilities are used by this software such as bash and printf which must be in the path. On some platforms, you may find GNU's Make is named gmake. You can check which name for make to use by passing the --version option to make or gmake.

$ make --version

If on the Windows platform, these tools can be provided by Msys2, the MingW/MSYS build environment, the Cygwin Project, or Microsoft's Windows Subsystem for Linux (WSL) that is available with Windows 10. It is, however, recommended by the author to use Msys2 as it is GNU compliant, and is typically faster due to linking with Microsoft's C Runtime instead of trying to simulate a full-fledged posix environment.

When using the software on the Windows platform, in all actuality a full-fledged posix environment is not necessary and you should be fine just having these tools in your path. You will be calling make directly, so its name also doesn't really matter.

jq (Command-line JSON processor)

JSON is used in numerous places and as a result, the jq tool is required to be in your path. This tool exposes the ability to query and transform JSON at the command-line. For more information, the homepage for jq is located at https://stedolan.github.io/jq,

In order to install jq, it is likely the tool is available via your package manager if you're using a Linux distribution. If it is not available in your package manager, or you're using Windows or another alternative platform, a list of already built binaries can be found at https://stedolan.github.io/jq/download. After downloading it, simply ensure that it is in your path by trying jq --version.

OpenSSH (ssh-keygen)

The SSH protocol is used to communicate with a master template and thus in order to facilitate authentication to a deployed template, an SSH client such as OpenSSH or PuTTY is required.

One part of OpenSSH which is required to build a master box is the ssh-keygen binary which comes with it. The ssh-keygen tool is used during building in order to pre-generate a public/private keypair which is used to authenticate to a master. These generated keys are then to be committed into the repository after the template has been deployed so that other users may have access to the master hosts available in the project.

On the Windows platform, a binary for ssh-keygen may be hard to come by. However, a GitHub project has been created under PowerShell/Win32-OpenSSH which may be used to get just the binary. Under the releases page found at https://github.com/PowerShell/Win32-OpenSSH/releases, one can download an archive of the release and extract just the ssh-keygen binary. More information about this can be found at the repository's wiki page found at https://github.com/PowerShell/Win32-OpenSSH/wiki

Once ssh-keygen has finally been made available, simply ensure that it is in your path. This is the only OpenSSH binary required for building templates.

OpenSSL (Cryptography and SSL/TLS Toolkit)

In order to facilitate authorization and authentication, OpenSSL is needed. This is used to generate public/private keys as well as calculate and verify signatures in a number of places. On posix environments, this tool is likely already installed. If not, it more than likely can be found in your distribution's package manager. Nonetheless, the source can be downloaded at https://www.openssl.org/source to build it.

If on the Windows platform, some urls to binaries that have been built by others can be found on the wiki page at https://wiki.openssl.org/index.php/Binaries. Once openssl has been made available through some means, simply ensure that the openssl binary is in your path.

VMware's OVFTool

If one wants to convert a VM built with HashiCorp's Packer into a packaged template in order to simplify deployment into VMware's platforms, VMware provides a tool to do this.

This is ovftool, and can be located at VMware's developer support page which is found at https://www.vmware.com/support/developer/ovf. Once the virtual machine has been built, ovftool can be used to convert it into different formats or to even deploy it against VMware's virtualization platform. An example of converting a virtual machine into an OVF template is as follows.

$ ovftool --compress=9 /path/to/machine/machine-name.vmx /target/template/path/machine-name.ovf

This will take a moment to compress the disks associated with the machine, and archive it into a few files. The manifest file which contains signatures for all files associated with the machine will be named machine-name.mf, along with all of the hard disks which will be prefixed with machine-name. The machine-name.ovf file can then be used to deploy the machine into your virtualization infrastructure.

lolfuzz3's Issues

Consolidating all three of the master's environments, "master", "bootstrap", and "master-bootstrap"

When building the SaltStack cluster on a master, the "master", "bootstrap", and "master-bootstrap" (for masterless) environments are used to ensure each member is provisioned properly to work with the cluster. This poses an issue as the target is written using the "base" environment and so these two environments need to be combined in some way. Although these three environments should never change except when updating the non-project-specific components, the way that they're presently organized makes it difficult to process and update independantly from the target's states that reside in the "base" environment.

If the combination of these is a possibility, then we could always have templates running with the "master" role apply the "base" environment for any target-specific provisioning which will remove the need for the user to manually apply their target-specific provisioning on deployment.

It might actually be possible to use the "master" environment only during bootstrap which will then switch its default environment to "base" similar to how the "bootstrap" environment works. After applying the "master" environment, the master's minion can update its environment and then cycle its service, so that it's completely switched into the "base" environment. This way once the "master" environment has been properly provisioned, it can then apply any target-specific provisioning. This switching of the environment should only be temporary so that the "master" environment is _always_ applied on first boot.

Upgrading Container Linux to Fedora CoreOS and deprecation of the `open-vm-tools` container

The master.project template is currently still using the original CoreOS which is deprecated. There's a couple of requirements on CoreOS such as rkt, and a few of the other things like an older ignition file. These will all need to be updated. Containers will also need to be transitioned to OCI containers that are started with Podman.

On the brightside, I hear there's a package manager in Fedora's CoreOS which means we might be able to install things like Buildah and such within the root filesystem layer.

Migrate from Container Linux to Fedora CoreOS

Container Linux was purchased by RedHat a little bit ago. As a result, CoreOS has had some significant changes and has been renamed to Fedora CoreOS. Rkt has been deprecated in favor of cri-o which is based on systemd. Packages can now be installed using rpm-ostree. Ignition has been updated, etc.

All of the container builders need to be re-designed to emit oci containers, and some of the management scripts will need to be updated so that they use podman, buildah, and friends.

Bootstrapping the etcd.target service on master.project and cleaning up the systemd startup process

The way the etcd.target service is bootstrapped is done at the incorrect time when building master.project. It's actually happening as part of the "master-bootstrap" environment when provisioning the template with salt-masterless at build-time, when really it should be happening at deploy-time. If issue #8 gets solved, then it would be significantly better to somehow do this as part of the process of provisioning with the "master" environment.

If at all possible, the "master" environment could detect other members of the cluster and use that to identify whether it's responsible for bootstrapping the instance of etcd. If this ends up being a valid solution, then we can also use the same logic to automatically deploy a distributed MinIO cluster with the store state, as well as one for Kafka too with the queue state. If absolutely all of this works properly, then we could even deploy k8s by default and simplify our container management too. However, this might still be clumsy, because we definitely _need_ the etcd instance to be running in order to manage job results returned by the "master" environment.

At the moment the usage of Terraform allows us to hack around this, but Terraform is not yet completely integrated into this project because of the immaturity of the terraform-provider-vsphere plugin.

Emitting the packer templates used to build each machine type

When choosing to build a template with this project, the packer template that's generated should be consolidated from all of its parts and written into the output directory. Despite already having a template for a particular virtualization platform, emitting the packer template in JSON would then allow someone to rebuild the template exactly.

Emitting the default SaltStack states and pillar of the master.project template as output

At the moment, the default states/pillar for the master box is hardcoded as part of the template build process. The provisioning steps are then responsible for deploying these states, and bootstrapping the box. As issue #5 aims to also output the JSON template that's generated, the default states/pillar for SaltStack will also need to be emitted in order to provision the box that the user is trying to deploy.