aztechian / bridgr Goto Github PK

Current implementation will fail when the distributed binary is run, because it will look for templates/yum.repo, and that isn't distributed with the app.

Use https://github.com/shurcooL/vfsgen (alternative choice is https://github.com/UnnoTed/fileb0x)

As a user, I would like to not have to get another piece of software for serving the artifacts after transferring across the air-gap. If bridgr could also act as an HTTP server in an alternate mode, it would make the transfer/setup easier.

I do not want to add a lot of work making Bridgr do all kinds of fancy HTTP tricks. If you want special redirections, SSL certificates, domain name hijinks, etc then users should be deploying their own HTTP server. Bridgr will support a simple, static HTTP server only.

As a user, I want the items I specify in the configuration file to be downloaded and repositories created for me.

Bridgr will use docker containers to encapsulate the knowledge of creating various repository types - YUM, Python, Gems, etc. Bridgr will act as a Docker client application to the docker daemon, and needs to programmatically tell the daemon to create and start containers with certain configuration options (bind mounts, entrypoints, script commands, etc).

Implement a docker client to manage a container lifecycle - pull, start, stop and remove. The best place (as of now) is probably in the YUM worker since its the first worker in existence that needs to do this. But, a design pattern will be needed for all other worker types.

This issue was generated by todo based on a TODO comment in 87aa9f9 when #7 was merged. cc @aztechian.

As a user of Bridgr, I want to make taking artifacts with me (to other projects, or across an air-gap) easier. By having a single file, this is easier than taking many file in a filesystem layout.

This could be a bad idea if the user wants to make a zip file of, say, 1TB worth of artifacts. However, I think we should implement it and leave it to the user to decide if it is proper for their use case.

GOARCH=arm and a new matrix setup in travis CI

This issue was generated by todo based on a TODO comment in 42a84a7 when #22 was merged. cc @aztechian.

As a user of Bridgr, I would like to take my specified docker images across the air-gap to be hosted in an internal registry service. Currently, I would need to take the individual .tar image files and load them into a registry before use. I would like to be able to simply point a registry container at the storage location and have it serve the images through the registry.

Before fixing #49 , the error message just said, "reference does not exist", but didn't say what image that was from. Improve the error message to show the image in question.

It may be necessary for some git repos to specify the checkout of a specific commit. Bridgr should support this either with overloading of the branch option or adding a commit option to the long-form config object.

Reorganize for better clarity, eases re-usability.

Might be possible to use a different package, even.

https://gist.github.com/TheHippo/7e4d9ec4b7ed4c0d7a39839e6800cc16

There is a file in place already, but little work has been done on it, and no app code yet uses it.

• Fill out log.go with appropriate methods. Read this first and really think if we need more than "info" and "debug" levels.
• Find a way to get the -debug runtime flag through to the Log struct.
• Create unit tests
• Move existing code to use new logger

For a YUM section like so:

yum:
  packages:
    - libselinux-python
    - docker
    - chrony

The following error is thrown on version 1.2.0:

goroutine 1 [running]:
bridgr/internal/app/bridgr/config.parseYum(0x16965e0, 0xc00030d110, 0xc0002e0730, 0x5, 0x5, 0x0, 0x0, 0x16965e0, 0xc00030d140, 0x0, ...)
	/home/travis/gopath/src/github.com/aztechian/bridgr/internal/app/bridgr/config/yum.go:45 +0x59e
bridgr/internal/app/bridgr/config.New(0x185cf80, 0xc000096940, 0x0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/aztechian/bridgr/internal/app/bridgr/config/config.go:62 +0x322
main.main()
	/home/travis/gopath/src/github.com/aztechian/bridgr/cmd/bridgr/main.go:65 +0xf8

This form should still be an acceptable config usage

This issue was generated by todo based on a TODO comment in 42a84a7 when #22 was merged. cc @aztechian.

https://goreportcard.com/report/github.com/aztechian/bridgr

This issue was generated by todo based on a TODO comment in dd0ccce. It's been assigned to @aztechian because they committed the code.

This issue was generated by todo based on a TODO comment in 5c94bbb when #43 was merged. cc @aztechian.

As a user of Bridger, when running in a CI server environment I would like to fail my pipeline if any of the workers types return an error status. This will allow me to stop my pipeline and be aware that some files I expect to be present are not.

This may filter down to the individual workers as well, as most of them probably print the error, but swallow the object and keep trying to get as much as they can.

This issue was generated by todo based on a TODO comment in 87aa9f9 when #7 was merged. cc @aztechian.

When running bridgr in Hosting mode, it would be useful to print out some info for sysadmins on how to configure other hosts to use the repositories provided by bridgr.

Ie,

Welcome to Bridgr.

============ YUM =============
Place this in /etc/yum.repos.d/bridgr.repo:
  [bridgr]
  name = Bridgr Yum Repo 
  baseurl = http://<ip>:8080/yum/$releasever/$basearch
  enabled = 1
  gpgcheck = 0
==============================

This should probably come from a template embedded in the app. See the assets content for example.

As a user executing bridgr, when I run bridgr files I only expect a packages/files directory to be created. Instead, I see subdirectories for all worker types which I did not ask for.

Line 62: warning: "verison" is a misspelling of "version" (misspell)
Line 74: warning: "recieved" is a misspelling of "received" (misspell)

This issue was generated by todo based on a TODO comment in c4953b6 when #89 was merged. cc @aztechian.

https://github.com/briandowns/spinner

with prefix for what worker type is running. Use stderr for spinner status.
Probably only do this for default (non-verbose) execution. Verbose mode just dumps lots of text to stdout.

Use environment variables for getting the credentials. Dynamic variable names to identify target domain:
BRIDGR_DOCKER_IO_USER
BRIDGR_DOCKER_IO_PASS
or maybe, in cases where concatenated credentials are ok
BRIDGR_DOCKER_IO

This issue was generated by todo based on a TODO comment in cee85af. It's been assigned to @aztechian because they committed the code.

If files in an S3 bucket could be accessed with Access ID and Secret Key, that would be nice to had.

The docker worker does not call Setup() before trying to write out images

As an end user, I don't want to see bridgr try (and maybe fail) to run workers for repository types that I haven't defined in my config.

For example, I don't have a ruby section in my config file, but bridgr still runs the ruby worker with an empty config and fails:

2019/09/16 13:20:32 Processing Ruby...
2019/09/16 13:20:40 Error processing Ruby: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory": unknown

It also creates an empty packages/ruby directory

Config struct and worker

Bridgr is stripping the docker.io/ off image tags despite it being explicitly stated in my manifest. For my particular application, I need the leading repository preserved.

add a filesytem abstraction to bridgr for better testing os calls.

This comment was generated by todo based on a TODO comment in 42a84a7 in #22. cc @aztechian.

Originally posted by @todo in #22 (comment)

As a user of bridgr, I would like to be able to verify that the binary I download has not been tainted or compromised between CI build, publishing, download and usage.
A checksum file will allow me to have confidence that Bridgr is what I expect it to be.

Modify the .travis.ci file to create and publish a .asc (or .sha256) for each built binary in the pipeline. These should be published to github releases on tagging (right along with the binaries themselves).

https://helm.sh/docs/chart_repository/

I was recently running a multibranch pipeline where two branches were running simultaneously. The yum docker container failed silently because the name was taken by the other running branch.
13:52:43 2019/11/25 13:53:17 Error processing Yum: Error response from daemon: Conflict. The container name "/bridgr_yum" is already in use by container "04045dbc895e064f089df450d147e0358a2ac504f93ca9bdc50d17a9dcdc456b". You have to remove (or rename) that container to be able to reuse that name.
If allowing simultaneous executions on the same host is ok, can bridgr exit with an error code in this use case to flunk a build?

Need to handle both user/password and token for github/gitlab.

Example github token:
https://github.com/src-d/go-git/blob/master/_examples/clone/auth/basic/access_token/main.go

Need to confirm that GitLab follows the same form of token auth as github.

Following workers downloading the artifacts, the checksum output should be run against all files from the workers, creating a checksum.sha256 file in each worker directory. Then, the top-level packages directory gets a checksum.sha256 containing entries for all of the worker checksum files.

This allows easy validation once the artifacts are transferred to know that everything has been successfully moved across the air gap.

This only works on a single subcommand right now. Allow this to work on an array of subcommands.

This comment was generated by todo based on a TODO comment in 42a84a7 in #22. cc @aztechian.

Originally posted by @todo in #22 (comment)

config and worker

look at using https://github.com/pkg/sftp

That is doing SFTP (ie, FTP over SSH). Need to determine if it's useful to also support FTPS and plain FTP.

Now that there is a decent collection of supported types in Bridgr, it may be better to have sub-modules for each type. For example

worker.Python -> worker.python.Python

Some expertise or research is needed here for best practices.

As a bridgr user, I want to have git repositories be part of my set of artifacts.

A bare repository is probably the best starting point for this, as it should also be hostable through a static web server. However, it makes sense to also support cloning repo working copy with a config file option.

Add auth support (HTTP user/password) for file types. Really, at this point, just HTTP(s) files.

FTP support would be completely different, and FTP isn't even yet implemented. That will be a separate ticket.

https://github.com/containers/image

would seem to indicate that it can work without a docker daemon - which would allow use in more places. However, we have a deep dependency on docker daemon for runtime with most other workers, so it may be moot.

This issue was generated by todo based on a TODO comment in 42a84a7 when #22 was merged. cc @aztechian.

yumdownloader that is used in the docker container doesn't know how to resolve URLs to packages (instead of package names). So, we need to filter these out from the list that's appended to the yumdownloader command.

Then, we need to use curl to download these packages before calling createrepo.

This could be done in the config parsing or in the template.