Coder Social home page Coder Social logo

samba-in-kubernetes / sambacc Goto Github PK

View Code? Open in Web Editor NEW
11.0 9.0 15.0 425 KB

A library and CLI tools for managing Samba servers in containers.

License: GNU General Public License v3.0

Python 97.72% Shell 1.94% Dockerfile 0.17% Makefile 0.16%

sambacc's Introduction

sambacc - A Samba Container Configuration Tool

About

The sambacc project aims to consolidate and coordinate configuration of Samba, and related components, when running in a container. The configuration of one or many server instances can be managed by the tool with the use of configuration files. These configuration files act as a superset of the well-known smb.conf, to configure Samba, as well as other low level details of the container environment.

Rationale

Samba is a powerful and unique tool for implementing the SMB protocol and a software stack to support it on unix-like systems. However, it's potentially challenging to set up and manage many instances of Samba by-hand, especially when running under a container orchestration system.

The idea behind sambacc is to automate much of the low level steps needed to set up samba daemons, users, groups, and other supporting components. The tool is also designed to consume configuration files that can be used across many container instances. sambacc is written in Python as samba provides Python bindings for some of the aspects of samba we need to control.

The sambacc library and samba-container CLI command are used by the samba-container project as part of the server container images.

Usage

File Server

The samba-container command is used to manage features related to the Samba file server and closely related components.

Without any additional arguments samba-container prints the synthesised samba (smb.conf) configuration based on the environment variables:

  • SAMBACC_CONFIG - configuration file(s)
  • SAMBA_CONTAINER_ID - Identity of this instance

Additionally, there are many other subcommands the tool supports. These include:

  • samba-container import - Import smb.conf-style settings into registry
  • samba-container import-users - Import users into /etc files and smb passdb
  • samba-container init - Initialize the container environment for use by samba services
  • samba-container run <service> - Initialize and run a named samba service

For a complete description of the subcommands supported, run:

samba-container --help

Active Directory Domain Controller

The samba-dc-container command is used to manage features related to the Samba AD DC server and related components.

Currently, samba-dc-container supports one subcommand. The run subcommand is used to start an AD DC server. This command supports various setup steps including steps to join an existing domain, provision a new domain, populate a domain with stock users/groups, etc.

For a complete description of the subcommands supported, run:

samba-dc-container --help

Features

  • Abstracts away some of the nitty-gritty details about what Samba expects in its environment
  • Imports specific smb.conf settings from "site wide" configuration files.
  • Imports users and groups
  • Starts smbd with container friendly settings
  • Starts winbindd with container friendly settings
  • Support for joining AD
  • Support for managing CTDB clustering
  • Support for creating/joining Samba Active Directory servers

Major TODOs

A lot. Various things that are missing include:

  • Features to perform more secure (password-less) domain joins
  • Better integration (as opposed to unit) testing
  • Better use of APIs versus executing CLI commands

Contributions and feedback would be very much appreciated.

Install

The sambacc library, samba-container command, and samba-dc-container are written assuming the software is being run within an OCI container environment. While there's nothing stopping you from trying to install it on something else the value of doing that will be rather limited.

The samba-container project includes sambacc and samba packages. If you are looking to use sambacc and not contribute to it, that's probably what you want.

Builds of sambacc are continuously produced within our COPR repository. These builds are then consumed by the container image builds.

Otherwise, the only method of install is from source control.

  • Clone the repo: git clone https://github.com/samba-in-kubernetes/sambacc
  • cd sambacc
  • Install locally: python -m pip install --user .

The test & build container may also be used to build source tarballs and wheels. Then you can distribute and install from the wheel if you need to.

Testing

Local testing

To run the entire unit test suite locally install tox and run tox in the repo root.

Because of the library and tooling that interacts with samba has some system level dependencies, not all tests can be run locally in isolated (virtualenv) environments.

Containerized testing

A more robust and isolated testing environment is provided in the form of the sambacc container image.

The container file and other sources are available at ./tests/container in the sambacc repo. This is the canonical way to run the test suite and is what is used by the CI tests. When run this way certain system packages can be installed, etc. to support running a wider range of test cases.

By default the container image is configured to check out sambacc master branch and execute the tests and build python source distributions, wheels, and RPM packages. You can test your local git checkout using the image by mounting it at /var/tmp/build/sambacc (example: podman run -v $PWD:/var/tmp/build/sambacc sambacc:ci).

To access the packages that are built using the container, mount a directory into the container at "/srv/dist" and set the environment variable SAMBACC_DISTNAME to a term of your choice (example: "latest"). This will then save the builds in a directory of that name in your output directory. Example:

$ mkdir -p $HOME/tmp/sambacc
$ podman run --rm \
  -v $HOME/tmp/sambacc:/srv/dist -e SAMBACC_DISTNAME=latest \
  quay.io/samba.org/sambacc:latest
$ ls $HOME/tmp/sambacc
latest
$ ls $HOME/tmp/sambacc/latest
sambacc-0.1.dev225+g10059ff-py3-none-any.whl  sha512sums
sambacc-0.1.dev225+g10059ff.tar.gz

You can combine the source directory mount and distribution directory mount in one command to produce builds for your own local development work if needed.

License

GPLv3 as per the COPYING file.

This is the same license as used by Samba.

Contributing/Contact

Patches, issues, comments, and questions are welcome.

Resources:

sambacc's People

Contributors

anoopcs9 avatar myback avatar obnoxxx avatar phlogistonjohn avatar spuiuk avatar synarete avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sambacc's Issues

Use logging in sambacc

Currently sambacc mainly uses python print calls. The ctdb work brings in logging module support. Convert more call sites of print to use logging when appropriate.

I've started doing this in newer code but it should be done more thoroughly in the codebase.

sambacc fails to do any data validation on json config

If there are any mistakes in the formatting of the json config, sambacc crashes when trying to access missing elements.
It would be better if sambacc did some sort of data validation early on instead, and printed a friendlier error.

Add support for more secure non-interactive domain join

Currently we support "insecure" domain joins that pass a username and password on the CLI or env vars. This is convenient for throwaway ADs on a trusted network, but not so great for more production-like ADs.
Unlike #4 this allows for putting trusted information somewhere the tooling can access it directly, but in a more secured manner. I'm primarily thinking of something like a k8s secret mounted into the container as a file.

Support conditional behavior for differing samba versions

Specifically samba 4.15 has made changes to some of the command line options. As noted in a samba-container issue the option --log-stdout has been changed to --debug-output.

I'm proposing we add a mechanism for handling small behavior changes across samba versions like this.

Currently I'm thinking we can add an environment variable that indicates the various behaviors. I'm thinking something like SAMBA_FEATURES and our initial value can be SAMBA_FEATURES=cli_debug_output. In the future other behaviors could be tracked like SAMBA_FEATURES=cli_debug_output,foo_bar,something_cool.

tox test warning for pytest in py3 env

PR #54 attempts to fix the warning:

WARNING: test command found but not installed in testenv
  cmd: /usr/bin/mypy
  env: /var/tmp/build/sambacc/.tox/py3-mypy

However, even after that is merged, there will still be a warning for py.test:

WARNING: test command found but not installed in testenv
  cmd: /usr/bin/py.test
  env: /var/tmp/build/sambacc/.tox/py3

While working on the warnings I found I couldn't resolve the latter within a few hours of work.
I'm partly creating this issue to capture some of the dead-ends and possible future paths.

The issue occurs because tox wants to use the python tools brought in by pip to the virtualenvs that get created for each environment. As the tox docs note the use of sitepackages=true can trigger this issue. PR #54 narrows down the incidences to only one - the py3 environment. We want sitepackages because we want to run some tests with things like samba's python modules present. However, these are C modules and are typically brought in via system packages. [1]

I tried one workaround which almost solved the problem, I configured all envs as sitepackages=false
and then for py3 only - I added a commands_pre to change the virtualenv to allow system site packages. Unfortunately, the pip command runs before commands_pre and as long as we specify the names of packages we rely on (inotify_simple, pyxattr). They'll still fail to install.

The last thing I can think of, which I don't want to implement at this time is a hack that will explicitly symlink in certain dependencies from the system without setting the virtualenvs to allow system site packages. At least for the container environment this wouldn't be too bad as we know exactly which RPMs match up with what python deps. I can imagine writing a small wrapper script to call from install_command that passes the deps to pip in most circumstances, but when enabled links in files listed via rpm -ql found in /usr/lib/python/site-packages and /usr/lib64/python/site-packages to the virtualenv.

[1] - xattr module too, but this at least has a package on pypi that pip could find. On the downside it would require a compiler and I would rather not need that to run our tests.

Revisit schemacheck tox env workaround

PR #98 added a MULTIDICT_NO_EXTENSIONS=1 env var workaround to get the fedora 39 based ci job working again. The issue is possibly due to:
aio-libs/multidict#887

But I have some doubts about that since Python 3.12 was already in Fedora 39 (I think?).

Regardless, revisit the workaround once the issue above has been resolved and there are python 3.12 wheels available for multidict.

Provisional - Issue assumes PR #98 will be merged.

Add support for interactive domain join

The sambacc tool should be able to support more "traditional" domain joins if the user is unable or unwilling to put the password for the joining user in a secret. In that case the user should be able to log into a container and run a command to join rather than it happen automatically in container init.

commit 89477bd prevents checkout of branch

Commit
89477bd Git checkout to a branch to avoid detached HEAD
adds the following change

-        git checkout "${node}"
+        git checkout -b "${node}" "${node}"

to tests/container/build.sh.

This is however problematic when we would like to build off a branch.
eg: When building the container off the latest commit, we would like to the master branch. This results in the following command
git checkout -b "master" "master"
which fails with
fatal: A branch named 'master' already exists.

The problem was seen when working on the samba-container PR
samba-in-kubernetes/samba-container#52

Add sambacc version to python wheel created

Add an identifier which can be used to identify the git commit used to build samba cc within the server image. This can help with debugging when trying to identify a problem in the samba image.

[ctdb/ss] ltdbtool fails to run when samba debugging is enabled

Currently we can pass samba debug level through the operator via the config param "samba-debug-level" (env var SAMBA_OP_SAMBA_DEBUG_LEVEL).
However, setting this causes ltdbtool to fail. Apparently, unlike most of the samba commands this one doesn't understand the typical --debuglevel option.

Tested with:

  • samba-operator: git: d96dcdb17f0fd8a8c1fd22d446691078e4ac3466+
  • samba-container: container: quay.io/samba.org/samba-server@sha256:e85f537555d76771e7e01dfc0795a80c6472a85c36a7e072bafa580cf20f6425
$ kubectl logs cshare1-0 -c ctdb-migrate
2021-11-02 20:25:03,743: INFO: Checking for /var/lib/samba/account_policy.tdb
2021-11-02 20:25:03,743: INFO: Converting /var/lib/samba/account_policy.tdb to /var/lib/ctdb/persistent/account_policy.tdb.0 ...
ltdbtool: invalid option -- '-'
Usage: ltdbtool dump [-e] [-p] [-s{0|32|64}] <idb>
       ltdbtool convert [-e] [-s{0|32|64}] [-o{0|32|64}] <idb> <odb>
       ltdbtool {help|-h}
Traceback (most recent call last):
  File "/usr/local/bin/samba-container", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/sambacc/commands/main.py", line 211, in main
    cfunc(CommandContext(cli))
  File "/usr/local/lib/python3.9/site-packages/sambacc/commands/ctdb.py", line 161, in ctdb_migrate
    ctdb.migrate_tdb(ctx.instance_config, ctx.cli.dest_dir)
  File "/usr/local/lib/python3.9/site-packages/sambacc/ctdb.py", line 465, in migrate_tdb
    _convert_tdb_file(tdb_path, dest_dir, pnn=pnn)
  File "/usr/local/lib/python3.9/site-packages/sambacc/ctdb.py", line 482, in _convert_tdb_file
    subprocess.check_call(list(cmd))
  File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ltdbtool', '--debuglevel=10', 'convert', '-s0', '/var/lib/samba/account_policy.tdb', '/var/lib/ctdb/persistent/account_policy.tdb.0']' returned non-zero exit status 255.

Improve timing behavior in sambacc

Improve timing behavior in sambacc. Currently, sambacc support two kinds of waiting: waiting on file changes or waiting by predefined sleeps. The file based wait depends on inotify and only works on a local file system. File wait works best but not enough situations so we should probably make time sleeps less hacky with sane timeouts.

Update formatting to match recent black version(s)

Recently, local tox runs started finding black formatting issues that were not flagged before, most likely due to new release of the black tool.
There's a workaround present in PR #98 that avoids taking the newest versions. It would be good to remove this workaround and update the formatting (as long at it makes sense).

This issue is a reminder to get that done.

samba-container needs a mechanism for updating running configurations

There should be a samba-container command that fetches the state of the sambacc configuration and applies it to any running instances. This is needed to support adding a new share to an existing smbd, for example.

This command should have a "server" / watch-and-react mode too.

Import configuration with libsmbconf API

Currently, when sambacc imports samba configuration from it's own JSON based fomat it uses the net conf import command line. Future versions of Samba will have a python binding for the smbconf library. This library can be used to read a text based config (smb.conf) or read/write registry based configuration. This would allow us to use the library directly rather than generating a text based configuration and shelling out to net command to update the registry.

It may also make post-init configuration updates simpler and easier to see exactly what is changing.

setup steps can become unwieldy when manually managing containers

sambacc provides a "setup steps" feature for some commands (mainly run) in which you can pass to have sambacc perform and explict set of setup steps before launching the main command (starting a long running process like smbd,winbind).

An example from a recent task did: run --setup=config --setup=users --setup=users_passdb --setup=nsswitch smbd
While it's nice that the setups steps are granular enough to provide a fine level of control over how sambacc sets up the environment its a bit to remember and type.
This was not a problem when samba-container images are mostly getting run via a higher level of orchestration like the samba-operator. However, when working in a low-orchestration environment this can be annoying

This is is a "I'm not commited to this, but let's think about it" to consider some sort of higher level way to invoke the setup steps - probably through the config file.

Investigate ad-dc provisioning with tdb acl backend

As per a discussion at sambaXP - they may be a way to enable the ad-dc to provision without the need for having acl xattrs. This would be beneficial for samba-container AD DC image as it means we could potentially run without being a privileged container.

The must-join command should retry

Currently, the must-join subcommand will attempt to use automated join methods once, and then wait forever to have a human interactively perform a join. If the command has good inputs, but a temporary issue blocked it from joining a human still has to intervene. Instead, we should periodically retry the given join params in case the failures were indeed transient. It should continue to support an interactive join.

Add ability to apply file system directory quotas on share dirs

When a share is exposed to samba there may be a need to limit the amount of data to samba. Ideally, this would be done at a layer outside of the samba container. However, there are cases where we are provided a single data volume and shares are directories within that volume. The sambacc library already has the ability to create the share directories and apply permissions to them.

The sambacc json configuration can provide a suggested quota value and parameter indicating kind of file system is in use. Unfortunately, this may be needed because unlike creating directories and applying permissions the tools/calls needed to set a directory or tree quota varies by filesystem. Not all file systems support directory quotas either.

addc: support alternate method for supplying the admin password

Currently, the only place sambacc sources the admin password from is in the sambacc config file. This means if you want to keep the password more secure you have to put the entire config file in a secret (or equivalent). This is inconvient if you want to keep your sambacc configurations together in one place.

We could have sambacc source the password from another file and/or an environment variable. Off the top of my head I know that both podman and k8s have methods for passing secret data via those means.

This is not an issue for domain member servers because those already take the join user/pass in a separate file from the configuration.

limit retries when acquring rados lock

Prevent the process from hanging indefinitely on acquiring the lock.


Maybe not too crucial but can be limit retries here instead of trying indefinitely?

          Maybe not too crucial but can be limit retries here instead of  trying indefinitely?

I think if the condition that causes the ObjectBusy exception persists indefinitely it could lead to a hung process that never completes.

        self, name: str, cookie: str, *, delay: int = 1, max_retries: int = 10

Originally posted by @avanthakkar in #123 (comment)

SAMBA_DEBUG_LEVEL and SAMBA_SPECIFICS environment variables don't work as expected

Failed to run a container with debug env var

~ podman run --rm -ti --privileged -e SAMBA_DEBUG_LEVEL=3 quay.io/samba.org/samba-ad-server
...
samba-tool: no such subcommand: --debuglevel=3

Ignore SAMBA_SPECIFICS=daemon_cli_debug_output env var for /usr/sbin/samba, but samba has this option

def samba_dc_foreground() -> SambaCommand:
    return samba_dc["--foreground"]

samba_cmds.CommandArgs.cmd_prefix doesn't work as expected.

Issue created to track any possible regression in the use of the cmd_prefix value.

Was testing with code in
https://github.com/spuiuk/sambacc/tree/ctdb_check-2022021002
This adds 2 new patches.

  1. Introduce ctdb-nodestatus check. This attempts to use the samba_cmds.CommandArgs.cmd_prefix to make it easier for use to write a unit test for this function.
  2. The unit test for the function.

However cmd_prefix is passed as the first argument for the exec call instead of being pre-pended to the test script as we expected. This results in the test failing.

Running tox fails while running the new test added with the following errors.

========================================================================== FAILURES ==========================================================================
_______________________________________________________________________ test_check_ok ________________________________________________________________________

tmp_path = PosixPath('/tmp/pytest-of-sprabhu/pytest-20/test_check_ok0')

    def test_check_ok(tmp_path):
        import os
    
        datapath = tmp_path / "_ctdb"
        datapath.mkdir()
    
        fake_ctdb = [
            "#!/bin/sh",
            'if [ "$1$TESTFAIL" == "nodestatus" ]',
            "then exit 0;",
            "else exit 1;",
            "fi",
        ]
        fake_ctdb_script = datapath / "ctdb"
        with open(fake_ctdb_script, "w") as fh:
            fh.write("\n".join(fake_ctdb))
            fh.write("\n")
        os.chmod(fake_ctdb_script, 0o755)
    
        test_cmd_prefix = [datapath]
        # simulate nodestatus == OK
        pid = os.fork()
        if pid == 0:
>           ctdb.check_nodestatus(cmd_prefix=test_cmd_prefix)

tests/test_ctdb.py:591: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sambacc/ctdb.py:490: in check_nodestatus
    samba_cmds.execute(cmd)
sambacc/samba_cmds.py:175: in execute
    os.execvp(cmd.name, cmd.argv())  # pragma: no cover
/usr/lib64/python3.9/os.py:574: in execvp
    _execvpe(file, args)
/usr/lib64/python3.9/os.py:616: in _execvpe
    raise last_exc
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

file = b'ctdb', args = [PosixPath('/tmp/pytest-of-sprabhu/pytest-20/test_check_ok0/_ctdb'), 'ctdb', 'nodestatus']
env = environ({'TOX_WORK_DIR': '/data/sprabhu/dev/ocs/sambacc/.tox', 'PATH': '/data/sprabhu/dev/ocs/sambacc/.tox/py39/bin:/u...TAFILE': '/data/sprabhu/dev/ocs/sambacc/.coverage', 'PYTEST_CURRENT_TEST': 'tests/test_ctdb.py::test_check_ok (call)'})

    def _execvpe(file, args, env=None):
        if env is not None:
            exec_func = execve
            argrest = (args, env)
        else:
            exec_func = execv
            argrest = (args,)
            env = environ
    
        if path.dirname(file):
            exec_func(file, *argrest)
            return
        saved_exc = None
        path_list = get_exec_path(env)
        if name != 'nt':
            file = fsencode(file)
            path_list = map(fsencode, path_list)
        for dir in path_list:
            fullname = path.join(dir, file)
            try:
>               exec_func(fullname, *argrest)
E               FileNotFoundError: [Errno 2] No such file or directory

/usr/lib64/python3.9/os.py:607: FileNotFoundError

default 'run' setup tasks don't include path ensure func

This means that either you need to list out all the stock setup steps manually (like: --setup=config --setup=users --setup=users_passdb --setup=nsswitch --setup=share_paths) or you don't otherwise automatically get path ensurance/permissions update.

Currently, the setup tasks are a bit of a jumble anyway so we may want to clean it up in general too.

The _daemon_stdout_opt function adds invalid options

Tools smbd and winbindd does not have options --log-stdout or --stdout

~ smbd --log-stdout
Invalid option --log-stdout: unknown option

~ winbindd --log-stdout
Invalid option --log-stdout: unknown option
def _daemon_stdout_opt(daemon: str) -> str:
    if daemon == "smbd":
        opt = "--log-stdout"
    else:
        opt = "--stdout"
    opt_lst = get_samba_specifics()
    if _DAEMON_CLI_STDOUT_OPT in opt_lst:
        opt = "--debug-stdout"
    return opt

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.