walterdejong / synctool Goto Github PK

Uncomment the "always_run /bin/date" command in the synctool.conf example file throws this excepition error (Python 2.6.6):

Traceback (most recent call last):
File "/opt/synctool/sbin/synctool", line 518, in
main()
File "/opt/synctool/sbin/synctool", line 466, in main
(upload_filename, upload_suffix) = get_options()
File "/opt/synctool/sbin/synctool", line 336, in get_options
synctool_config.read_config()
File "/var/lib/synctool/sbin/synctool_config.py", line 63, in read_config
errors = synctool_configparser.read_config_file(synctool_param.CONF_FILE)
File "/var/lib/synctool/sbin/synctool_configparser.py", line 92, in read_config_file
errors = errors + func(arr, configfile, lineno)
File "/var/lib/synctool/sbin/synctool_configparser.py", line 687, in config_always_run
cmdfile = arr[1]
IndexError: list index out of range

Hi,
i'm testing now this great tool and i see that there is not an option to prevent Synctool to spread the whole repository content on every node during the initial sync master->node?

For example, why not sync only to the overlay_dirs declared in the used synctool.conf file?
Supposing you have a cluster with the /overlay/cluster1 and /overlay/common dirs declared, it may be useful an option which appends options like this to the rsync command:
--include "/overlay/cluster1" --include "/overlay/common" --exclude "/overlay/*

This problem is somewhat related to issue #15; synctool on the target node uses the same config as the synctool-master and it also expects all commands to be present on the node, even though the rsync_cmd, ssh_cmd, scp_cmd, and ping_cmd are only used on the master node.

This calls for a change in how the configuration works; client configs may be different from the master config.
Maybe accept something like synctool.conf._group

Hi Walter,

When I do:
synctool --erase-saved
synctool lists only the removal of .saved files. However, when I do:
synctool --erase-saved -f
also file changes are being processed! I accidentally overwrote some node changes that needed to be uploaded to the overlay tree.

This is unexpected behavior. I would suggest that the changes are listed in the dryrun, or that --erase-saved -f does not do anything but erase .saved files. I would prefer the latter; it would be more consistent with the --tasks behavior.

Kind regards
Onno

Why this application uses the ping command with -t 1 (ttl=1) parameter ?

synctool uses #! /usr/bin/env python to select the python interpreter.
Some Linux distros have a default python command that is Python 3, but synctool is Python 2 code.

The python command is documented here:
https://www.python.org/dev/peps/pep-0394/

where they basically leave it up to the distro to figure it out;

• env python may give you python2
• env python may give you python3
• env python may give No such file or directory even on systems where Python is available

This leaves us in a position where it's best to use env python2, even though python2 may not exist on old systems.

see subject

Probably also applies for dsh-pkg -R / --remove

Hi,

I'm trying to run synctool with ssh + ControlPersist for faster ssh subsequent connections.

The ssh config is like this:

ControlMaster auto
ControlPath   /home/synctool/.ssh/control/%h_%p_%r
ControlPersist 4h

The problem is that on the first run synctool hangs (but the corresponding control socket is created) and on the second run it runs properly.

dsh commands may generate message on stderr. These will not be shown with a nodename prepended to them on the masternode, so you will be unable to see where the error originated from.

I was reading the docs and playing around with synctool, yet again. And when trying to get my config sorted, I bumped into #51. On one hand, I can understand why it is discouraged for the master to be both a managed node and the cluster master. (It's sort of a catch-22, I do agree).

On the other hand we don't always have the luxury of having a dedicated master node with a stable FQDN. (My 'cluster' would really be a heterogenous collection of different VPS boxes for my various pet projects; so it's not really a cluster, but I'd be glad to manage them with synctool)

So, I was just wondering, why my laptop couldn't be the master node for managing all my toy boxes? Essentially by omitting the master keyword it could be assumed that the current machine (which has the "config repo") is the master, and all other nodes are essentially slaves.

The docs state that you shouldn't manage the master node itself with synctool. Or at least, it is discouraged. There currently is a problem with this where it runs rsync to the master node. Since synctool only syncs subtrees, other subtrees may be deleted as it runs rsync --delete.
This will happen with a config like

master mynode.domain.net

node master1 group1 group2

We are currently using synctool to install packages on servers, before doing configuration synchronization. For this we are using a kind of hack.

There is a separate overlay called overlay.migrations which kinda looks like this:

overlay.migrate/var/lib/migrations:
total 80
drwxr-xr-x 2 root root 4096 Jan 25 10:09 .
drwxr-xr-x 3 root root 4096 Nov  6 15:46 ..
-rw-r--r-- 1 root root    0 Sep 28  2011 0001
-rwxr-xr-x 1 root root 1655 Jan 25 10:09 0001.post
-rw-r--r-- 1 root root    0 Sep 28  2011 0002
-rwxr-xr-x 1 root root 2571 Oct 11 14:55 0002.post
-rw-r--r-- 1 root root    2 Sep 28  2011 0003._mysql
-rwxr-xr-x 1 root root 1896 Jan 25 10:06 0003.post
-rw-r--r-- 1 root root    0 Oct  1  2011 0004._worker
-rwxr-xr-x 1 root root 2415 Mar 17  2012 0004.post

Than I have an alias alias synctool-migrate="synctool -c /var/lib/synctool/synctool-node-migrate.conf" so that config file uses this overlay.

I think that it would be beneficial to have a separate folder (alongside overlay delete and purge) with scripts to run on target node before updating the configuration so that one can check and install packages on target node before updating configuration.

Here is an example of what I do on a migration file:

1. install custom package repository
2. install package munin
3. install common munin plugins

Usually this has to be done before updating /etc/munin/munin-node.conf and reloading the service, this is why I propose to have a separate step in which some scripts are being executed on node before checking the configuration.

commit b732f98 added https URLs for checking and downloading updates. But python urllib does not verify the server certificate. Should be done, otherwise the download still isn't secure.

Currently there is no documentation on what steps synctool is taking to do the configuration synchronization. It would be useful for example to know what files are available when running a post script.

From what I learned this is how it goes (for version 6+):

1. synchronize delete folder, delete files accordingly and run the corresponding .post script
2. synchronize purge folder if needed on node
3. synchronize overlay folder
4. run through template files and generate them
5. change what need to be changed and save a beckup copy
6. run the .post files

Thanks again for this great tool!

I was just wondering about something, and I feel, is missing from synctool. I thought I'd share my ideas, just in case, someone else is interested.

So, consider the following scenario:

I want to configure apache, so I add /etc/httpd.conf (or whatever my preferred distro calls the file...) to the overlay directory. Then I'd add a .pre script, that would call synctool-client-pkg to ensure, apache is actually installed. And a .post to restart apache if there was a config change. That's all fine and dandy, a missing config file would trigger the install, "upload" the config file, and then restart the service.

However, what if, just for the sake of the argument, I am running a hosting service, and I want to ensure certain PHP and/or Python libraries are installed (and up to date) for my customers, but I am happy with the default config. I reckon, I have no way of triggering this when running synctool or synctool-client.

I'd like to propose a new directory (alongside the overlay/delete/purge and task ones), that would contain empty files, with the names of the packages that should be installed; of course abiding by the current rules of groups and all.

As a further enhancement, I think, there is a case for allowing tagging of the files (similar to the _template tag) to use a specific package manager instead of the default for the node. This would allow for example installing pip with the default package manager, and then use pip to install Python packages.

Or, generalising the idea even further, there could be a way of defining hooks: Applying the usual rules, the files matching the current node would be interpreted as task names, and automatically called upon running synctool or synctool-client; STDIN and/or arguments for the script could come from the content of the file, very similar to how templates are handled.

fping accepts microsecond timeouts, which is nice for dsh-ping.

fping displays just "foo is alive" or "foo is unreachable", no packet counts or loss rate. (similar to Solaris ping)

fping should be suggested in synctool.conf.example and be mentioned in the documentation.

is it possible to define non standard ssh port in nodes configuration? we have few nodes that have ssh port different than 22.

idea by Onno Zweers: use --no-post to skip running of .post scripts whenever you like

I think that the purge folder name is misleading since usually purge implies mass-deleting something.

I think a more appropriate name would be mirror since everything in that folder is copied as is to the other machine.

Werner Maier says:

given the following setting:
require_extension yes
ignore .gitignore

now there is a discrepance between: "ignore .gitignore" and
"require_extension" - and such the ignored files throw a warning.

# synctool --version     
6.3-beta
# synctool --check-update
A newer version of synctool is available: version 6.2

Clearly, 6.2 is not newer.

Checking version strings doesn't work. It should have a release date(and time) and check that.

Hi Walter,

I often have to use
synctool -d filename
to check the differences and see if it is really a good idea to overwrite the changed content with the (maybe) stale content of the synctool repository.

I'd suggest supporting -d without additional parameter - if -1 is uses.
then one would be able to do this:

synctool -n node1 -1 /path/to/changed/file -d
synctool -n node1 -1 /path/to/changed/file -f'

this would be much easier to handle in cmdline with cmdline histories ;)

cheers
Werner

You can redefine group "all". However, a default synctool or dsh run still includes all nodes, rather than the nodes that are in group "all".

Hi Walter,

[root@admin ~]# synctool --version
5.1

[root@admin ~]# synctool --check-update

[root@admin ~]# synctool --download
downloading synctool-5.2.tar.gz ... 100%

[root@admin ~]# synctool --verbose --check-update
accessing URL http://www.heiho.net/synctool/LATEST.txt

[root@admin ~]# lynx -dump http://www.heiho.net/synctool/LATEST.txt
5.2 8080cdca5d6b0deffd2c7cb19125ddc2

Kind regards
Onno

synctool is Python 2 code. Python 2 goes end-of-life soon: https://pythonclock.org/

This is incorrect; if a target node has a different version of Python interpreter this will give problems.
Better exclude *.pyc from the rsync run ...

My master node: debian sid with Python 2.7.2+
My slave nodes: centos 6.2

"synctool-pkg --list" works
"synctool-pkg --install ntp" also works

But "synctool-pkg --list ntp" dosen't work, and drop me an error message:
"error: excessive arguments on command line"

This is not like what the document says.

Although you live in 2013 :D, the license file is wrong.

You need to update the FSF address, so just copy a latest one from:

http://www.gnu.org/licenses/gpl-2.0.txt

And I hope you can release the 5.3 again with the correct license included.

Thanks.

Sometimes you get this:

# dsh -g all uptime
warning: ignored: node1
warning: ignored: node1,nodex,nodey,nodez
...

Looks kind of dumb.

For example, master node is Linux, target node is Solaris:

[asm1] /var/lib/synctool # /opt/synctool/sbin/synctool -c /var/lib/synctool/synctool-silo.conf
/var/lib/synctool/synctool-silo.conf:31: no such command '/bin/diff'
/var/lib/synctool/synctool-silo.conf:32: no such command '/usr/sbin/ping'
/var/lib/synctool/synctool-silo.conf:33: no such command '/bin/ssh'
/var/lib/synctool/synctool-silo.conf:34: no such command '/bin/scp'
/var/lib/synctool/synctool-silo.conf:35: no such command '/usr/local/bin/rsync'

On the master node, the config parser should not care about the existence of "remote" commands.

Also, you can't define commands or on_update triggers per group.

It was reported that rsync will connect to the short hostname even though the fqdn is given.

The correct way to configure is to use ipaddress:host.domain.net.
The documentation says:

The optional hostname specifier tells synctool that a host that has this fully qualified hostname, must be this node.

I understand the confusion though. If no ipaddress: is given for a host, synctool tries the nodename (which is a short name). Maybe it makes more sense to try the given hostname, if specified. This could lead to other issues ... maybe ipaddress: should be mandatory if a hostname: is specified?

I have group cc-dev and node cc-qa in that group:
node cc-qa cc-dev ipaddress:some.host.name

If I add extension to my file like somefile._cc-dev or somefile._cc-qa synctool says:

cc-qa: DRY RUN, not doing any updates

but if I change extension to ._all everything works:

synctool -g cc-dev
cc-qa: DRY RUN, not doing any updates
cc-qa: /etc/nginx/ssl/somefile mismatch (file size)
cc-qa: not running command $masterdir/overlay/etc/nginx/ssl/somefile.post

I have version 5.2

The changelog link at http://walterdejong.github.io/synctool/ is broken.

Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
$ ping foo
foo is alive

Same goes for the fping command.

Hello,

How to use -1 or --single option? I have:
# ls -l overlay/var/spool/cron/crontabs/root._cc-prod
-rw------- 1 root crontab 1154 Nov 25 13:37 overlay/var/spool/cron/crontabs/root._cc-prod

But:
# synctool -1 var/spool/cron/crontabs/root._cc-prod -g cc-prod
instance1: DRYRUN not doing any updates
instance1: overlay/var/spool/cron/crontabs/root._cc-prod is not in the overlay tree
ccbeta1: DRYRUN not doing any updates
ccbeta1: overlay/var/spool/cron/crontabs/root._cc-prod is not in the overlay tree

What's wrong?

I moved the website to github pages. Consequently, synctool --check-update broke:

synctool --check-update
Traceback (most recent call last):
  File "/opt/synctool/sbin/synctool_master.py", line 19, in <module>
    synctool.main.master.main()
  File "/opt/synctool/lib/synctool/main/wrapper.py", line 26, in wrap
    ret = func(*args, **kwargs)
  File "/opt/synctool/lib/synctool/main/master.py", line 714, in main
    if not synctool.update.check():
  File "/opt/synctool/lib/synctool/update.py", line 83, in check
    latest_version = get_latest_version()
  File "/opt/synctool/lib/synctool/update.py", line 29, in get_latest_version
    tup = get_latest_version_and_checksum()
  File "/opt/synctool/lib/synctool/update.py", line 47, in get_latest_version_and_checksum
    error('webserver at %s: %s' % (VERSION_CHECKING_URL, err.reason))
AttributeError: 'HTTPError' object has no attribute 'reason'

And it doesn't print the error message nicely ...

Same thing applies to synctool --download.

root# synctool -n devel-sles12 -1 /home/jurriaan -f
--fix specified, applying changes
devel-sles12: /home/jurriaan should have owner jurriaan.jurriaan (31020.31031), but has 1000.users (1000.100)
devel-sles12: /home/jurriaan should have mode 0750, but has 0755

root# synctool -n devel-sles12 -1 /home/jurriaan -f
--fix specified, applying changes
devel-sles12: /home/jurriaan should have owner jurriaan.jurriaan (31020.31031), but has 1000.users (1000.100)

root# synctool -n devel-sles12 -1 /home/jurriaan -f
--fix specified, applying changes
devel-sles12: /home/jurriaan is up to date