Remote audit logs about aushape

[stevegrubb]

I just verified that setting distribute_network = yes in the aggregating server should make the audit events go to the plugins of audispd.

[Keysuke]

I managed to get everything working by setting log_format = ENRICHED on the server. The audisp daemon does not stop anymore after receiving remote events.
Thank you @spbnick and @stevegrubb.

[spbnick]

Hi @Keysuke, thank you for your report!
Do you run aushape under audispd, or single-shot? If it's the former, have you tried feeding the logs to aushape directly? If yes, what does it do?

If you run aushape under audispd, could you provide the contents of audispd configuration file invoking it?

Thank you.

[Keysuke]

Here are my configuration files, I am using audispd.
Everything works for local events. Only remote events (those coming from the auditd listening on a TCP port) do not get processed.
/etc/audisp/plugins.d/aushape.conf

active = yes
direction = out
path = /usr/bin/aushape-audispd-plugin
type = always
format = string

/usr/bin/aushape-audispd-plugin

#!/bin/sh
exec /usr/bin/aushape -l json --events-per-doc=none --fold=all -o syslog

[spbnick]

Do they actually get delivered to aushape?
Have you checked if aushape produces any error output?
You can do that if e.g. you add &>> /var/log/aushape-audispd-plugin.out to the end of aushape invocation in /usr/bin/aushape-audispd-plugin.

Do you see the messages from remote hosts in /var/log/audit/audit.log?
If you do, can you try feeding that log to aushape?

Thank you.

Meanwhile I'll do some testing of my own.

[Keysuke]

/var/log/audit/audit.log gets the messages from local and remote entries.

By adding &>> /var/log/aushape-audispd-plugin.out to the plugin file, I do not get any message in /var/log/aushape-audispd-plugin.out, in any case.

Using
exec /usr/bin/aushape -l json --events-per-doc=none --fold=all -f /var/log/aushape.log
in /usr/bin/aushape-audispd-plugin, the local events generate entries in /var/log/aushape.log. Remote events do not.

[spbnick]

Thank you, @Keysuke, I'm trying to reproduce this locally.

[spbnick]

I think this is audispd not sending remote logs to aushape. To verify comment out the aushape invocation in /usr/bin/aushape-audispd-plugin, and add cat >/var/log/audispd-pipe.log, then check that file after restarting auditd and generating some remote logs.

I'll try to figure out why that happens.

[spbnick]

@Keysuke, do you have distribute_network = yes in /etc/audit/auditd.conf on the server where you run aushape?

[Keysuke]

I set distribute_network = yes in my server. The problem is still here : I am only getting local entries.
I also tried on a Fedora 25 aggregating server, without any success.

[spbnick]

Thank you for trying it, @Keysuke. I'll see what the problem is on Wednesday, when I return from my travel.

[spbnick]

Sorry, the first day was pretty busy, will have to return to this tomorrow.

[spbnick]

It seems that on my host, for some reason, auditd closes pipe to audispd after receiving the first remote event. Audispd shuts down along with all plugins for that reason. @Keysuke, do you see the same?

[spbnick]

@stevegrubb, I dug some more into this and I see that auditd sends an event with zero-size payload to audispd, instead of the remote event it received. In response to that audispd shuts down. This is with audit 2.7.6-1.

Here is the relevant piece of strace from auditd:

writev(5, [{iov_base="\0\0\0\0\20\0\0\0\267\4\0\0F\0\0\0", iov_len=16}, {iov_base="audit(1497608176.826:6143): addr=192.168.122.40 port=56296 res=success", iov_len=70}], 2) = 86
write(1, "node=localhost.localdomain type=DAEMON_ACCEPT msg=audit(1497608176.826:6143): addr=192.168.122.40 port=56296 res=success\n", 121) = 121
epoll_ctl(8, EPOLL_CTL_ADD, 0, {EPOLLIN, {u32=0, u64=4294967296}}) = 0
epoll_wait(8, [{EPOLLIN, {u32=0, u64=4294967296}}], 64, 59743) = 1
read(0, "\376\0\0\377\0\0\0\0\0\0\23\1\216\0\0\0node=fedora24-dev type=USER_CMD msg=audit(1497608176.813:14039559): pid=5787 uid=1000 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/nkondras/projects/pkgs.devel.redhat.com/rpms/freeradius\" cmd=\""..., 8970) = 291
writev(5, [{iov_base="\0\0\0\0\20\0\0\0c\4\0\0\0\0\0\0", iov_len=16}, {iov_base="", iov_len=0}], 2) = 16
write(1, "node=fedora24-dev type=USER_CMD msg=audit(1497608176.813:14039559): pid=5787 uid=1000 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/nkondras/projects/pkgs.devel.redhat.com/rpms/freeradius\" cmd=\"whoami\" terminal"..., 275) = 275
write(0, "\376\0\0\377\0\0\0\0\0@\0\0\216\0\0\0", 16) = 16

The read reads the remote message and the second writev is trying to write it to audispd.

[Keysuke]

By restarting the auditd service on the remote server, I get activity on the centralizing server : the DAEMON_ACCEPT and the DAEMON_CLOSE events (which are created locally), are processed in aushape.

[spbnick]

Curiouser and curiouser. @Keysuke, do you ever see audispd stopping after receiving the remote messages?

[spbnick]

Also, @Keysuke, can you list audit package versions on both hosts? Which aushape version are you using?

[Keysuke]

Yes, it seems like audispd stops working after receiving remote events. Even for local events.
I'm using audit-2.6.5-3.el7_3.1.x86_64 and audispd-plugins-2.6.5-3.el7_3.1.x86_64 on both hosts.

[spbnick]

Right, thank you, @Keysuke. Let's see what @stevegrubb has to say.

[stevegrubb]

Something doesn't sound right. On the remote systems:
Need to set a node name method
configure au-remote to point to server
set format to enriched

On aggregating server
also set a node name method
set format to enriched
configure receiving port (usually 60)
open hole in firewall
distribute_network = no
at this point disable aushape

start the server and then the client
generate some events on the client
verify that they are being recorded to disk correctly with ausearch on the aggregating server.

When the remote events are verified good on disk, then enable aushape on the aggregating server and change distribute_network to yes. Restart auditd.

If any program exits, check the logs and see what its saying. Audisp and auditd are usually pretty good about saying why something is closing.

[spbnick]

Thank you, @stevegrubb. I played with this for a while and found that this works as long as the aggregating server has log_format = ENRICHED. If I set log_format = RAW, no matter what the remote system uses, the audispd will stop right after receiving the first remote message, because it gets an event with zero-length payload. The message in the journal is simply something like this:

Jun 20 13:19:47 localhost.localdomain auditd[13720]: dispatcher 13722 reaped

Here is the code that does the exiting: https://github.com/linux-audit/audit-userspace/blob/master/audisp/audispd.c#L868

@Keysuke, could you please try setting log_format = ENRICHED on the server and check if that works?

[stevegrubb]

Hmm...auditd is supposed to treat each event separately and switch between protocol1 & 2 seamlessly (raw & enriched respectively). This sounds like its not so seamless. :-) I'll check into this. In the mean time, it is intended that whenever you are aggregating events, all nodes should be using enriched events so that the remote uid/gid is resolved before sending.

[spbnick]

@stevegrubb, yes I agree it is better to use enriched events when aggregating. Thanks, Steve!

[stevegrubb]

I know this issue is closed, but i wanted to mention that I fixed the auditd portion of the problem with this commit:

linux-audit/audit-userspace@878f1e1

[spbnick]

Thanks a lot, Steve!