cisagov / lme

Logging Made Easy (LME) is a no-cost and open logging and protective monitoring solution serving all organizations.

Logging Made Easy

Initially created by NCSC and now maintained by CISA, Logging Made Easy is a self-install tutorial for small organizations to gain a basic level of centralized security logging for Windows clients and provide functionality to detect attacks. It's the coming together of multiple open software platforms which come at no cost to users, where LME helps the reader integrate them together to produce an end-to-end logging capability. We also provide some pre-made configuration files and scripts, although there is the option to do it on your own.

Logging Made Easy can:

• Show where administrative commands are being run on enrolled devices
• See who is using which machine
• In conjunction with threat reports, it is possible to query for the presence of an attacker in the form of Tactics, Techniques and Procedures (TTPs)

Disclaimer

LME is currently still early in development.

If you have an existing install of the LME Alpha (v0.5 or older) some manual intervention will be required in order to upgrade to the latest version, please see Upgrading for further information.

This is not a professional tool, and should not be used as a SIEM.

LME is a 'homebrew' way of gathering logs and querying for attacks.

We have done the hard work to make things simple. We will tell you what to download, which configurations to use and have created convenient scripts to auto-configure wherever possible.

The current architecture is based upon Windows Clients, Microsoft Sysmon, Windows Event Forwarding and the ELK stack.

We are not able to comment on or troubleshoot individual installations. If you believe you have have found an issue with the LME code or documentation please submit a GitHub issue. If you have a question about your installation, please visit GitHub Discussions to see if your issue has been addressed before.

Who is Logging Made Easy for?

From single IT administrators with a handful of devices in their network to larger organizations.

LME is for you if:

• You don’t have a SOC, SIEM or any monitoring in place at the moment.
• You lack the budget, time or understanding to set up your own logging system.
• You recognize the need to begin gathering logs and monitoring your IT.
• You understand that LME has limitations and is better than nothing - but no match for a professional tool.

If any, or all, of these criteria fit, then LME is a step in the right direction for you.

LME could also be useful for:

• Small isolated networks where corporate monitoring doesn’t reach.

Overview

The LME architecture consists of 3 groups of computers, as summarized in the following diagram:

Figure 1: The 3 primary groups of computers in the LME architecture, their descriptions and the operating systems / software run by each.

Table of contents

Installation:

• Prerequisites - Start deployment here
• Chapter 1 - Set up Windows Event Forwarding
• Chapter 2 – Sysmon Install
• Chapter 3 – Database Install
• Chapter 4 - Post Install Actions

Logging Guidance

• Log Retention
• Additional Log Types

Reference:

• FAQ
• Troubleshooting
• Dashboard Descriptions
• Guide to Organizational Units

Maintenance:

• Backups
• Upgrading
• Certificates

lme's Issues

Document Code release process within GitHub

🐛 Summary

Document code release process within GitHub.

Process for when LME has an incremental code change.
Process for when LME has a dependency update (ELK stack, Ubuntu, etc.)
Process for when a new capability is merged into LME.

To reproduce

Steps to reproduce the behavior:

1. Do this
2. Then this

Expected behavior

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

Update upgrading.md for release 1.3.0

Description

Update upgrading.md for release 1.3.0 and fix link in Chapter3.md

To Reproduce

Click link on line 247 (Grab the self-signed certificate.....). Will get a 404 error

Verify fields in parsed logs.

From Adam's comments:

Check if we are loading the elasticsearch ingest node pipelines from the winlogbeat setup command and that logstash is configured to use the ingest node pipelines.

Reference this Slack Canvas discussion: https://dhscisa.enterprise.slack.com/docs/T02QH7E1MHA/F067EAFHZT9

Create testbed from snapshots

As a test user I would like to create a testbed from previously saved snapshots so I can save time spinning up a new testbed.

Dashboard spamming login events from the Event Log Collector server itself

🐛 Summary

In the Activity Dashboard I'm seeing thousands of logons from the Windows Server that I configured as the Event Log Collector. The WEC sent 33K, while the other 6-7 clients only sent about a dozen logs so far each.

Expected behavior

I can't tell if it's showing the clients logs, or if these are all generated by the server itself. Is this normal behavior for the collector and I should create an exclusion?

Any helpful log output or screenshots

In the screenshot the Event Collector server name is "WEC". The other 6-7 clients are sending logs but so far only a dozen logs each.

Chapter 3 - Checklist

Hi,

I'm upto Chapter 3 Checklist, however the kibana webpage doesn't load. I've tried on the Linux ELK server itself (https://127.0.0.1) and that doesn't load either (unable to connect)

I can't see port 443 as listening when running sudo ss -ltn
I'm not sure what I'm missing. What troubleshooting can i do?

Thanks

Run initial Threat emulation test using MITRE CALDERA within the LME environment.

🐛 Summary

Run initial Threat emulation using MITRE CALDER in an LME environment

Next step will be to create or run scripts that can generate threat emulation mock data using MITRE CALDER that can be used for testing code and generating logs for dashboards and alerts.

Some recommendations : https://github.com/ScarredMonk/SysmonSimulator

ElasticSearch Container crash every few seconds

My ElasticSearch container crash every few seconds after Updating from 0.5.1 to 1.0
In the logs I can see the following error:

Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
{"@timestamp":"2023-10-30T10:21:48.331Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es","error.type":"java.lang.IllegalStateException","error.message":"Unable to access 'path.repo' (/usr/share/elasticsearch/backups)","error.stack_trace":"java.lang.IllegalStateException: Unable to access 'path.repo' (/usr/share/elasticsearch/backups)\n\tat [email protected]/org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:66)\n\tat [email protected]/org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:252)\n\tat [email protected]/org.elasticsearch.bootstrap.Security.createPermissions(Security.java:178)\n\tat [email protected]/org.elasticsearch.bootstrap.Security.configure(Security.java:125)\n\tat [email protected]/org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:188)\n\tat [email protected]/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66)\nCaused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/backups\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:355)\n\tat [email protected]/org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:326)\n\tat [email protected]/org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:64)\n\t... 5 more\n"}
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/loggingmadeeasy-es.log

Any idea, how can I fix this? I was follow exactly your upgrade guide.

Docker version: 20.10.23, build 7155243
Ubuntu 20.04.6 LTS

Folder /usr/share/elasticsearch/ doesn't exist. So there are no log files there.

Best Regards
Patrick

Add local branch naming convention

Is your feature request related to a problem? Please describe.
It would be helpful to be able to pair a branch with the developer and to identify a branch with the issue it is fixing.

Describe the solution you'd like
Create naming convention for future branch creation.

Dashboard spamming login events from the Event Log Collector server itself #22

See #22

ELK Stack Buffer Overflow Bug

🐛 Summary

A critical Kibana buffer overflow bug affects Kibana versions 8.0.0 to 8.10.2. LME installs version 8.7.1.
See https://discuss.elastic.co/t/kibana-8-10-3-7-17-14-security-update/344735 for more details.

File to update: https://github.com/cisagov/LME/blob/main/Chapter%203%20Files/docker-compose-stack.yml.

To reproduce

N/A

Expected behavior

N/A

Error "Unable to determine retention policy"

🐛 Summary

I am running into an error at step 3.2.2 when running the deploy.sh script. The command I am running is:
sudo ./deploy.sh install even after following steps on issue 19 (#19)

To reproduce

Steps to reproduce the behavior:

1. Have a clean install of Ubuntu 22.04.3 LTS
2. Run apt update and apt upgrade
3. Reboot
4. Run "lvextend -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv"
5. Then "resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv"
6. Reboot (just in case)
7. Then running steps on 3.2.2
   (# Install Git client to be able to clone the LME repository
   sudo apt update
   sudo apt install git -y

Download a copy of the LME files

sudo git clone https://github.com/cisagov/lme.git /opt/lme/

Change to the LME directory containing files for the Linux server

cd /opt/lme/Chapter\ 3\ Files/

Execute script with root privileges

sudo ./deploy.sh install
)

Expected behavior

I expected it should detect correctly the disk space and set a retention policy and proceed with install and show the passwords for access to the dashboard, seems like the process setups elastic and I can see the page but as I don't have any user and password I cannot use it.

Any helpful log output or screenshots

Paste the results here:

administrator@vmlme01:/opt/lme/Chapter 3 Files$ sudo ./deploy.sh install
Will execute the following intrusive actions:
        - apt update/upgrade
        - install docker (please uninstall before proceeding, or indicate skipping the install)
        - initialize docker swarm (execute `sudo docker swarm leave --force`  before proceeding if you are part of a swarm
        - automatic os updates via unattened-upgrades)
Proceed ([y]es/[n]o):y
[X] Updating OS software
Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:2 http://es.archive.ubuntu.com/ubuntu jammy InRelease
Hit:3 http://es.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:4 http://es.archive.ubuntu.com/ubuntu jammy-backports InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[X] Installing prerequisites
Reading package lists...
Building dependency tree...
Reading state information...
net-tools is already the newest version (1.60+git20181103.0eebece-1ubuntu5).
zip is already the newest version (3.0-12build2).
curl is already the newest version (7.81.0-1ubuntu1.14).
curl set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
This OS was detected as: ubuntu
[X] Configuring Auto Updates
Reading package lists...
Building dependency tree...
Reading state information...
unattended-upgrades is already the newest version (2.8ubuntu1).
unattended-upgrades set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Enter the IP of this Linux server: 172.16.xx.xx
Enter the Fully Qualified Domain Name (FQDN) of this Linux server. This needs to be resolvable from the Windows Event Collector: vmlme01.xxx.xxx
[X] Configuring winlogbeat config and certificates to use 172.16.xx.xx as the IP and vmlme01.xxx.xxx as the DNS
This script will use self signed certificates for communication and encryption. Do you want to continue with self signed certificates? ([y]es/[n]o): y
Skip Docker Install? ([y]es/[n]o): n
Do you have an old elastic user password? ([y]es/[n]o): n
[!] Note: Depending on your OpenSSL configuration you may see an error opening a .rnd file into RNG, this will not block the installation
[X] Making root Certificate Authority
[X] Signing root CA
Certificate request self-signature ok
subject=C = US, ST = DC, L = Washington, O = CISA, CN = Swarm
[X] Making Logstash certificate
[X] Signing logstash cert
Certificate request self-signature ok
subject=C = US, ST = DC, L = Washington, O = CISA, CN = vmlme01.ijarque.com
[X] Making Winlogbeat client certificate
[X] Signing wlbclient cert
Certificate request self-signature ok
subject=C = US, ST = DC, L = Washington, O = CISA, CN = wlbclient
[X] Making Elasticsearch certificate
[X] Sign elasticsearch cert
Certificate request self-signature ok
subject=C = US, ST = DC, L = Washington, O = CISA, CN = elasticsearch
[X] Making Kibana certificate
[X] Sign kibana cert
Certificate request self-signature ok
subject=C = US, ST = DC, L = Washington, O = CISA, CN = kibana
[X] Installing Docker
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c install -m 0755 -d /etc/apt/keyrings
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg
+ sh -c chmod a+r /etc/apt/keyrings/docker.gpg
+ sh -c echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-ce-rootless-extras docker-buildx-plugin >/dev/null
+ sh -c docker version
[X] Configuring Docker swarm
Swarm initialized: current node (gsznx0x0q5fccf74s4nyncrhl) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-1a5q6dbwujblpohkhdqeboxh7e2yttd1vh1ljkx172rkfl1wvd-edv6pu5n2oeldvtg6dg9uofji 172.16.xx.xx:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

[X] Adding certificates and keys to Docker
5izatgmr7qhbz1h4d3aobk2iq
bjcw7cbl3x4t4eagtzthuad3x
yi8msvau8t0p1tbwxkwlyetoy
gk5twe4ny6146pirtstlza1gb
0zguahmk5yyf3k7cqjztzp78d
xv5qer0y61t7zyi8rcu115ygb
dezgkymbhdeivdxovuqqe6tie
[X] Updating logstash configuration with logstash writer
blhag7qyeidsbciabcs5tw54y
[X] Creating custom logstash conf
xd492kw80q2zw5l0uzcr1gdfm
vm.max_map_count = 262144
Creating network lme_esnet
Creating service lme_logstash
Creating service lme_elasticsearch
Creating service lme_kibana
[X] Waiting for elasticsearch to be ready
[X] Setting elastic user password
{}
[X] Setting kibana system password
{}
[X] Setting logstash system password
{}
[X] Setting logstash writer role
{"role":{"created":true}}
[X] Setting dashboard update role
{"role":{"created":true}}
[X] Creating logstash writer user
{"created":true}
[X] Setting logstash writer password
{}
[X] Creating dashboard update user
{"created":true}
[X] Setting dashboard update user password
{}
[X] Configuring elasticsearch Replica settings
{"error":{"root_cause":[{"type":"parse_exception","reason":"unknown key [template] in the template "}],"type":"parse_exception","reason":"unknown key [template] in the template "},"status":400}{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [[_all]]","index_uuid":"_na_","index":"[_all]"}],"type":"index_not_found_exception","reason":"no such index [[_all]]","index_uuid":"_na_","index":"[_all]"},"status":404}
[X] Generating files_for_windows zip
  adding: tmp/lme/ (stored 0%)
  adding: tmp/lme/wlbclient.crt (deflated 24%)
  adding: tmp/lme/wlbclient.key (deflated 24%)
  adding: tmp/lme/root-ca.crt (deflated 25%)
  adding: tmp/lme/winlogbeat.yml (deflated 52%)
test of /opt/lme/files_for_windows.zip OK

[X] Setting Elastic pipelines
{"acknowledged":true}[X] We think your main disk is 111G
[!] Unable to determine retention policy - exiting

Add any screenshots of the problem here.

Elastic Password and RedHat 8 issue

I’m having to convert the script over to run on RedHat 8. The issue is Docker and Elastic Search. Are Docker and Elastic Search free services. It ask about a password from an old Elastic account, in which I do not have and when I went to the site to see if I could acquire one – it states it’s a paid service. Do you know of anyone that has ran this successfully on a Redhat machine. I’m trying not to create an Ubuntu machine, our network is hard set for Redhat.

Add a reverse proxy for services and file hosting

🐛 Summary

Add a reverse proxy http server so files, data and services can be behind a single point of authentication and access.

Also add a caddy server to automatically pull relevant certs.

Enforce encryption.

What's wrong? Please be specific.

To reproduce

Steps to reproduce the behavior:

1. Do this
2. Then this

Expected behavior

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

[BUG] Function configelasticsearch doesn't work

The function configelasticsearch in its current form is met with an error during install.

function configelasticsearch() {
  echo -e "\n\e[32m[X]\e[0m Configuring elasticsearch Replica settings"

  #set future index to always have no replicas
  curl --cacert certs/root-ca.crt --user "elastic:$elastic_user_pass" -X PUT "https://127.0.0.1:9200/_template/number_of_replicas" -H 'Content-Type: application/json' -d' {  "template": "*",  "settings": {    "number_of_replicas": 0  }}'
  #set all current indices to have 0 replicas
  curl --cacert certs/root-ca.crt --user "elastic:$elastic_user_pass" -X PUT "https://127.0.0.1:9200/_all/_settings" -H 'Content-Type: application/json' -d '{"index" : {"number_of_replicas" : 0}}'
}

[X] Configuring elasticsearch Replica settings
{"error":{"root_cause":[{"type":"parse_exception","reason":"unknown key [template] in the template "}],"type":"parse_exception","reason":"unknown key [template] in the template "},"status":400}{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [[_all]]","index_uuid":"na","index":"[_all]"}],"type":"index_not_found_exception","reason":"no such index [[_all]]","index_uuid":"na","index":"[_all]"},"status":404}

This is for 2 reasons.

First: The commands are outdated and were used with elasticsearch 6.*

_template and _all are outdated

https://discuss.elastic.co/t/deprecation-warnings-on-system-indices/180827

The entire function can be re-done to look like this:

function configelasticsearch() {
  echo -e "\n\e[32m[X]\e[0m Configuring elasticsearch Replica settings"

  #set future index to always have no replicas
  curl --cacert certs/root-ca.crt --user "elastic:$elastic_user_pass" -X PUT "https://127.0.0.1:9200/_index_template/number_of_replicas" -H 'Content-Type: application/json' -d'{
  "index_patterns": ["*"],
  "template": {
    "settings": {
      "number_of_replicas": 0
    }
  },
  "priority": 1
}'


  #set all current indices to have 0 replicas
  curl --cacert certs/root-ca.crt --user "elastic:$elastic_user_pass" -X PUT "https://127.0.0.1:9200/*/_settings" -H 'Content-Type: application/json' -d '{"index" : {"number_of_replicas" : 0}}'
}

The 2nd problem:

The configelasticsearch function is called at the wrong time for the 2nd curl command to work. It has to be performed AFTER bootstrapindex function because the indexes do not exist until after this function.

Final question:

What does this do and is it needed? It doesn't keep the install from happening successfully, but could maybe create longer term issues with scaling, etc? I'm not super knowledgeable on replicas and everything behind it.

Research automation for tagging and release notes

Is your feature request related to a problem? Please describe.

During the release of v1.1.0 of LME we discussed automation of the tagging process and deploying releases automatically. Currently this process is done manually. The purpose of this issue is to research into ways we can go about this, then execute on a method that works best for the team.

Describe the solution you'd like

• Automate tagging based on the branch name
  • when the release-1.1.0 branch is merged into main, the new release shall be marked v1.1.0
• Automate creation of release notes
  • Header should contain the following information: [<version-number>] - Timberrrrr! - <date-of-release>
  • Sections for what was added, changed, fixed, and additional notes.

Considerations to look into:

1. Can we provide specific details to be included in the release markdown (under added, changed, fixed, etc.) or are we restricted to what the commit messages were?
2. How can we better utilize Github Actions to test LME prior to a release?

Describe alternatives you've considered

There's most likely some sort of Git release manager tool that already exists: https://github.com/GitTools/GitReleaseManager

Additional context

What our release notes currently look like:

Sample from GitReleaseManager: https://github.com/chocolatey/ChocolateyGUI/releases/tag/0.12.0

This issue is complete when:

• #135 is resolved

Introducing MacOS log Captures

🐛 Summary

Introducing MacOS log captures for LME.
What's wrong? Please be specific.

To reproduce

Steps to reproduce the behavior:

1. Do this
2. Then this

Expected behavior

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

Restructure documentation and scripts so that updating version numbers is easier.

Look into adding variables to documentation and scripts to make updating versions easier.

These are items from the LME presentation that need to be re-evalutated.

Deploy.sh requires improvements before release (tracked via github.com)

Current LME minimal requirements need to be properly benchmarked so users can deploy and expect resources to handle load (e.g. 90GB supports 17 clients for 1 day)

Explore configuration as code

Explore adding support for non-Windows machines

Explore EDR tools

Create pipeline to implement and stream relevant detection rules to the ELK stack deployment

Evaluate other logging solutions (e.g. Wazuh) same as ticket #127

What alternate logging solutions do we want to evaluate? Wazuh? Greylog? Zeek?

See ticket#127 https://github.com/orgs/cisagov/projects/63?pane=issue&itemId=60616342

LME Operational Limits Question

Hi,
On the Logging Made Easy program, what are the operational limits of the program in regards to the number of sources that can feed it? Would 630 windows workstations sending their logs be disable? What would server requirements be? I haven’t download it yet to see if there is more info included with the program.

Unable to bring up webpage

Recently installed LME and was working fine. Can't access the Elastic Search webpage. (site cannot be reached).
I'm able to SSH to the server and can run: sudo docker stack ps lme.
I get a few running and a few shutdowns.

Other than starting all over again, how can I troubleshoot?

Elasticsearch fails to boot on Linux server

🐛 Summary

Accessing https://<LINUX_SERVER_IP/HOSTNAME> on failed. Running Resolve-DnsName LS1 returns IP address of the Linux server which confirms its resolvable from the event collector, so this rules out any issues with port 443 being open.

Ran sudo docker stack ps lme from the Linux server, noticed that the elasticsearch service tries to startup but fails with a "task: non-zero exit (1)" error. Running docker service logs lme_elasticsearch provides more insight as to why the elasticsearch service fails.

Underlying issue seems to be that elasticsearch does not have the correct permissions to access /usr/share/elasticsearch/backups:

To reproduce

Steps to reproduce the behavior:

1. Stop and restart Linux server
2. Login to Linux server, run sudo docker stack ps lme to see if elasticsearch, kibana, and logstash are running

Expected behavior

Expect elasticsearch, kibana, and logstash services to be in the state of running in sudo docker stack ps lme.
If all services are running and winlogbeat service is running on the event collector, expect https://<LINUX_SERVER_IP/HOSTNAME> to be accessible from the event collector.

[BUG] Error code is 12175

BEFORE CREATING THE ISSUE, CHECK THE FOLLOWING GUIDES:

• FAQ
• Troubleshooting
• Search current/closed issues for similar questions, and utilize github/google search to see if an answer exists for the error I'm encountering.

If the above did not answer your question, proceed with creating an issue below:

Describe the bug

The forwarder is having a problem communicating with subscription manager at address https://example.com:5985/wsman/SubscriptionManager/WEC. Error code is 12175 and Error Message is <f:WSManFault xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="12175" Machine="client.com><f:Message>The server certificate on the destination computer (example.com:5985) has the following errors:
Encountered an internal error in the SSL library. </f:Message></f:WSManFault>.

To Reproduce

Collector is a Windows server 2019, test systems are Window Server 2022. Went through the procedures multiple times, resulting in same error. Tried the listed troubleshooting snippets and a few of my own to ensure that the servers were communicating on the correct ports.

To increase the speed and relevance of the reply we suggest you list down debugging steps you have tried, as well as the following information:

Please complete the following information

Collection Server

• OS: Server 2019
• Browser: Edge / TLS 1.2
• Software version: Sysmon v15.11.0.0, Winlogbeat 8.5.0]

Server:

• OS: [e.g. Ubuntu 22.04]

• Software Versions:

  • ELK: [e.g. 8.7.1]
  • Docker:
    docker-buildx-plugin/jammy,now 0.11.2-1ubuntu.22.04jammy amd64 [installed]
    docker-ce-cli/jammy,now 5:24.0.7-1ubuntu.22.04jammy amd64 [installed]
    docker-ce-rootless-extras/jammy,now 5:24.0.7-1ubuntu.22.04jammy amd64 [installed]
    docker-ce/jammy,now 5:24.0.7-1ubuntu.22.04jammy amd64 [installed]
    docker-compose-plugin/jammy,now 2.21.0-1ubuntu.22.04jammy amd64 [installed]

• The output of these commands:

free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi        28Gi       350Mi        18Mi       2.8Gi       2.5Gi
Swap:          8.0Gi       218Mi       7.8Gi

df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              3.2G  3.5M  3.2G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  196G   21G  167G  11% /
tmpfs                               16G     0   16G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/sda2                          974M  235M  672M  26% /boot
tmpfs                              3.2G  104K  3.2G   1% /run/user/1000
overlay                            196G   21G  167G  11% /var/lib/docker/overlay2/8d96648f521ab12d49cb0169bfab4501b89aab0ed6053a8f5a4dec72f7d2d101/merged
tmpfs                               16G   20K   16G   1% /var/lib/docker/containers/8cdc1187f463870439857d7ca41ef8bba14663154326dacda1b578a62f6e45e9/mounts/secrets
overlay                            196G   21G  167G  11% /var/lib/docker/overlay2/b62af5bca1d36c10c873fb3a5ce60665dfef0d08943dde6837c4e801bb0edf95/merged
tmpfs                               16G   12K   16G   1% /var/lib/docker/containers/546eae30b2a7c9e2bc7a1104c53eafd9f7bb0707d56df380f0f57d0b6abff40b/mounts/secrets
overlay                            196G   21G  167G  11% /var/lib/docker/overlay2/b58f864c17ed892c290544d18dc810354425511f9a7e5138d0880b89cfd9406d/merged
tmpfs                               16G   12K   16G   1% /var/lib/docker/containers/b4b468438d4d85a1c838439369c9a3968b2130d6a2666a1541d940f6d99572f2/mounts/secrets

 uname -a
Linux lme-elk-srv 5.15.0-89-generic #99-Ubuntu SMP Mon Oct 30 20:42:41 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

 lsb_release -a
LSB Version:    core-11.1.0ubuntu4-noarch:printing-11.1.0ubuntu4-noarch:security-11.1.0ubuntu4-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

- Relevant container logs: 

for name in $(sudo docker ps -a --format '{{.Names}}'); do echo -e "\n\n\n-----------$name----------"; sudo docker logs $name | tail -n 20; done

Increase the number of lines if your issue is not present, or include a relevant log of the erroring container
- Output of the relevant /var/log/cron_logs/ file

## Additional context
I added the Collection Server in the test bed and since it is local, then it shows up in the data! but nothing from the other servers.

3.2.4 Download Files For Windows Event Collector

Hi,

All was going well with the deployment until I reached 3.2.4

I don't seem to have access as the administrator or the Ubuntu server to /opt/lme folder where the zip folder resides. All 3 methods are met with access denied.

Any ideas on how i resolve this?

thanks

Look into Curator

Because of the difficulty of supporting each and every case when it comes to hard drive size we may want to look into curator:

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

It can be installed as a docker container

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/docker.html

You can configure it to detect space usage by indices:

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/filtertype_space.html

Then you create an 'action' which is what curator will perform when the filter requirements are met:

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/delete_indices.html

Could be potential here to write "actions" and "filters" that gives us more control over space management than the default elastic lifecycle policy.

Incorporate SBOM

Add Support for a multi-server install.

🐛 Summary

LME currently supports a single server install which might fail to scale *if a medium size organization uses the current LME elk stack with 100+machines.
What's wrong? Please be specific.

To reproduce

Steps to reproduce the behavior:

1. Do this
2. Then this

Expected behavior

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

Modify the dashboard menu to remove the home page and link new dashboards for 1.1.0

💡 Summary

What is the work, as a high-level summary?
Modify the dashboard menu to remove the home page and link new dashboards for 1.1.0

Why does this work belong in this project?

This would be useful because we are removing the home page.

Acceptance criteria

All of the dashboards have the dashboard menu that links to all of the other dashboards.

• Criterion

Create Testbed from Scripts

Look at latest Sysmon release

What side effects will there be? Will the uninstall Sysmon script be useful when updating Sysmon?

[BUG] Providing an old elastic password will still generate a new password in step 3.2.2

Describe the bug

In Chapter 3 ./deploy.sh install will update the Linux system, install certs, configure Docker to run the ELK stack, and update permissions in /opt/lme. The script will prompt the user if they have an existing elastic password or not.

Currently a new elastic password is generated whether you indicate yes or no to the prompt. For some users neither the existing or new password authenticates when logging into the Kibana frontend.

To Reproduce

Either go through a fresh install of LME on the Linux server, or run ./deploy.sh uninstall from /opt/lme on your existing installation to start over. Run ./deploy.sh install and go through the prompts, specify Y when asked if you want to provide an existing elasticsearch password, enter elasticsearch password, then continue with the install process.

After completing Chapter 3, navigating to https://<LINUX-IP>/LS1 loads the Kibana login page as expected, but neither the existing or new elasticsearch password will be accepted.

Expected behavior

• If an existing elastic password is specified, then do not create a new elastic password.
  • Save the existing elastic password as specified by the user
  • Generate the remaining credentials (logstash system/writer, kibana keys) with the existing elastic password
• If an existing elastic password is not specified, then a new password should be created.
  • Generation of remaining credentials is unaffected

Screenshots

setpasswords() function for where this process takes place in deploy.sh

LME/Chapter 3 Files/deploy.sh

Line 98 in 6ff3566

function setpasswords() {

Add markdown file with descriptions of dashboards

Is your feature request related to a problem? Please describe.
We don't have documentation on what each of the dashboards display, their functionality, etc.

Describe the solution you'd like

• List all dashboards, descriptions of what they do, what fields are displayed, their functionality, etc.
• Add a DASHBOARD_ABOUT.md, DASHBOARD_DESCRIPTION.md, or similarly named file and link under Table of Contents section

Additional context
Add any other context or screenshots about the feature request here.

Introduction of automation tools

🐛 Summary

Using automation tools like Ansible, puppet for configuration management to facilitate the installation of LME.
What's wrong? Please be specific.

To reproduce

Steps to reproduce the behavior:

1. Do this
2. Then this

Expected behavior

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

Getting error with Step 3.2.2 when running the deploy.sh script

🐛 Summary

I am running into an error at step 3.2.2 when running the deploy.sh script. The command I am running is:
sudo ./deploy.sh install

The error is at the step:
[X] Configuring elasticsearch Replica settings
{"error":{"root_cause":[{"type":"parse_exception","reason":"unknown key [template] in the template "}],"type":"parse_exception","reason":"unknown key [template] in the template "},"status":400}{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [[_all]]","index_uuid":"na","index":"[_all]"}],"type":"index_not_found_exception","reason":"no such index [[_all]]","index_uuid":"na","index":"[_all]"},"status":404}

The script continues with:
[X] Generating files_for_windows zip
updating: tmp/lme/ (stored 0%)
updating: tmp/lme/root-ca.crt (deflated 25%)
updating: tmp/lme/wlbclient.key (deflated 24%)
updating: tmp/lme/wlbclient.crt (deflated 24%)
updating: tmp/lme/winlogbeat.yml (deflated 51%)
test of /opt/lme/files_for_windows.zip OK

[X] Setting Elastic pipelines
{"acknowledged":true}[X] We think your main disk is 98G

Then I get another error right after this step that reads:
[!] Unable to determine retention policy - exiting

Then the script exits.

To reproduce

After the script fails to complete, I tried the following commands to clean up and rerun the deploy.sh script:
sudo docker swarm leave --force
sudo docker stack rm lme
sudo docker volume rm lme_logstashdata
sudo docker volume rm lme_esdata

sudo ./deploy.sh install

Expected behavior

The Chapter 3, 3.2.2 instructions state that I should receive a list of usernames and passwords, but the above error is exiting before I am able to see these passwords.

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

[BUG] EVENT ID 10129 After steps 1 and 2.

Hello all. After completing chapter 1 and 2, I get an event ID 10129, stating the following :
"The WS-Management client is not listening for pushed events because there was a failure binding to the URL (...) in HTTP.SYS.
It then asks me to use netsh http and make sure that URL is set to Network Service.
Error code given is 5: %%5 apparently.

The reason I'm not giving the URL is because it doesn't matter which URL I add to Network Service, whenever I add one, and retry the service or restart the WEC, I get the same event ID with a different URL asking me to do the same thing. I've added about 15 URLs before deciding it's just a bug that will continue to recreate itself. Any tips?

On the client side I can see Sysmon64 is running, and the on the wec I can see I have 1 computer active on my subscription. The error was there before I continued to chapter 2, but I decided to try chapter 2 in case that fixed the issue.
I cannot see any logs in the forwarded events section of the event viewer.
Very new to this - please let me know if you need any more information from me!

Deploy.sh: Allow for custom data retention path

🐛 Summary

#changes to chapter3:
if the default mount of the operating system is too small, make sure to change:

#docker-compose-stack-live.yml:
#services->elasticsearch-> volumes->target:
change the filepath to the new mount that has enough space
also we'll need to add notes on how to add a new mount to store the data, this is probably best practice so that the os is seperate from the data storage

TODO: should allow for custom path for data retention size check

What's wrong? Please be specific.

To reproduce

Steps to reproduce the behavior:

1. Do this
2. Then this

Expected behavior

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

[BUG] No logic in install for certain hard drive sizes

In the function data_retention you have this logic:

if [ "$DISK_80" -lt 30 ]; then
echo -e "\e[31m[!]\e[0m LME Requires 128GB of space usable for log retention - exiting"
exit 1
elif [ "$DISK_80" -ge 90 ] && [ "$DISK_80" -le 179 ]; then
RETENTION="30"
elif [ "$DISK_80" -ge 180 ] && [ "$DISK_80" -le 359 ]; then
RETENTION="90"
elif [ "$DISK_80" -ge 360 ] && [ "$DISK_80" -le 539 ]; then
RETENTION="180"
elif [ "$DISK_80" -ge 540 ] && [ "$DISK_80" -le 719 ]; then
RETENTION="270"
elif [ "$DISK_80" -ge 720 ]; then
RETENTION="365"
else
echo -e "\e[31m[!]\e[0m Unable to determine retention policy - exiting"
exit 1
fi

In it's current form there is no logic that accounts for 30-89. So, anyone who tries to install LME falling within that range will be met with "Unable to determine retention policy - exiting"

Additionally, the echo statement that states "LME Requires 128 GB of space" doesn't align with the first rule which states if less than 30 GB.

Fix Documentation Errors

Fix File Downloads panel in Process Explorer Dashboard.

The LME ./deploy.sh script hangs on

🐛 Summary

What's wrong? Please be specific.

To reproduce

Steps to reproduce the behavior:

1. Do this
2. Then this

Expected behavior

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

Dashboard options and Alerting

🐛 Summary

What's wrong? Please be specific.

To reproduce

Steps to reproduce the behavior:

1. Do this
2. Then this

Expected behavior

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

Introducing Linux Log Captures

🐛 Summary

Introducing Linux Log Captures.
What's wrong? Please be specific.

To reproduce

Steps to reproduce the behavior:

1. Do this
2. Then this

Expected behavior

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

No logs in Elastic coming through

i am getting these errors

"Failed to connect to backoff(async(tcp://EB.local:5044)): dial tcp 10.99.0.35:5044: connectex: No connection could be made because the target machine actively refused it.","service.name":"winlogbeat",

Can you help please, all the info i find on Elastic website say to make changes to certain files like
filebeat.yml
logstash.conf

I used the script here provided by the team

Please help me out.

Look into issues with multiple adapters on WEC

We don't currently support multiple NIC's and adapters... add something in the documentation that talks about not having multiple adapters enabled on the WEC server

The LME ./deploy.sh script hangs on "Waiting for Elasticsearch to be ready"

🐛 Summary

What's wrong? Please be specific.
The execution of the script "deploy.sh" does not go to the end and hangs on the error message " Waiting for Elasticsearch to be ready"

To reproduce

Steps to reproduce the behavior:

1. Do this: install a Linux Ubuntu 22. 04. 3 LTS (Codename: Jammy)
2. configure a static IP address on the server
3. Open the appropriate ports on the server with ufw : Inbound TCP 5044; Inbound TCP 443; Inbound TCP 22 for SSH management
4. Then this: run the script provided for the LME implementation:

Install Git client to be able to clone the LME repository

sudo apt update
sudo apt install git -y

Download a copy of the LME files

sudo git clone https://github.com/cisagov/lme.git /opt/lme/

Change to the LME directory containing files for the Linux server

cd /opt/lme/Chapter\ 3\ Files/

Execute script with root privileges

sudo ./deploy.sh install

Expected behavior

What did you expect to happen that didn't? I was expecting the script to run till the end and give me a screen like this after successful installation of Elasticsearch:
##################################################################################

Kibana/Elasticsearch Credentials are (these will not be accessible again!!!!)

Web Interface login:

elastic:

System Credentials

kibana:

logstash_system:

logstash_writer:

dashboard_update:

##################################################################################

Any helpful log output or screenshots

Paste the results here:
Creating network lme_esnet
Creating service lme_elasticsearch
Creating service lme_kibana
Creating service lme_logstash
Waiting for elasticsearch to be ready

Add any screenshots of the problem here.

Improve Maintainability and testing of dashboards..

🐛 Summary

All dashboards are currently part of the same file in:

https://github.com/cisagov/LME/blob/main/Chapter%204%20Files/dashboards.ndjson

What's wrong? Please be specific.
Need to make it easier to update and maintain these dashboards.

To reproduce

Steps to reproduce the behavior:

1. Do this
2. Then this

Expected behavior

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

Create testbed from scripts

Is your feature request related to a problem? Please describe.
It takes too long to build the testbed and we cannot do it hands off. When running tests we need the pipeline to be able to spin up environments without human interaction.

Describe the solution you'd like
We need a script to spin up a testbed from one script.

Describe alternatives you've considered
Tried creating testbeds from snapshots and the domain controller requires human intervention to restore.

OpenSearch and Open SIEM verses Elasticsearch CISA LME

Delete. The original content was not a question.

See if there is more recent

💡 Summary

https://github.com/cisagov/LME/blob/main/Chapter%201%20Files/lme_wec_config.xml - Chapter 1 Files/lme_wec_config.xml
The xml schema refers to http://schemas.microsoft.com/2006/03/windows/events/subscription

Is there a newer config file?