A humble bundle of observability and monitoring for your Lemmy cluster.
If you're an admin, you know how important it is to be able to check the various details of your instance in a glance. That is especially important when troubleshooting an outage or a drop in latency.
Even after the outage, you'd want to peruse the system statistics prior to the outage and during it and understand what was the root cause.
One of the standard ways in the industry to achieve that is to install an observability stack in your cluster that scrapes yours hosts and services for useful information, stores them and allows you to query them in various ways.
lemmy-synapse is an attempt to bring that fresh breeze of observability to your instance with ease.
Once installed, you'd have access to the power of Prometheus and Grafana to massage and visualise your instance stats the way you need it.
Additionally, you'd have access to a small number of pre-configured dashboards which you can start using right away.
📸 Here are a couple of screenshots for your browsing pleasure:
Based on the data from a handful of live lemmy-synapse installations:
- It occupies a steady amount of 300-340MB of RAM.
- CPU usage is almost always negligible and stays below 3-4% of a single core.
Great! You have decided to give lemmy-synapse a shot. There are only a couple of prerequisites and things you should figure out in advance.
Needless to say, you must have SSH access to your server.
To be able to setup lemmy-synapse, your instance PostgreSQL must be accessible from outside the Docker network.
One simple way to test it out is to run the following command on your server.
psql -U lemmy -d lemmy -h localhost
If you see a password prompt, you're good to go.
For reason beyond lemmy-synapse, Lemmy's installation repo (aka lemmy-ansible) does not
expose PG ports to outside of Docker network. Doing so is easy and only requires
modifying docker-compose.yml, so that the postgres block looks like this:
...
postgres:
image: docker.io/postgres:15-alpine
hostname: postgres
environment:
- POSTGRES_USER=lemmy
- POSTGRES_PASSWORD=...
- POSTGRES_DB=lemmy
ports:
- "5432:5432"
volumes:
- ./volumes/postgres:/var/lib/postgresql/data:Z
- ./customPostgresql.conf:/etc/postgresql.conf
restart: always
command: postgres -c config_file=/etc/postgresql.conf
logging: *default-logging
... bmakelib is a library that I wrote and maintain to help me write cleaner make files (yes, I'm a big fan of GNU Make 😁).
Please follow the installation steps to, well, install it.
MacOS Only
The installation process uses GNU tar to package lemmy-synapse. If
you're using Homebrew, simply install gnu-tar formula.
Don't forget to run the following in your terminal so that GNU tar becomes the active tar application.
export PATH="$HOMEBREW_PREFIX/opt/gnu-tar/libexec/gnubin:$PATH"
In order to scrape metrics, lemmy-synapse will attach to your instance's Docker network and probe its services (namely PostgreSQL.)
To find out the name, you can try the following command:
docker network ls | perl -nalE 'say $F[1] if $F[1] =~ /(lemmy|_default)/'
lemmy-synapse creates a special PostgreSQL user with very limited permissions. In order
to do that you'd need to know the password to the PG admin user (which is lemmy unless
you have manually configured your PG server.)
To find out the password you can run the following command:
hjson-cli -c lemmy.hjson | jq -r '.database.password'
The last step is to review the information in installation-config.yml and fill in the
blanks.
Installing lemmy-synapse is, hopefully, just a one-liner:
Linux
make \
ansible.user=<REMOTE_USER> \
ansible.password-auth=no \
ansible.lemmy-synapse-server=<YOUR_INSTANCE> \
install
MacOS
make \
-I"$HOMEBREW_PREFIX/include" \
ansible.user=<REMOTE_USER> \
ansible.password-auth=no \
ansible.lemmy-synapse-server=<YOUR_INSTANCE> \
install
💡 If you're using SSH password authentication, make sure you set
ansible.password-auth=yes when calling Make.
💡 During the installation, Ansible will pause with a "BECOME password" prompt. That's simply the Ansible way of asking for the sudo password on the remote machine.
Once the installation is done, Grafana (which is the metrics visualisation and reporting
tool) should be running and ready to accept connections on port 3000.
One simple and secure way to access it is via SSH port forwarding. For example, assuming
you've run the following command, you can access Grafana at http://localhost:3000 with
user admin and password admin.
ssh -L3000:localhost:3000 <YOUR_INSTANCE>
Happy monitoring!
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent
