marefr / grafana-agent-edgeos

Running Grafana Agent on Ubiquiti EdgeOS

Grafana Agent on EdgeOS

Note: This work is in-progress. Use at your own risk!

This is a short guide for getting Grafana Agent running on the Ubiquiti EdgeRouter platform. EdgeOS 2.0+ is required to make use of systemd.

Work here was highly inspired by Monitoring EdgeRouter with Node Exporter, however by using Grafana Agent there's no need for a standalone Prometheus server to scrape your metrics.

Setup

1. Enter a root shell

   sudo -i

2. Create the required directories

   mkdir -p /config/firstboot.d /config/grafana-agent

3. Fetch the firstboot script

   This script will run every time the device have been updated with a new bootloader, i.e. after firmware upgrade etc.

   curl -o /config/firstboot.d/grafana-agent.sh https://raw.githubusercontent.com/marefr/grafana-agent-edgeos/main/firstboot.d/grafana-agent.sh
   chmod 755 /config/firstboot.d/grafana-agent.sh

4. Fetch the default grafana-agent configuration file and systemd service files

   curl -o /config/grafana-agent/grafana-agent.yaml https://raw.githubusercontent.com/marefr/grafana-agent-edgeos/main/grafana-agent/grafana-agent.yaml
   curl -o /config/grafana-agent/environment-file https://raw.githubusercontent.com/marefr/grafana-agent-edgeos/main/grafana-agent/environment-file
   curl -o /config/grafana-agent/grafana-agent.service https://raw.githubusercontent.com/marefr/grafana-agent-edgeos/main/grafana-agent/grafana-agent.service

5. Download the latest grafana-agent binary

   Different EdgeRouter models use either MIPS or MIPS-LE, so make sure you download the right tarball for your platform.
   ER-4 is MIPS, ER-X is MIPSLE.

   Note, currently only MIPS-LE supported.

   curl -fL0 https://github.com/marefr/grafana-agent-edgeos/releases/latest/download/agent-linux-mipsle.tar.gz.sha256 > /tmp/agent-linux-mipsle.tar.gz.sha256
   curl -fL0 https://github.com/marefr/grafana-agent-edgeos/releases/latest/download/agent-linux-mipsle.tar.gz > /tmp/agent-linux-mipsle.tar.gz
   cd /tmp
   sha256sum -c agent-linux-mipsle.tar.gz.sha256
   tar -zxvf agent-linux-mipsle.tar.gz

6. Move the extracted binary to /config/grafana-agent

   mv -f agent-linux-mipsle /config/grafana-agent/

7. Cleanup /tmp

   Disk space is limited on some of these devices so could be a good idea to cleanup things.

   rm -rf /tmp/agent-linux-mipsle*

8. Run the firstboot script

   /config/firstboot.d/grafana-agent.sh

9. Install the Linux Server integration in your Grafana Cloud instance

   • Follow the guide
   • In the Run this command instruction, extract the GCLOUD_STACK_ID and GCLOUD_API_KEY for later use as metrics username and password.

10. Install the Loki integration in your Grafana Cloud instance

    Follow the guide to create and get a Grafana Cloud logs endpoint URL, username, password and URL.

11. Edit grafana-agent configuration

    Configure Grafana Cloud endpoint URLs, usernames and API keys for logs and metrics. See Configuration Reference for all available configuration parameters.

    vi /config/grafana-agent/grafana-agent.yaml

12. Start the agent

    Note: Running the agent might hang the EdgeRouter requiring a reboot of your device. Keeping an eye of CPU and system load in the Integration - Linux Node / Nodes dashboard is a good idea allowing you to quit the agent if things doesn't go as expected.

    systemctl start grafana-agent
    systemctl status grafana-agent

13. Browse dashboard and logs

    Browse the Integration - Linux Node / Nodes dashboard:

    Jump into Explore and browse logs:

14. (Optional) enable systemd service

    Note: Running the agent might hang the EdgeRouter and rebooting with systemd service enabled could lead to a situation where you cannot access your router since the agent will be started at boot. Recommended to try running the agent first a couple times and monitor it before enabling the systemd service.

    systemctl enable grafana-agent

Thanks

Inspiration borrowed from tailscale-edgeos.

grafana-agent-edgeos's Issues

Ideas to limit resource usage?

Hi @marefr and thanks for making this available! I was playing with this to monitor an ER-X SFP, however I can't get the agent to start and stay up (it gets OOM killed, manages to ship some logs to Loki but no metrics).

I even tried limiting it's CPU usage via systemd using

CPUWeight=20
CPUQuota=50%

but to no success. Do you have any idea how to better limit resource usage of the grafana-agent? I know it's not exactly built with low power platforms like MIPS in mind, but I was looking through the grafana-agent.yaml configuration reference and didn't discover any obvious knobs to turn.

Also it would be cool if you could post how you built the MIPS binaries, maybe there's also some golang level trickery that can make grafana-agent more amenable to low power platforms.

MIPS support

In the README you mention:

• "currently only MIPS-LE supported"

Is there any special reason why MIPS "Big Endian" is not supported?
Does it fail to build, or is it just because you don't have hardware to test it?