Coder Social home page Coder Social logo

opensourceictsolutions / zabbix-mysql-partitioning-perl Goto Github PK

View Code? Open in Web Editor NEW
31.0 4.0 21.0 62 KB

This script is a script written in Perl to partition the Zabbix database tables in time based chunks. We can use this script to replace the Zabbix housekeeper process which tends to get too slow once you hit a certain database size.

Home Page: https://oicts.com

License: GNU General Public License v3.0

Perl 100.00%
zabbix mysql partitioning perl database mariadb zabbix-server

zabbix-mysql-partitioning-perl's Introduction

Important notes/bugs:

#1 The script is FIXED for MySQL 8. It works for both MySQL and MariaDB latest versions and should still work with older versions as well.

#2 The weekly partitioning is FIXED. Thanks @beinvisible

#3 Zabbix 6 removed the auditlog_details table. The script is compatible with this version now, make sure to uncomment the fix for older Zabbix versions.

#4 For RHEL9 based systems use the CRB repository (Rocky Linux 9 specific) to get Perl-DataTime. dnf config-manager --set-enabled crb

Make sure to uncomment the correct lines (see blog post), the default is setup for MySQL 5.6 or MariaDB.

Also, see common issues at the bottom of the blog post.

Zabbix MySQL partitioning Perl script

Disclaimer: This script isn't made by us, the original author is https://github.com/dotneft and the script was initially (slightly) modified by Rihards Olups. We've added it to Github so we can maintain it and provide easy access to the entire Zabbix community.

Welcome to the Opensource ICT Solutions GitHub, where you'll find all kinds of useful Zabbix resources. This script is a script written in Perl to partition the Zabbix database tables in time based chunks. We can use this script to replace the Zabbix housekeeper process which tends to get too slow once you hit a certain database size.

With the use of MySQL partitioing using fixed History and Trend storage periods for all items we can mitigate this issue and grow our Zabbix database even further.

How to use the script

Make sure to partition the database first. If you do not know how, check out this blog post: https://blog.zabbix.com/partitioning-a-zabbix-mysql-database/13531/

Or check out our Zabbix book for a detailed description: https://www.amazon.com/Zabbix-Infrastructure-Monitoring-Cookbook-maintaining-dp-1801078327/dp/1801078327

Place the script in:

/usr/share/zabbix/

Then make it executable with:

chmod +x /usr/share/zabbix/mysql_zbx_part.pl

Now add a cronjob with:

crontab -e

Add the following line:

55 22 * * * /usr/share/zabbix/mysql_zbx_part.pl >/dev/null 2>&1

We also need to install some Perl dependencies with:

yum install perl-DateTime perl-Sys-Syslog perl-DBI perl-DBD-mysql

If perl-DateTime isn't available on your RHEL based installation make sure to install the powertools repo with:

yum config-manager --set-enabled powertools

On RHEL 9 based:

dnf config-manager --enable crb

or for genuine-RedHat:

subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms

Or Oracle Linux

dnf config-manager --set-enabled ol8_codeready_builder

On a Debian based systems (like Ubuntu) run:

apt-get install libdatetime-perl liblogger-syslog-perl libdbd-mysql-perl

MAKE SURE TO UNCOMMENT THE CORRECT LINES FOR THE VERSION YOU NEED. Check the blog post for more information.

# MySQL 5.5
# MySQL 5.6 + MariaDB
# MySQL 8.x (NOT MariaDB!)

Uncomment the following line for Zabbix 5.4 and OLDER:

#       $dbh->do("DELETE FROM auditlog_details WHERE NOT EXISTS (SELECT NULL FROM auditlog WHERE auditlog.auditid = auditlog_details.auditid)");

That's it! You are now done and you have setup MySQL partitioning. We could execute the script manually with:

perl /usr/share/zabbix/mysql_zbx_part.pl

Then we can check and see if it worked with:

journalctl -t mysql_zbx_part

Partitioning by week

NOTE: See version 2.1 for older (before 20-09-2021) partitioned databases. Otherwise use 3.0+ upwards (recommended to use current).

By default history tables are partitioned by day and trends are partitioned by month. It is also possible to partition both types of tables by week.

To do so change the period value to week under my $tables =. Also make sure to use a different naming convention for your partition names (2021_w36) and while partitioning make sure to use the correct UNIXTIMESTAMP.

The weekly partitioning setup IS NOT described in the Zabbix blog.

zabbix-mysql-partitioning-perl's People

Contributors

beinvisible avatar bvbaekel avatar larcorba avatar richlv avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

zabbix-mysql-partitioning-perl's Issues

dont run :(

mysql> select version();
+-------------------------+
| version() |
+-------------------------+
| 8.0.33-0ubuntu0.22.04.2 |
+-------------------------+
1 row in set (0.00 sec)

mysql> SELECT plugin_status FROM information_schema.plugins WHERE plugin_name ='partition';
Empty set (0.00 sec)

mysql> show variables like '%partitioning%';
Empty set (0.00 sec)

But my table is partitioning:

-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2023_01_27.ibd
-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2023_01_28.ibd
-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2023_01_29.ibd
-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2023_06.ibd
-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2023_09.ibd
-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2023_08.ibd
-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2023_07.ibd
-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2023_12.ibd
-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2023_11.ibd
-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2023_10.ibd
-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2024_02.ibd
-rw-r----- 1 mysql mysql 114688 May 15 16:25 trends#p#p2024_01.ibd
-rw-r----- 1 mysql mysql 12582912 May 15 16:25 trends#p#p2023_01_30.ibd
-rw-r----- 1 mysql mysql 18874368 May 15 16:25 trends#p#p2023_01.ibd
-rw-r----- 1 mysql mysql 176160768 May 15 16:26 trends#p#p2023_02.ibd
-rw-r----- 1 mysql mysql 197132288 May 15 16:27 trends#p#p2023_03.ibd
-rw-r----- 1 mysql mysql 192937984 May 15 16:27 trends#p#p2023_04.ibd

if i run "perl mysql_zbx_part_8.pl", this is a result.

root@zabbix-server:/etc/zabbix# perl mysql_zbx_part_8.pl
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: Cannot remove all partitions, use DROP TABLE instead at mysql_zbx_part_8.pl line 149.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: Cannot remove all partitions, use DROP TABLE instead at mysql_zbx_part_8.pl line 149.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: Cannot remove all partitions, use DROP TABLE instead at mysql_zbx_part_8.pl line 149.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: MAXVALUE can only be used in last partition definition at mysql_zbx_part_8.pl line 126.
DBD::mysql::db do failed: Cannot remove all partitions, use DROP TABLE instead at mysql_zbx_part_8.pl line 149.

Zabbix 6 auditlog

In Zabbix 6 the auditlog was reworked and auditlog_details no longer exists. I have commented this line out of the script for now and it is partitioning fine but I'm unsure if it is leaving junk in the auditlog and if that query should be reworked somehow for the new auditlog table.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.