Coder Social home page Coder Social logo

ptarmiganlabs / butler-cw Goto Github PK

View Code? Open in Web Editor NEW
11.0 7.0 8.0 665 KB

Cache warming for Qlik Sense. Proactively load Sense apps into memory in order to minimise load time and maximise user experience

Home Page: https://ptarmiganlabs.com

License: MIT License

JavaScript 97.21% Dockerfile 2.79%
qliksense senseops qlik devops

butler-cw's Introduction

A cache warming tool for Qlik Sense

Butler CW

Source Continuous Integration Project Status: Active – The project has reached a stable, usable state and is being actively developed.

CW = Cache Warming, i.e. the process of proactively forcing Sense apps to be loaded into RAM, so they are readily available when users open them. Butler CW does this for Qlik Sense Enterprise on Windows (QSEoW).

Background

Many years ago Qlik Sense Enterprise (the Windows Server version, which was the only one available at the time) moved to something called "shared persistence". It simply means that the Sense apps are stored on a central file server, rather than on each individual server in the Sense environment.
Shared persistence is today the only supported storage model for QSEoW.

The upside of shared persistence is that there is a single version of each app - the one stored on the central file server.

The downside is that when a user opens a large Sense app, it takes significant time to load the app from the file server into Qlik's associative engine (a.k.a. the QIX engine) of the server that the user is connected to.

True, the load time will depend on things like how fast disks are used on the file server, network speed etc - but even with best possible disks and servers that Amazon, Google or Microsoft offer in their cloud platforms, large apps might take several minutes to load into the engine. This means the user opening the app in question would have to wait that time before starting to use the app. Not a good user experience.

Solution

Butler CW consists of three main parts:

  • Standalone binaries for Windows, Linux and macOS. All binaries are scanned for virus and malware. The Windows and macOS binaries are signed using industry standard methods. There is also a Docker image.
  • Config file (YAML format) used by the Butler CW binary.
  • A separate config file (YAML) for specifying with what frequency what apps should be loaded into which servers.

Features

  • Control when and on what Sense server specific apps should be loaded/cache warmed.
  • Very flexible, human readable cache warming schedule definition for each app. Choose between using UTC or other local time zones.
  • Control per app if all sheets in the app should be opened. If enabled, this will effectively pre-calculate all charts in all sheets, for the default selection state in the app. This can dramatically shorten load times when an app is first accessed by end users.
  • Apps can be configured to have an initial cache warming run when Butler CW is first started.
  • Heartbeats sent to infrastructure monitoring tools. Useful if you want to monitor and ensure Butler CW is alive and well.
  • Uptime metrics (how long Butler CW has been running, how much memory it's using etc) written to log files.
  • Send data about all cache warming runs, for all apps, to MQTT for later use by subscribers of the MQTT topics.
  • Store the app schedule file in GitHub or in a file on disk.
  • Logs written to disk and console, with configurable log levels. Choose between using CET or local time zones in log files.
  • ... and more. The main config file is well documented and serves as the ultimate list of what's available in terms of features.

Command line options

Butler CW has a few settings that can be configured via the command line:

PS C:\code\butler-cw> .\butler-cw.exe --help
Usage: butler-cw [options]

Butler CW makes sure that the most important apps are always loaded in your Qlik Sense Enterprise on Windows environment.
CW = Cache Warming, i.e. the  process of proactively forcing Sense apps to be loaded into RAM memory.

Options:
  -V, --version                 output the version number
  -c, --config-file <file>      Path to config file (default: "production.yaml")
  -l, --log-level <level>       log level (choices: "error", "warn", "info", "verbose", "debug", "silly", default: "info")
  -a, --app-config-file <file>  Path to config file with cache warming definitions
  -h, --help                    display help for command
PS C:\code\butler-cw>

Configuration files

There are two config files:

Main config file

This YAML file contains general config info that Butler CW needs.

Part of the information is sensitive, for example the location of the Sense server certificates used when connecting to the Sense servers, as well as other parameters used to run the actual cache warming service. This file is stored on local disk.

The default name and location for this file is production.yaml in the same directory where the Butler CW binary resides.
If the config file is located and/or names something else the --config-file command line option can be used to specify which config file to use.

Cache warming definitions file

This YAML file contains data about which apps should be cache warmed, on what servers and how often.

The config file's location is either specified using the --app-config-file command line option or in the appConfig.diskConfigFile setting in the main config file.

Most likely there will be several people (Sense developers and admins) that need to edit this file.
For that reason this file can be stored either on local disk or on GitHub or some other GitHub-compatible revision control system.

If stored on GitHub (or other similar system), Butler CW will read the config file from there and you will get all the traceability and peer review capabilities offerd by GitHub.

The workflow looks like this:

  1. Sense developers that want their apps pre-loaded onto Sense servers fork the GitHub repository where the YAML config file lives.
  2. The developer make some additions to the YAML config file and send a pull request to the upstream repository (which is managed by the Sense admins).
  3. When the changes are accepted in the upstream/main Git repository, Butler CW will at next restart use the new config file.

There are instructions in the template file included in the GitHub repo that explains how this file works.

MQTT support

MQTT is a lightweight, robust, publish-subscribe (pub-sub) protocol used in the IoT sector and elsewhere. It's very easy to use and makes it trivial to send data from one source system to any number of destination systems ("subscribers") that are interested in this particular data.

Butler CW can be configured to send a MQTT message every time an app is cache warmed.
The configuration is done in the mqttConfig section of the main config file:

# MQTT config parameters
mqttConfig:
  out: 
    enable: true              # Should info about cache run/warming events be sent as MQTT messages?
    baseTopic: butler-cw/     # Topic to send cache run events to. Should end with /
    tzFormat: UTC             # LOCAL or UTC. Default is UTC
  broker:                     # MQTT server/broker config
    uri: mqtt://<MQTT server ip/FQDN>:<port>      ## Port is usually 1883

Installation and setup

Running as a stand-alone binary

Basically the same as for any app in the Butler family:

  1. Grab the latest release from the release page.
    Extract it to a suitable place on your Windows/Linux/Mac computer. A place like d:\tools\butler-cw could make sense on a Windows Server with a system c: drive and a d: for non-system applications.
  2. Copy the template config files (here and here) to desired location, for example a config subdirectory.
    Rename them as you see fit, for example production.yaml and apps.yaml.
  3. Edit ./config/production.yaml as needed, using paths etc specific to your local system.
  4. Edit ./config/apps.yaml, specifying when Sense apps should be loaded into servers.
    The frequency field in this config file is quite flexible, you can use any format listed here.

Note:

The appStepThroughSheets field in ./config/apps.yaml controls whether Butler CW should iterate through all sheets and chart objects in the app.
If enabled, and there are lots of sheets and charts, a lot of RAM might be used when loading the app into Sense's engine.

The user experience will on the other hand be great - sheets and the charts on them will load instantly - even those charts that previosuly took long time to render due to complex calculations and/or large data volumes.

It is impossible to give firm guidance on what levels of caching and stepping through sheet that is suitable - you have to start on a low level and work your way up until you find a solution that works in your Qlik Sense environment.

Start Butler CW by running .\butler-cw.exe --config-file .\config\production.yaml --app-config-file .\config\apps.yaml.
Here we are using PowerShell on Windows, specifying the location of the config files using the --config-file and --app-config-file command line options.

PS C:\code\butler-cw> .\butler-cw.exe --config-file .\config\production.yaml --app-config-file .\config\apps.yaml
Config file option value: C:/code/butler-cw/config/production.yaml
Config file, full path & file: C:/code/butler-cw/config/production.yaml
Config file path: C:/code/butler-cw/config
Config file name: production
Config file extension: .yaml
App config file option value: C:/code/butler-cw/config/apps.yaml
App config file, full path & file: C:/code/butler-cw/config/apps.yaml
App config file path: C:/code/butler-cw/config
App config file name: apps
App config file extension: .yaml
2023-04-13T12:17:34.060Z info: --------------------------------------
2023-04-13T12:17:34.060Z info: Starting Butler CW.
2023-04-13T12:17:34.060Z info: Log level is: info
2023-04-13T12:17:34.076Z info: App version is: 4.2.0
2023-04-13T12:17:34.076Z info: --------------------------------------
2023-04-13T12:17:34.107Z info: MAIN: Started Docker healthcheck server on port 12398.
2023-04-13T12:17:34.169Z info: -------------------------------------------------------------------------
2023-04-13T12:17:34.169Z info: First runs for app e28f9c40-6138-42f4-81c3-1d61860baa27, "Parking tickets by city":
2023-04-13T12:17:34.169Z info: 1: Thu, 13 Apr 2023 17:00:00 GMT
2023-04-13T12:17:34.169Z info: 2: Fri, 14 Apr 2023 17:00:00 GMT
2023-04-13T12:17:34.169Z info: 3: Sat, 15 Apr 2023 17:00:00 GMT
2023-04-13T12:17:34.169Z info: 4: Sun, 16 Apr 2023 17:00:00 GMT
2023-04-13T12:17:34.169Z info: 5: Mon, 17 Apr 2023 17:00:00 GMT
2023-04-13T12:17:34.169Z info: 6: Tue, 18 Apr 2023 17:00:00 GMT
2023-04-13T12:17:34.185Z info: 7: Wed, 19 Apr 2023 17:00:00 GMT
2023-04-13T12:17:34.185Z info: 8: Thu, 20 Apr 2023 17:00:00 GMT
2023-04-13T12:17:34.185Z info: 9: Fri, 21 Apr 2023 17:00:00 GMT
2023-04-13T12:17:34.185Z info: 10: Sat, 22 Apr 2023 17:00:00 GMT
2023-04-13T12:17:34.185Z info: -------------------------------------------------------------------------
2023-04-13T12:17:34.185Z info: First runs for app c840670c-7178-4a5e-8409-ba2da69127e2, "Meetup.com":
2023-04-13T12:17:34.185Z info: 1: Thu, 13 Apr 2023 12:20:00 GMT
2023-04-13T12:17:34.185Z info: 2: Thu, 13 Apr 2023 12:40:00 GMT
2023-04-13T12:17:34.185Z info: 3: Thu, 13 Apr 2023 13:00:00 GMT
2023-04-13T12:17:34.185Z info: 4: Thu, 13 Apr 2023 13:20:00 GMT
2023-04-13T12:17:34.185Z info: 5: Thu, 13 Apr 2023 13:40:00 GMT
2023-04-13T12:17:34.185Z info: 6: Thu, 13 Apr 2023 14:00:00 GMT
2023-04-13T12:17:34.185Z info: 7: Thu, 13 Apr 2023 14:20:00 GMT
2023-04-13T12:17:34.185Z info: 8: Thu, 13 Apr 2023 14:40:00 GMT
2023-04-13T12:17:34.185Z info: 9: Thu, 13 Apr 2023 15:00:00 GMT
2023-04-13T12:17:34.201Z info: 10: Thu, 13 Apr 2023 15:20:00 GMT
2023-04-13T12:17:39.207Z info: Starting: initial warming of app e28f9c40-6138-42f4-81c3-1d61860baa27, "Parking tickets by city"
2023-04-13T12:17:39.207Z info: Done: initial warming of app e28f9c40-6138-42f4-81c3-1d61860baa27, "Parking tickets by city"
2023-04-13T12:17:39.207Z info: Starting: initial warming of app c840670c-7178-4a5e-8409-ba2da69127e2, "Meetup.com"
2023-04-13T12:17:39.207Z info: Done: initial warming of app c840670c-7178-4a5e-8409-ba2da69127e2, "Meetup.com"
2023-04-13T12:17:39.688Z info: App loaded: e28f9c40-6138-42f4-81c3-1d61860baa27
2023-04-13T12:17:39.704Z info: App e28f9c40-6138-42f4-81c3-1d61860baa27: Cached 0 visualizations on 0 sheets.
2023-04-13T12:17:39.704Z warn: Next cache warming for app e28f9c40-6138-42f4-81c3-1d61860baa27: Fri, 14 Apr 2023 17:00:00 GMT

Running as a Windows service

Use the excellent nssm tool to install Butler CW as a service.

Instructions are available on the nssm site.

Running in a Docker container

If you have access to a Docker or Kubernetes environment this is a good option for running Butler CW.

  • Make use of your existing container infrastructure, or use those offered by Amazon, Google, Microsoft etc.
  • Benefit from the extremely comprehensive tools ecosystem (monitoring, deployment etc) that is available for Docker/Kubernetes.
  • Updating Butler CW to the latest version is as easy as stopping the container, then doing a "docker pull ptarmiganlabs/butler:latest", and finally starting the container again.

Installing and getting started with Butler CW in Docker can look something like this when working on MacOS. Windows and Linux of course looks slightly different.

Create a directory for Butler CW. Config files and logs will be stored here.

proton:code goran$ mkdir -p butler-cw-docker/config/certificate
proton:code goran$ mkdir -p butler-cw-docker/log
proton:code goran$ cd butler-cw-docker
proton:butler-cw-docker goran$
  • Copy the two YAML config files from the GitHub repository, rename and edit them as described above and place them in the ./config directory.
  • Copy docker-compose.yml from the GitHub repository to the main directory.
  • Export certifiates from the QMC in Qlik Sense Enterprise, place them in the config/certificate directory.

Let's do this one step at a time.
What files are there?

proton:butler-cw-docker goran$ ls -la
total 8
drwxr-xr-x   5 goran  staff   160 Sep 27 22:21 .
drwxr-xr-x  47 goran  staff  1504 Sep 27 10:41 ..
drwxr-xr-x   5 goran  staff   160 Sep 27 10:43 config
-rw-r--r--   1 goran  staff   383 Sep 27 22:21 docker-compose.yml
drwxr-xr-x   2 goran  staff    64 Sep 27 22:21 log
proton:butler-cw-docker goran$
proton:butler-cw-docker goran$ ls -la config/
total 16
drwxr-xr-x  5 goran  staff   160 Sep 27 10:43 .
drwxr-xr-x  5 goran  staff   160 Sep 27 22:21 ..
-rw-r--r--  1 goran  staff   855 Sep 27 22:22 apps.yaml
drwxr-xr-x  5 goran  staff   160 Sep 27 10:43 certificate
-rw-r--r--  1 goran  staff  1140 Sep 27 22:22 production.yaml
proton:butler-cw-docker goran$
proton:butler-cw-docker goran$ ls -la config/certificate/
total 24
drwxr-xr-x  5 goran  staff   160 Sep 27 10:43 .
drwxr-xr-x  5 goran  staff   160 Sep 27 10:43 ..
-rw-r--r--@ 1 goran  staff  1166 Sep 27 10:43 client.pem
-rw-r--r--@ 1 goran  staff  1702 Sep 27 10:43 client_key.pem
-rw-r--r--@ 1 goran  staff  1192 Sep 27 10:43 root.pem

What does a real-world config file look like?
Here's one for Docker (the only difference is how the certificate files are referenced.)

proton:butler-cw-docker goran$ cat config/production.yaml
# Rename this file to production.yaml, and fill in data as needed below.

# Logging configuration
logLevel: info          # Log level. Valid log levels are silly, debug, verbose, info, warn, error
fileLogging: true       # true/false to enable/disable logging to disk file

# Configuration of Butler CW's scheduler
scheduler:
  startup: 
    showPerAppSchedule: 
      enable: true          # Should the first itemCount scheduled runs be shown for each app, on startup?
      itemCount: 10         # Number of coming runs to show for each app
  timeZone:                 # Valid values are UTC and LOCAL. Default value is UTC
    scheduleDefine: UTC     # How should times in the apps config file be interpreted? 
    logs: UTC               # What time format should be used in log files?
  showNextEvent:        
    enable: true            # Should date/time of next cache warming event be sent to log file? (true/false) 
    logLevel: warn       # Log level to use for the next event log entry. Valid values are silly, debug, verbose, info, warn, error
    tzFormat: UTC           # LOCAL or UTC. Default is UTC

# Heartbeats can be used to send "I'm alive" messages to any other tool, e.g. a infrastructure monitoring tool
# The concept is simple: The remoteURL will be called at the specified frequency. The receiving tool will then know 
# that Butler CW is alive.
heartbeat:
  enabled: true
  remoteURL: https://healthcheck.ptarmiganlabs.net/ping/138514b0-882a-4a44-8548-96f7d16c9242
  # frequency: every 1 minute     # https://bunkat.github.io/later/parsers.html
  frequency: every 10 seconds     # https://bunkat.github.io/later/parsers.html

# Docker health checks are used when running Butler CW as a Docker container. 
# The Docker engine will call the container's health check REST endpoint with a set interval to determine
# whether the container is alive/well or not.
# If you are not running Butler CW in Docker you can safely disable this feature. 
dockerHealthCheck:
  enabled: true    # Control whether a REST endpoint will be set up to serve Docker health check messages
  port: 12398      # Port the Docker health check service runs on (if enabled)

# Uptime monitor
# When enabled, Butler CW will write info on uptime and used memory to log files
uptimeMonitor:
  enabled: true                   # Should uptime messages be written to the console and log files?
  frequency: every 10 seconds     # https://bunkat.github.io/later/parsers.html
  logLevel: verbose               # Starting at what log level should uptime messages be shown?

# Paths to client certificates to use when connecting to Sense server. Can be pem or pvk/cer
# remove or comment out if running on desktop
clientCertPath: /nodeapp/config/certificate/client.pem
clientCertKeyPath: /nodeapp/config/certificate/client_key.pem
clientCertCAPath: /nodeapp/config/certificate/root.pem

# MQTT config parameters
mqttConfig:
  out: 
    enable: true              # Should info about cache run/warming events be sent as MQTT messages?
    baseTopic: butler-cw/     # Topic to send cache run events to. Should end with /
    tzFormat: UTC             # LOCAL or UTC. Default is UTC
  broker:                     # MQTT server/broker config
    uri: mqtt://1.2.3.4:1883       ## Port is usually 1883

# QIX version to use
qixVersion: 12.170.2

# Config on what apps should be cached
appConfig:
  # Valid options are disk, github
  configSource: disk
  
  # Leave strings empty if disk config not used
  diskConfigFile: ./config/apps.yaml

  # Leave strings empty if github not used
  github:
    host: api.github.com
    username: <username>
    password: pwd or access token
    owner: <repo owner>
    repo: <repo name, e.g. qliksense-cache-warming>
    path: config/apps.yaml

# Is connection to Sense engine secure (https)?
isSecure: true

proton:butler-cw-docker goran$
proton:butler-cw-docker goran$ cat config/apps.yaml
# Rename this file to apps.yaml, then edit it as needed.

# This file is used to specifiy what Qlik Sense apps should be loaded by the cache warmer, when/how often they should be loaded, 
# what filters should be applied when apps are opened etc 
#
# Frequency attribute must follow rules described here: https://breejs.github.io/later/parsers.html#text
#
# NOTE!
# 1. Second level scheduling does not work as expected. 
#    For example, 'every 55 seconds' will trigger on each whole minute, plus 5 seconds before each whole minute.
#    Use minutes as most detailed level of scheduling. 
#    More info at https://github.com/ptarmiganlabs/butler-cw/issues/116
#
#
# Fields for each app:
# server: IP or FQDN of the Sense server where the app should be loaded
# appId: Id of app to load
# appDescription: Free text description of the app. Can be anything - no check is done Sense APIs using this text
# appStepThroughSheets: Set to true to have Butler CW step through all sheets of the app, triggering all charts etc to calculate in all sheets
# doInitialLoad: Set to true to do an initial load of this app when Butler CW is started.
# freq: Text value representation of how often the app should be loaded

apps:
  - server: sense1.mydomain.com
    appId: c36bfbdb-0c4b-4d57-9939-b851d2af1cb5
    appDescription: License monitor
    appStepThroughSheets: true
    doInitialLoad: true
    freq: every 5 mins every weekend
  - server: sense1.mydomain.com
    appId: dead6f4a-da0b-4b9c-82a2-3f94fdc72599
    appDescription: Meetup.com
    appStepThroughSheets: true
    doInitialLoad: false
    freq: at 5:00 pm
  - server: sense2.mydomain.com
    appId: 492a1bca-1c41-4a01-9104-543a2334c465
    appDescription: 2018 sales targets
    appStepThroughSheets: true
    doInitialLoad: false
    freq: every 2 hours
proton:butler-cw-docker goran$

What does the docker-compose.yml file look like?

proton:butler-cw-docker goran$ cat docker-compose.yml
# docker-compose.yml
version: '3.3'
services:
  butler-cw:
    image: ptarmiganlabs/butler-cw:latest
    restart: always
    container_name: butler-cw
    volumes:
      # Make config file accessible outside of container
      - "./config:/nodeapp/config"
      - "./log:/nodeapp/log"
    environment:
      - "NODE_ENV=production"
    logging:
      driver: json-file
      options:
        max-file: "5"
        max-size: "5m"
    networks:
      - butler-cw

networks:
  butler-cw:
    driver: bridge

proton:butler-cw-docker goran$

Ok, all good. Let's start Butler CW using docker-compose:

proton:butler-cw-docker goran$ docker-compose up
Creating network "butler-cw-docker_butler-cw" with driver "bridge"
Pulling butler-cw (ptarmiganlabs/butler-cw:latest)...
latest: Pulling from ptarmiganlabs/butler-cw
7d63c13d9b9b: Already exists
bb262aff53d8: Already exists
24467fa1084c: Already exists
d318401bbcfd: Already exists
fef5c41ac380: Already exists
355f82e6ba49: Pull complete
58031546d982: Pull complete
49882da52baa: Pull complete
990f8e00f68e: Pull complete
b33e3e1b2464: Pull complete
Digest: sha256:064508343e754fd2a37e0222e37d016309ba4a96c73c519cac11db5f1bb784ee
Status: Downloaded newer image for ptarmiganlabs/butler-cw:latest
Creating butler-cw ... done
Attaching to butler-cw
butler-cw    | 2021-10-20T12:23:15.419Z info: --------------------------------------
butler-cw    | 2021-10-20T12:23:15.421Z info: Starting Butler CW.
butler-cw    | 2021-10-20T12:23:15.422Z info: Log level is: verbose
butler-cw    | 2021-10-20T12:23:15.422Z info: App version is: 3.1.0
butler-cw    | 2021-10-20T12:23:15.423Z info: --------------------------------------
butler-cw    | 2021-10-20T12:23:15.430Z verbose: MAIN: Starting Docker healthcheck server...
butler-cw    | 2021-10-20T12:23:15.449Z info: MAIN: Started Docker healthcheck server on port 12398.
butler-cw    | 2021-10-20T12:23:20.491Z verbose: Starting loading of appid e28f9c40-6138-42f4-81c3-1d61860baa27
butler-cw    | 2021-10-20T12:23:20.502Z verbose: Starting loading of appid c840670c-7178-4a5e-8409-ba2da69127e2
butler-cw    | 2021-10-20T12:23:20.750Z info: App loaded: e28f9c40-6138-42f4-81c3-1d61860baa27
butler-cw    | 2021-10-20T12:23:20.766Z info: App e28f9c40-6138-42f4-81c3-1d61860baa27: Cached 0 visualizations on 0 sheets.
butler-cw    | 2021-10-20T12:23:20.792Z info: App loaded: c840670c-7178-4a5e-8409-ba2da69127e2
butler-cw    | 2021-10-20T12:23:21.424Z info: App c840670c-7178-4a5e-8409-ba2da69127e2: Cached 13 visualizations on 2 sheets.
butler-cw    | 2021-10-20T12:23:30.486Z verbose: --------------------------------
butler-cw    | 2021-10-20T12:23:30.488Z verbose: Iteration # 1, Uptime: 15 seconds, Heap used 12.88 MB of total heap 15.76 MB. Memory allocated to process: 63.12 MB.
butler-cw    | 2021-10-20T12:23:30.490Z verbose: Starting loading of appid e28f9c40-6138-42f4-81c3-1d61860baa27
butler-cw    | 2021-10-20T12:23:30.637Z info: App loaded: e28f9c40-6138-42f4-81c3-1d61860baa27
butler-cw    | 2021-10-20T12:23:30.649Z info: App e28f9c40-6138-42f4-81c3-1d61860baa27: Cached 0 visualizations on 0 sheets.
butler-cw    | 2021-10-20T12:23:45.468Z verbose: --------------------------------
butler-cw    | 2021-10-20T12:23:45.470Z verbose: Iteration # 2, Uptime: 29 seconds, Heap used 13.45 MB of total heap 16.01 MB. Memory allocated to process: 63.77 MB.
...
...

Warning: Setting the log level to debug in the config file will create lots of log output.

Finally, let's take a look at what Docker tells us about the currently running containers:

➜  ~ docker ps
CONTAINER ID   IMAGE                            COMMAND                  CREATED         STATUS                    PORTS     NAMES
565152b1be90   ptarmiganlabs/butler-cw:latest   "docker-entrypoint.s…"   2 minutes ago   Up 20 seconds (healthy)             butler-cw
➜  ~

Great, Butler CW is running and reporting a healthy status. All good!

Storing configuration in Git

While a good idea, this is a slightly complex topic.

Different revision control tools and services (Github, BitBucket etc) work in slightly different ways when it comes to allowing access to files.

In public Github (GH), it is for example possible to read files in public repositories without being logged into Github.

In Github Enterprise (GHE) on the other hand, each company can configure whether their own employees should be able to read files from GHE repositories without being logged into GHE, or whether such access requires the user to be logged into GHE.

Bottom line is that Butler CW in current version logs into GH or GHE, and then read the apps.yaml file. The rationale is that most companies will not place their config files on public GH, but rather in GHE or other privat revision control system that require login.

It is worth noting that Butler CW can quite easily be extended to support other revision control tools/solutions.

Remember!

You cannot pre-load all apps. Focus on a few of the most used apps, and/or some of the biggest ones, where the impact of cache warming will be the greatest.
Also, remember to do an occasional review what apps are being used, and if needed adjust the cach warming strategy. No point in forcing apps that only a few people use into RAM.

Strange stuff, caveats and things not quite working (yet)

The projects issue list is where bugs and feature requests should go, but there can also be things that aren't really bugs, but still don't work quite as you might expect.. Maybe they will go onto the bug list at some point - TBD.

  • If the config setting scheduler.showNextEvent.enable is enabled, Butler will right after doing a cache run show when the next cache run will take place.
    This can be nice if you want to ensure that the schedule you created works as expected.
    The caveat is: For the first cache run it sometimes displays an incorrect time for the second schedule run. Maybe it should be classified as a bug.. but it looks like it's tricky to fix, and the impact is really quite minor as it only affects what's shown in the logs right after Butler CW is started.

Links & references

GitHub repository: https://github.com/ptarmiganlabs/butler-cw
Docker image on Docker Hub: https://hub.docker.com/u/ptarmiganlabs

More info about the Butler family of Qlik Sense utilities can be found at the Ptarmigan Labs site, in the GitHub repositories or in the Butler SOS site.

Inspiration

Inspiration to this project largely came from Joe Bickley's Cache Initializer project.
At the core Joe's tool does the same as Butler CW, but for individual apps.

I wanted a stand-alone tool that by itself handles multiple apps on multiple servers. Also, as the rest of the Butler suite is written in Node.js, I wanted the cache warmer to use the same underlying tech as the rest of the Butler family.

Joe's tool does one thing Butler CW does not do: Allow filters to be applied when opening the app. This is a useful feature and might be included in future versions of Butler CW.

butler-cw's People

Contributors

dependabot[bot] avatar erikwett avatar github-actions[bot] avatar mountaindude avatar nungster avatar renovate-bot avatar renovate[bot] avatar snyk-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

butler-cw's Issues

Bug in schedule

Describe the bug
I started butler-cw with parameter "freq: at 11:15 am" in config.
Bug 1 - Warming cache started immediately
Bug 2 - It use UTC time(in logs too), not local (maybe it is feature?)

To Reproduce
Use parameter "freq: at 11:15 am" in config

Expected behavior

  1. Start according to schedule.
  2. Use local time(?)

Describe environment:

  • OS: Debian 10
  • Containerisation no
  • Version of Butler CW 3.1.1

Config file(s)
...
freq: at 11:15 am

Multi-arch Docker manifest is not working as intended

Describe the bug
When running in a multi-arch (amd64 and arm64) k8s cluster, containers fail to start on arm64 nodes.

Expected behavior
Containers should automatically start on amd64, arm64 or arm nodes.

Solution idea
The issue might be caused by the variant field in the manifest. It seems that the manifests of other (working) multi-arch images Docker images don't have that variant field. Might be worth removing it as a first attempt at resolving the issue.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

  • fix(deps): update dependency mqtt to v4.3.8
  • fix(deps): update dependency @breejs/later to v4.2.0
  • fix(deps): update dependency commander to v11.1.0
  • fix(deps): update dependency enigma.js to v2.14.0
  • fix(deps): update dependency eslint to v8.57.0
  • fix(deps): update dependency fs-extra to v11.2.0
  • fix(deps): update dependency moment to v2.30.1
  • fix(deps): update dependency ws to v8.17.1
  • chore(deps): update actions/checkout action to v4
  • chore(deps): update actions/setup-node action to v4
  • chore(deps): update crazy-max/ghaction-virustotal action to v4
  • chore(deps): update docker/build-push-action action to v5
  • chore(deps): update docker/login-action action to v3
  • chore(deps): update docker/metadata-action action to v5
  • chore(deps): update docker/setup-buildx-action action to v3
  • chore(deps): update docker/setup-qemu-action action to v3
  • chore(deps): update github artifact actions to v4 (major) (actions/download-artifact, actions/upload-artifact)
  • chore(deps): update github/codeql-action action to v3
  • chore(deps): update googlecloudplatform/release-please-action action to v4
  • chore(deps): update node.js to v20
  • fix(deps): update dependency commander to v12
  • fix(deps): update dependency eslint to v9
  • fix(deps): update dependency eslint-config-prettier to v9
  • fix(deps): update dependency winston-daily-rotate-file to v5
  • 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

docker-compose
docker-compose.yml
dockerfile
Dockerfile
  • node 19-bullseye-slim
github-actions
.github/workflows/codeql-analysis.yml
  • actions/checkout v3
  • github/codeql-action v2
  • github/codeql-action v2
  • github/codeql-action v2
.github/workflows/debugmqtt.yaml
  • potaesm/github-actions-mqtt-request 1.0.0
.github/workflows/docker-image-build.yml
  • potaesm/github-actions-mqtt-request 1.0.0
  • actions/checkout v3
  • docker/setup-qemu-action v2
  • docker/setup-buildx-action v2
  • docker/login-action v2
  • docker/metadata-action v4
  • docker/build-push-action v4
  • potaesm/github-actions-mqtt-request 1.0.0
.github/workflows/release-please.yml
  • GoogleCloudPlatform/release-please-action v3
  • actions/checkout v3
  • github/codeql-action v2
  • crazy-max/ghaction-virustotal v3
  • actions/upload-artifact v3
  • actions/upload-artifact v3
  • actions/upload-artifact v3
  • actions/checkout v3
  • actions/setup-node v3
  • actions/download-artifact v3
  • ncipollo/release-action v1
  • actions/checkout v3
  • actions/setup-node v3
  • actions/download-artifact v3
  • ncipollo/release-action v1
  • actions/checkout v3
  • actions/setup-node v3
  • actions/download-artifact v3
  • ncipollo/release-action v1
.github/workflows/virus-scan.yml
  • crazy-max/ghaction-virustotal v3
npm
package.json
  • @breejs/later ^4.1.0
  • @octokit/rest ^19.0.13
  • axios ^1.4.0
  • commander ^11.0.0
  • config ^3.3.9
  • enigma.js ^2.11.0
  • eslint ^8.43.0
  • eslint-config-airbnb-base 15.0.0
  • eslint-config-prettier ^8.8.0
  • eslint-plugin-import ^2.27.5
  • eslint-plugin-prettier ^4.2.1
  • fastify ^4.18.0
  • fastify-healthcheck ^4.4.0
  • fs-extra ^11.1.1
  • js-yaml ^4.1.0
  • moment ^2.29.4
  • moment-precise-range-plugin ^1.3.0
  • mqtt ^4.3.7
  • upath ^2.0.1
  • winston ^3.9.0
  • winston-daily-rotate-file ^4.7.1
  • ws ^8.13.0
  • prettier ^2.8.8

  • Check this box to trigger a request for Renovate to run again on this repository

enigmaOpen error: {}

Hello,
First of all, thanks so much for this work, it's very helpful. I am running into an issue shown in the attached image. Basically, when an app is scheduled to run, at every specified time it runs, there is an enigmaOpen error, but with no message:

error: enigmaOpen error: {}

Can you help me or at least explain to me why this is happening / if you've seen this before? The error is thrown at line 56:

 try {
        g = await s.open();
    } catch (err) {
        globals.logger.log('error', `enigmaOpen error: ${JSON.stringify(err)}`);
        return;
    }

enigmaError

The strange thing is that if doInitialLoad is set to "true" - it runs the first time successfully, but everything on the scheduler does not with the above message.

I have tried running natively as well as in docker, but the same issue happens.

Non-essential files included in Docker image

Describe the bug
In the spirit of keeping the Docker image nice and small, it should only include files needed for running Butler CW.
Specifically, files only needed during development should not be included.

Log uptime and memory usage to InfluxDB

Store Butler CW's memory usage to InfluxDB, similar to how Butler SOS does this.

This will require a fair amount of new code in Butler CW, as well as new options in the main config file.
Something like this:

  # Uptime monitor
  uptimeMonitor:
    enabled: true                   # Should uptime messages be written to the console and log files?
    frequency: every 15 seconds     # https://bunkat.github.io/later/parsers.html
    logLevel: verbose               # Starting at what log level should uptime messages be shown?
    influxdb: 
      enabled: true    # Should data on Butler CW's own memory use be stored in Influxdb?
      hostIP: <IP or FQDN of Influxdb server>
      hostPort: <Port where Influxdb is listening>    # Optional. Default value=8086
      auth:
        enable: false                 # Does influxdb instance require authentication (true/false)?
        username: <username>          # Username for Influxdb authentication. Mandatory if auth.enable=true
        password: <password>          # Password for Influxdb authentication. Mandatory if auth.enable=true
      dbName: SenseOps
      # Default retention policy that should be created in InfluxDB when Butler CW creates a new database there. 
      # Any data older than retention policy threshold will be purged from InfluxDB.
      retentionPolicy:
        name: 2_weeks
        duration: 2w
        instanceTag: DEV              # Tag that can be used to differentiate data from multiple Butler CW instances

Trigger cache runs via REST API

Is your feature request related to a problem? Please describe.
New feature idea:

Make it possible to trigger cache warm runs by calling a REST API that's exposed by Butler CW.
Settings in the main and schedule config files should be respect, the API call would just be a way to temporarily override schedules defined in the schedule config file.

Switch to using fastify-healthcheck for doing Docker health checks

The current way of doing Docker health checks in Butler CW uses a soon-to-be-deprecated feature in the Fastify library.

Instead do things the same way as in the Butler SOS project, using the fastify-healthcheck npm library.
Code examples available in Butler SOS repo.

Start cache warm after successful task realod

Is your feature request related to a problem? Please describe.
Some apps are updated every day at different times

Describe the solution you'd like
Is there any way to warm up the app after it reload?

Output basic Sense info to logs on Butler startup

It would be nice to get some basic info about the underlying Qlik Sense environment written to the Butler logs, whenever Butler is started.
Things such as overall Sense version, QIX engine version, info about the hardware Sense is running on etc.

error

Hi,
I configured everything as per the instruction. but getting some error while trying to run node index.Js

error

these two i didnt fill

"owner": "",
"repo": "<repo name, e.g. qliksense-cache-warming>",

Error when Running node index.js

Hello,
I get this error whenever I run "node index.js". Can you provide any insight? And please let me know if you need more information.

PS D:\butler-cw-master> node index.js
2019-03-29T12:40:07.301Z info: Starting Qlik Sense cache warmer.
2019-03-29T12:40:10.337Z verbose: Starting loading of appid a099cb6b-53fb-4e70-b213-2832073b0c02
(node:7508) UnhandledPromiseRejectionWarning: TypeError: Converting circular structure to JSON
at JSON.stringify ()
at loadAppIntoCache (D:\butler-cw-master\index.js:211:57)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:7508) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a
promise which was not handled with .catch(). (rejection id: 1)
(node:7508) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a
non-zero exit code.

Error Installing on Windows Server 2016

I am trying to install butler-cw on a Windows 2016 server and Qlik Sense June 2018. The Qix engine is 12.120.7 (edited js file accordingly). When I go to run it, I get:
C:\butler-cw-master>node index.js
2018-08-31T23:22:48.919Z - info: Starting Qlik Sense cache warmer.
module.js:478
throw err;
^

Error: Cannot find module 'enigma.js/schemas/qix/12.170.2/schema.json'
at Function.Module._resolveFilename (module.js:476:15)
at Function.Module._load (module.js:424:25)
at Module.require (module.js:504:17)
at require (internal/module.js:20:19)
at Object. (C:\butler-cw-master\index.js:45:19)
at Module._compile (module.js:577:32)
at Object.Module._extensions..js (module.js:586:10)
at Module.load (module.js:494:32)
at tryModuleLoad (module.js:453:12)
at Function.Module._load (module.js:445:3)
at Module.runMain (module.js:611:10)
at run (bootstrap_node.js:394:7)
at startup (bootstrap_node.js:160:9)
at bootstrap_node.js:507:3

Use tags and custom properties to specify which app(s) should be included in a cache warming

Is your feature request related to a problem? Please describe.
New feature idea.
Currently the apps.yaml schedule file uses app IDs to determine which apps to cache warm.
That could be done using tags and/or custom properties instead.
This would be good as you would no longer need to keep track of app IDs, but rather control cache warming by more easy to remember tags and custom properties.

For tags, the solution could be:

  • Let's assume there is a tag (available for apps) called "cachewarm-sales".
  • Assign that tag to the sales apps that should be cache warmed.
  • Enter that tag name in Butler CW's schedule file instead of the app's IDs.
  • Butler CW will cache warm those apps according to the schedule in the config file.

For custom properties, the solution could look like this:

  • In the schedule config file, for an entry you enter a) a custom property name, and b) the value which Butler CW should look for.
  • If Butler CW finds any apps with the correct custom property/value combo, those apps will be cache warmed according to the schedule in the config file.

Having an issue

Hello,

Receiving follow error, Could you please check it for me

C:\butler-cw-master>node index.js
WARNING: No configurations found in configuration directory:C:\butler-cw-master\config
WARNING: To disable this warning set SUPPRESS_NO_CONFIG_WARNING in the environment.
C:\butler-cw-master\node_modules\config\lib\config.js:203
throw new Error('Configuration property "' + property + '" is not defined');
^

Error: Configuration property "defaultLogLevel" is not defined
at Config.get (C:\butler-cw-master\node_modules\config\lib\config.js:203:11)
at Object. (C:\butler-cw-master\index.js:19:27)
at Module._compile (internal/modules/cjs/loader.js:688:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:699:10)
at Module.load (internal/modules/cjs/loader.js:598:32)
at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
at Function.Module._load (internal/modules/cjs/loader.js:529:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:741:12)
at startup (internal/bootstrap/node.js:285:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:739:3)

Error after upgrading to Qliksense April 2020 version

After upgrading QS server april 2020 , cache warmer stopped working and showing up this error
2020-05-21T22:31:05.910Z error: enigmaOpen error: {"target":{"_events":{},"_eventsCount":4,"readyState":3,"protocol":"","_binaryType":"nodebuffer","_closeFrameReceived":false,"_closeFrameSent":false,"_closeMessage":"","_closeTimer":null,"_closeCode":1006,"_extensions":{},"_isServer":false,"_receiver":null,"_sender":null,"_socket":null,"url":"wss://server.domain.corp:4747/app/ba4ca9f6-1935-4509-8a8e-501b3d19d44d","_req":null},"type":"error","message":"read ECONNRESET","error":{"errno":"ECONNRESET","code":"ECONNRESET","syscall":"read"}}

Show first few scheduled runs for each app when Butler CW starts

Is your feature request related to a problem? Please describe.
On/off flag in config file to control whether or not to show when each app's first few runs will take place.

Addition to main config file:

scheduler:
  startup: 
    showPerAppSchedule: 
      enable: true        # Should the first itemCount scheduled runs be shown for each app, on startup?
      itemCount: 10

Disable uptime monitor and mqtt support by default

Describe the bug
The default config file should have all non-critical features turned off by default.
These features can then be enabled as needed when installing Butler CW.

Specifically, disable (set to false) the following features in the template config file:
mqttConfig.out.enable: false
uptimeMonitor.enabled: false

Sub-minute scheduling does not work

Describe the bug
Having a schedule of every 55 seconds results in this execution plan:

2021-10-21T14:53:55.945Z
2021-10-21T14:54:00.945Z
2021-10-21T14:54:55.945Z
2021-10-21T14:55:00.945Z
2021-10-21T14:55:55.945Z
2021-10-21T14:56:00.945Z
2021-10-21T14:56:55.945Z
2021-10-21T14:57:00.945Z
2021-10-21T14:57:55.945Z
2021-10-21T14:58:00.945Z

Which is not what's expected.

This is however happening due to how the underlying scheduling library works (discussion here).
For now this will not be fixed in Butler CW. A proper/better implementation may be added in future versions of Butler CW.

connectivity problem

Hi Göran I try to install the butler-cw on my Server and have a Problem with connectivity please check the image

image

Heartbeats sent to https endpoints fail

Describe the bug
If heartbeats are enabled and sent to a https endpoint, the heartbeat call will fail with a "...unable to verify the first certificate" error.

Expected behavior
It should be possible to send heartbeats to either http or https endpoints.

Describe environment:
Bug verified in Node.js 14.5 on mac OS Big Sur, Docker and in Kubernetes.

Send MQTT message when cache warming event takes place

Is your feature request related to a problem? Please describe.
Feature.

MQTT is incredibly useful as an integration tool. Thus:
When an app is loaded into a server, send a MQTT message to a configurable MQTT broker and topic.

Slim down release packages to only include what's needed to run Butler CW

Is your feature request related to a problem? Please describe.
Improvement.

Currently the automatically created release packages includes quite a few files that are only used during development of Butler CW.
These files probably more than anything else confuse first-time users of Butler CW.

Suggestion is to only include files needed for standalone execution of Butler CW in release ZIP artefact.

Throwing broken promise warning with MQTT disabled

Describe the bug
I got the following Warning when running Butler-CW with MQTT disabled. I looked into it, and it looks in loadAppIntoCache it doesn't check for whether or not MQTT is enabled when it runs mqttClient.publish() on line 184. I think it needs to have the sort of if statement checking if mqttConfig.out.enable is set that we see on line 331.

(node:15104) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'publish' of undefined
    at [REDACTED]\index.js:184:24
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
(node:15104) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:15104) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

To Reproduce
Disable MQTT and don't put a valid URL in mqttconfig.broker.uri, then try to cache-warm an app.

Expected behavior
Ideally with the MQTT server disabled it wouldn't attempt to publish to it and we wouldn't get this warning.

Describe environment:

  • OS: Windows Server 2019
  • Containerisation - No
  • Version of Butler CW used - 4.0.2
  • Command used to start Butler CW - node index.js

Config file(s)

mqttConfig:
  out:
    enable: false
    tzFormat: UTC
    baseTopic: butler-cw/
  broker:
    uri: mqtt://0.0.0.0

Make Winston logger handle errors properly

The logging module of Butler CW does not currently handle error logging as well as it should.
Specifically, in some cases there is no stack trace or error message when errors occur.

This can be easily fixed by adding the following to the Winston.createLogger call:

winston.format.errors({ stack: true })

Replace later with @breejs/later

Is your feature request related to a problem? Please describe.
The later library (which is used by Butler CW) is no longer maintained.
There's however a drop-in replacement in the form of @breejs/later.

Switch Butler CW to use the latter instead of the former.

Make it possible to disable the Docker health check service of Butler CW

Add config option and corresponding code in Butler CW:

  # Docker health checks are used when running Butler CW as a Docker container. 
  # The Docker engine will call the container's health check REST endpoint with a set interval to determine
  # whether the container is alive/well or not.
  # If you are not running Butler CW in Docker you can safely disable this feature. 
  dockerHealthCheck:
    enabled: true         # Control whether a REST endpoint will be set up to serve Docker health check messages
    port: 12398           # Port the Docker health check service runs on (if enabled)

Trigger cache runs by incoming MQTT messages

Is your feature request related to a problem? Please describe.
New feature idea:

Make it possible to trigger cache warm runs by sending MQTT messages to Butler CW.
Settings in the main and schedule config files should be respect, the MQTT message would just be a way to temporarily override schedules defined in the schedule config file.

getLayout error

Describe the bug
I don't know, maybe it is not a bug.

Screenshots
image

Describe environment:

  • OS: Debian 10
  • Containerisation no
  • Version of Butler CW used 2.3.12

Add uptime monitor, log to log files

Add config options and associated code to Butler CW:

  # Uptime monitor
  uptimeMonitor:
    enabled: true                   # Should uptime messages be written to the console and log files?
    frequency: every 15 seconds     # https://bunkat.github.io/later/parsers.html
    logLevel: verbose               # Starting at what log level should uptime messages be shown?

Add container health check

Expose health check endpoints from container, to allow container supervisors to better monitor the state of the container.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.