informatiqal / automatiqal-cli

Automate Qlik Sense deployments in descriptive format

Automatiqal CLI (Beta)

Warning UNDER DEVELOPMENT

Automatiqal CLI is a NodeJS wrapper around Automatiqal package that allows automating Qlik Sense administration/deployment tasks by describing them in yaml/json files.

As the name suggests Automatiqal CLI is a command line/terminal tool.

Note At the moment only Qlik Sense Enterprise on Windows is supported! SaaS support is on the roadmap

Installation

Install as global module: npm install -g automatiqal-cli

Usage

• All available commands can be displayed at any time by running automatiqal with the --help flag or running it without any flag:

  $ automatiqal --help $ automatiqal

  --file       -f     Location of the file, containing the run book data
  --variables, -v     Location of the variable file (if needed)
  --json              Indicates that the run book file is in JSON format
  --output,    -o     Saves the result in the provided path
  --connect,   -c     Test the connectivity. No tasks are ran
  --sample,    -s     Generate sample run book and variables files in the current folder
  --help,      -h     Shows this message
  

• --file - pass the location of the yaml file $ automatiqal --file path/to/deployment/file.yaml

• --variables - pass the location of the variables file $ automatiqal --file path/to/deployment/file.yaml --variables path/to/deployment/variables-file.yaml

• --json - if the source file is in json format then --json flag should be added as an extra parameter $ automatiqal --file path/to/deployment/file.json --json

• --output - to save the result of the run book in a json file $ automatiqal --file path/to/deployment/file.yaml --output path/to/result.json

• --sample - generate sample yaml file in the current folder $ automatiqal --sample

Runbook examples

Have a look at the examples folder for list of example yaml files (btw these runbooks are used to test Automatiqal CLI itself)

---

Example of importing an app and publishing it to a new stream:

name: Sample run book
edition: windows
environment:
  host: ${host}
  port: 443
  proxy: jwt
  authentication:
    token: ${jwt_token}
tasks:
  - name: Import application
    description: Import brand new qvf
    operation: app.upload
    details:
      file: ${qvfLocation}
      name: Something New
  - name: Create stream
    operation: stream.create
    details:
      name: New Stream
  - name: Publish app
    operation: app.publish
    source: Import application
    details:
      stream: New Stream
      name: Something New (Published)

Creating new external task, trigger scheduling and tagging the task:

name: External Task operations
edition: windows
environment:
  host: ${host}
  port: 4242
  authentication:
    cert: ${certificate}
    key: ${certificate_key}
    user_dir: ${user_dir}
    user_name: ${user_name}
tasks:
  - name: Create tags
    operation: tag.createMany
    details:
      names: ["Tag Value 1", "Tag Value 2"]
  - name: Create external task
    operation: externalTask.create
    details:
      name: New external task
      path: c:\ProgramFiles\Qlik\Sense\ServiceDispatcher\Node\node.exe
      parameters: d:\temp\index.js
  - name: Add trigger schema to task
    operation: externalTask.addTriggerSchema
    filter: name eq 'New external task'
    details:
      name: Simple daily trigger
      repeat: Daily
      repeatEvery: 1
      startDate: 2022-02-07T14:30:00.000
  - name: Add multiple trigger schema to task
    operation: externalTask.addTriggerMany
    filter: name eq 'New external task'
    details:
      - name: Weekly trigger
        repeat: Weekly
        startDate: 2022-02-07T12:30:00.000
        daysOfWeek:
          - Monday
          - Wednesday
          - Sunday
      - name: Monthly trigger
        repeat: Monthly
        enabled: false
        startDate: 2022-02-07T13:30:00.000
        daysOfMonth: [1, 5, 10]
  - name: Update external task
    operation: externalTask.update
    filter: name eq 'New external task'
    details:
      tags:
        - Tag Value 1
        - Tag Value 2

Documentation

Have a look at Automatiqal package wiki pages or the official user documentation on how to structure the file and list of operations. More information will be added soon here as well

Schema

Great little addition is the availability of YAML schema. The schema greatly helps when writing runbooks. Please refer to the schema's repo on how to use it (in VSCode)

Warning The schema is also under development and some bugs/issues are possible/expected

Limitations

• no way to test the runbook (aka dry run) automatiqal #6

Demonstration

Short video how writing the runbook feels (with the json schema)

Related projects

• Automatiqal - Automatiqal CLI is the CLI wrapper of Automatiqal. The heavy lifting is done by the Automatiqal package. Automatiqal package can be used programmatically to process runbooks
• Qlik Repo API - Automatiqal itself is using Qlik Repo API under the hood to communicate with Qlik Repo Service. Qlik Repo API abstracts the raw HTTP requests and exposes multiple methods that can be used without knowing the Qlik Repository Rest API specifics
• Qlik SaaS API - Qlik SaaS API abstracts the raw HTTP requests and exposes multiple methods that can be used without knowing the Qlik SaaS Rest API specifics (to be used in Automatiqal to write SaaS runbooks)
• Qlik Rest API - this package is the at the bottom of all. It allows communicating with various Qlik Rest API endpoints (Repository, Proxy, Engine etc)

automatiqal-cli's Issues

Multiple runbooks in a single file?

Any reasons why there should not be possible to have multiple run-books in a single file (aka array of run-books)?

Don't load commented lines

At the moment the initial checks will be performed on the whole file content. This includes the (potential) commented blocks/lines.

Ideally these lines should not be loaded at all.

Sensitive output

When output is generated sensitive data should not displayed or at least it should be masked. Talking about passwords (data connections).

But first some analysis should be made to identify the possible operations that are affected + how to solve this - is it by masking the data or just "ignoring" it and not sowing these properties?

onError tasks indication

In the output indicate (somehow) that the tasks was part of onError block.

Good addition to #4

Process multiple runbooks

Allow running multiple runbooks:

• list of files to process
• process all files in folder
  • allow excluding/ignoring files

For SaaS - make sure that sensitive data is replaced with placeholder

Similarly to QSEoW logic. Make sure that sensitive data (such as data connection passwords) are not part of the output

Do not output binday data

for example app.upload. This might be automatiqal originated issue though

Unused variables

Display warning message for unused variables.

Unused variables are defined in the variable file but not referenced in the runbook.

No warning for global variables

Environment and command line variables

Another couple of ways to provide variables. This issue will be about the global progress on both:

• environment variables - check if env variables are set and to be used #112
• command line variables - a way to provide variables and their values via additional command #113

Priority. If all variables methods are present - global, file, environment and command line then the priority is as follows (high to low):

• command line
• file
• environment
• global

No output for binary

When --output argument is provided then tasks, which result in export (app.export for example), should not populate the binary content into the data property of the output json

Multiple result checks

At the moment only one result (check) can be performed for each test. It should be possible to define multiple check for each test. For example: result to be greater than 100 AND not less than 50

Initial checks - variables without variables file

Parse the runbook and check if any variables are specified (strings between ${ and }). If variables are found but no variables file is provided - throw and error and exit before running anything

Allow duplicate task names in onError blocks

onError blocks are seen as clean up areas in case of an error. Because of this they can share common activities. Is it viable to allow duplicating of task names if they are found in onError blocks?

Case sensitive arguments

Make sure that the arguments are not case sensitive

_{Huly®: INFOR-1148}

Output - indicate if task was part of onError

In the output show if task was part of onError workflow (Informatiqal/automatiqal#178)

How to be shown? Add "onError" next to the task name? Additional column?

Output review

Review the output (if --output argument is provided) code. To be sure that things are as expected

Read runbook from http(s)

It will be quite nice if the runbook can be read from internet, Good use case is to keep the runbooks on GitHub/Lab (etc). and instead of downloading them on the local machine and run them the cli to directly get the content from there

Raw output command

Additional raw command argument that is similar to output command but instead of saving the result into external file it will directly print the result into the console.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency @types/node to v22.4.0
• Update dependency rollup to v4.20.0
• Update dependency typescript to v5.5.4
• Update dependency vitest to v1.6.0
• Update dependency vitest to v2
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

npm

package.json

• @informatiqal/variables-loader ^0.0.6
• @p-mcgowan/minimist ^2.1.0
• @rollup/plugin-commonjs ^26.0.1
• @rollup/plugin-json ^6.1.0
• @rollup/plugin-node-resolve ^15.2.3
• @rollup/plugin-replace ^5.0.7
• @rollup/plugin-typescript ^11.1.6
• @types/js-yaml ^4.0.9
• @types/node 22.0.2
• @types/node-fetch ^2.6.11
• @types/semver ^7.5.8
• automatiqal ^0.4.3
• console-table-printer ^2.12.1
• dotenv 16.4.5
• esm ^3.2.25
• js-yaml ^4.1.0
• node-fetch ^3.3.2
• nyc 17.0.0
• rollup 4.19.1
• rollup-plugin-delete 2.0.0
• rollup-plugin-polyfill-node ^0.13.0
• semver ^7.6.3
• ts-node 10.9.2
• tslib ^2.6.3
• typedoc 0.26.5
• typescript ^5.0.4
• vitest ^1.5.2
• node >=14.19.1

---

• Check this box to trigger a request for Renovate to run again on this repository

Variables loader

Incorporate @informatiqal/variables-loader package to load variables values

Global variables file

Similar to Informatiqal/test-o-matiq-cli#32

Ability to provide --global/-g flag. If this flag is present then .automatiqal file will be loaded. The file should be located inside the current user home folder.

If global and "local" variables flags are provided and both files are available AND there is an overlap in the variables then the "local" ones have priority over the global

Source/variables for details

In some cases some of the details properties are dynamic (aka depends on result of previous task). For example:

• create brand new virtual proxy
• link the new virtual proxy to an existing proxy service

In reality the link process is actually update of the proxy service. The data send during the update process (related to VP) is to add the id(s) of the virtual proxy, that is going to be linked, to virtualProxies property. The virtualProxies property accepts an string array of VP id(s). And in our case the id of the new VP is unknown until the VP is actually created.

The below example shows where the new VP id should be added

  - name: Create VP
    operation: virtualProxy.create
    details:
      prefix: test
      description: Testing
      sessionCookieHeaderName: X-Qlik-Session-Test
  - name: Add virtual proxy to proxy service
    operation: proxy.update
    filter: serverNodeConfiguration.proxyEnabled eq True and serverNodeConfiguration.name ne 'Central'
    details:
      virtualProxies:
        - ???

For this case there should be a way to pass the result of Create VP to the details.virtualProxies. There is already ${...} syntax that is used for the variables replacement. Something similar should be use in this instance as well:

• ^{Create VP}
• ^^{Create VP}
• $${Create VP}

Similarly to source property only the id properties will be extracted from the "source" task

Sample command to ask about QSEoW or SaaS

Since Automatiqal now support SaaS runbooks the sample command should ask for SaaS or QSEoW examples to be generated

Handle `export` methods

The export methods are (usually. to double check) resulting in file being saved. At the moment this is not the case.

This issue is to remind that this need to be implemented.

Also to perform initial check

• location property exists in the task
• if the location property exists. check if the location actually exists/is accessible

Array of files

Some operations details have more than one file property (contentLibrary.impotyFileMany for example). The runbook process should handle this scenario. At the moment only one file (as property) is handled

Update examples

Some of the examples are outdated. Mostly regarding task triggers. They need to match the latest schema changes

Schema validation

Once the runbook (either file or url) is loaded and right before is passed for execution the content should be validated against the json schema. Its better to validate it first before execute it

Connection testing

Argument that will test only the connectivity of the run book, without running any of the tasks

List variables - values and location(s)

At the moment --listvars command is just showing the variable names and the number of occurrences. This command can be enhanced in a couple of ways:

• location(s) - show where (in which file) the variable is used. aka runbook or/and external runbooks
• value - addional argument? additional command? to view what will be the content of the variable when the specified runbook is ran (something like an early check)

For SaaS - make sure to replace the binary contents in the output

The binary contents (when uploading and downloading) are replaced in the output with placeholder message. This issue is to remind myself that the same should be done/valid and for SaaS runbooks

Output errored tasks

When task finish with error state - write the error in the output ... somehow.
Another addition to #4

Environment variables

Add flag to allow load variables values from environment variables

Command line variables

Add flag to allow load variables values from command line

Import/include tasks from files

It should be very useful if its possible to import tasks, into the current runbook, from another files. More details to follow

_{Huly®: INFOR-1136}

Runbook summary output

Introduce new argument --summary/-s.

When provided the runbook summary will be saved at that location.

Generate stats file

At the moment the general stats are only displayed in the console.
New command should be added to allow storing the output to a file

Review samples

Review samples that a generated with --sample/-s command. They seems not correct

Force global variable?

#106 will introduce global variables. If both -g and -v arguments are provided (and both files exists) then the file variables will overwrite the global ones (if there is an overlap).

Is there a need for a special syntax that tells the process to "ignore" that rule and always use the global variables (per variable)?

At the moment the variables are defined like this:

host: ${host}

If host is specified in the global and local variables file then it will be replaced with the local one. What if we had something like this:

host: $!{host}

Having this syntax will ignore the local variable and will use the global.

Food for thoughts.

Special variables

The special about these variables is that their values will not be defined bu the user and their values will be generated by Automatiqal CLI itself. Such variables are:

• ${GUID} - generate random uuid. The uuid will exclude the - symbol. 89621de8f6944ce6befadf2862f220fd
• ${TODAY} - generate date for today. No ability to specify date format ... yet. The default will be YYYYMMDD
• ${NOW} - generate timestamp. No ability to specify format. Default format will be YYYYMMDDHHmmss

Export methods with path property

Content library export methods (dont think there are others) have path property.

When exporting these files we have to create the respective folder structure in the export location