Victoria Metrics command line tool
The tool has been migrated to main VictoriaMetrics repository. It is released with other VictoriaMetrics components starting from v1.53.0. See VictoriaMetrics releases page.
If I hit RETURN at continue prompt, vmctl exits. Manually entering "Y" works.
$ vmctl prometheus --prom-snapshot snapshot/
Prometheus import mode
Prometheus snapshot stats:
blocks found: 1;
blocks skipped by time filter: 0;
min time: 1599746400000 (2020-09-10T23:00:00+09:00);
max time: 1599753600000 (2020-09-11T01:00:00+09:00);
samples: 242811645;
series: 2088649.
Found 1 blocks to import. Continue? [Y/n]
2020/09/11 15:22:50 Total time: 1.464266137s
The documentation for InfluxDB mode should outline the way how data from Influx maps to VM data model: ts name, labels, values, timestamps.
Using vmctl prometheus, Inf values in the source data are written to VM as 0.
I'm not sure where it happens. I did confirm that the values are still Inf at the time of pp.im.Input() call in prometheus.go do().
./vmctl influx --influx-database telegraf --influx-user telegraf --influx-password telegraf
InfluxDB import mode
2020/01/21 13:37:06 Exploring scheme for database "telegraf"
2020/01/21 13:37:06 fetching fields: command: "show field keys"; database: "telegraf"; retention: "autogen"
2020/01/21 13:37:06 found 1102 fields
2020/01/21 13:37:06 fetching series: command: "show series "; database: "telegraf"; retention: "autogen"
Found 15148092 timeseries to import. Continue? [Y/n] y
2020/01/21 13:37:36 influx error: response error: error parsing query: found DURATION, expected identifier, string, number, bool at line 1, char 8
InfluxDB version: 1.7.6
vm version: 1.28.0
There should be a way how to remove or add extra labels to timeseries before import.
As of Victoria metrics v1.36.0 the long awaited replication feature is here :)
So like the planned feature "data rebalancing" planned for vmctl, I think a new one should be added: replication enforcing.
It should check a cluster for the given replication treshold and (re)create data on another nodes if needed (should it delete data too if too many of them are found ?). This would allow creating missing data for points ingested before the replication feature but also usage of vmctl as a repair tool: changing the replication level on an existing cluster for example.
Trying to export data from Influx to VM.
But some queries in Influx return with errors due to invalid characters.
Can you add an option to ignore Influx errors? Or can we somehow get around this ?
2020/05/10 16:58:45 Exploring scheme for database "prometheus_scripts"
2020/05/10 16:58:45 fetching fields: command: "show field keys"; database: "prometheus_scripts"; retention: "autogen"
2020/05/10 16:58:45 found 3048 fields
2020/05/10 16:58:45 fetching series: command: "show series"; database: "prometheus_scripts"; retention: "autogen"
2020/05/10 16:58:46 found 19388 series
Found 19388 timeseries to import. Continue? [Y/n] y
7490 / 19388 [--------------------------------------------------------------------------------------------->___________________________________________________________________________________________________________________________________________________] 38.63% 61 p/s2020/05/10 17:05:36 influx error: request failed for "node_systemd_unit_state"."value": response error for select "value" from "node_systemd_unit_state" where "__name__"='node_systemd_unit_state' and "instance"='some_instance' and "job"='some_job' and "name"='dev-mapper-centos\x2dswap.swap' and "state"='activating': error parsing query: found \x, expected identifier, string, number, bool at line 1, char 172
In Influx itself, it looks like this:
select "value" from "node_systemd_unit_state" where "__name __" = 'node_systemd_unit_state' and "instance" = 'some_instance' and "job" = 'some_job' and "name" = 'dev-mapper-centos \ x2dswap.swap' and " state "= 'activating'
ERR: error parsing query: found \ x, expected identifier, string, number, bool at line 1, char 172
I configured in YML file like below:
management:
metrics:
export:
influx:
db: pizza
step: '1m '
uri: http://xxx:8086
========================================
I'm running influxdb in GCP server and ip address and port is open to public and I access in grafana but it's thowring below error please help
[SpringContextShutdownHook] ERROR i.m.influx.InfluxMeterRegistry - failed to send metrics to influx
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
I just set up a VictoriaMetrics simple cluster: 2 insert nodes behind a load balancer, 2 select nodes behind a load balancer, and 2 storage nodes. I am able to remote write to it just fine with our cur Prometheus server. When trying to do vmcctl from the Prometheus server with a snapshot to the vminsert load balancer endpoint I get a:
failed to create VM importer: ping to "https://remore-fqdn:8480" failed: Get "https://remote-fqdn:8480/health": unexpected EOF
When I do a curl to https://remote-fqdn:8480/health I get the expected OK. Here is what my insert command looks like:
./vmctl prometheus --prom-snapshot /opt/prometheus-data/snapshots/20200512T211728Z-7b43c0d7bc688062/ --vm-addr https://remote-fqdn:8480 --vm-account-id 0
Am I missing something? Does this need to go to the vmselect endpoint (seems weird to me)? Any help would be appreciated.
Sometimes it is necessary to migrate data between VictoriaMetrics instances. For example, when migrating data from single-node version to cluster version of VictoriaMetrics. Or when migrating data from a single cluster to another cluster.
So let's add support for data migration between VictoriaMetrics instances into vmctl tool.
High level overview of the task:
vmctl should accept label filters and time range for metrics to migrate. For example -label-filters={foo="bar",baz=~"aaa.*"} -start=2020-07-01T00:00:00Z -end=2020-08-01T00:00:00Z.vmctl must read the requested data from the configured source via /api/v1/export and ingest the data to the configured destination via /api/v1/import.The /api/v1/export may return big amounts of data, so it would be great if vmctl could stream data between source and destination without the need to read all the data from the source before writing it to the destination.
Sometimes it is needed to move time series from some data source to VictoriaMetrics while adding extra labels to all the imported time series.
VictoriaMetrics and vmagent support extra_label query args during data ingestion over HTTP-based data ingestion protocols. See VictoriaMetrics/VictoriaMetrics#1007 for details. It would be great if vmctl would support -extra_label command-line flag and pass it to extra_label query arg during data import for all the supported data sources.
It is possible to add extra labels via relabeling. See this article for details. But this is more complex and more error-prone approach comparing to the solution described above.
vmctl doesn't have login/password for vm http auth.
For example:
--vm-user value --vm-password value
failed to create VM importer: ping to "http://localhost:8428" failed: bad status code: 401
But there is an undocumented feature : --vm-addr http://telegraf:telegraf@localhost:8428
Hello!
Thanks for VictoriaMetrics!
druchoo create issue for vmctl rpm packages.
patsevanton/victoriametrics-rpm#2
Please create/add vmctl binary.
Add support for migrating historical data from Graphite to VictoriaMetrics.
Trying to transfer a large data from influx to VM.
SHOW SERIES CARDINALITY query returns 21696367. The process lasts a very long time and ends at "fetching series". After 2-3 hours get OOM.
Use
./vmctl influx --influx-addr "http://my_influx:8086" --influx-database db_name --vm-addr "http://vm_insert:8680" --vm-account-id 1 --vm-concurrency 6 --influx-concurrency 6 --influx-chunk-size 20000
Prometheus data written via remote_write has a specific schema documented here:
https://docs.influxdata.com/influxdb/v1.7/supported_protocols/prometheus/#how-prometheus-metrics-are-parsed-in-influxdb
Therefore it would be great to have a migration mode that enables to move data from InfluxDB into VictoriaMetrics which is then converted back into the original Prometheus format, e.g. no db label and no _value suffix on the timeseries.
At the end the data should look like it was directly written from Prometheus into VictoriaMetrics, just like the step over InfluxDB has never happened.
Our setup is quite simple : exporters -> prometheus -> VictoriaMetrics <- promxy <- grafana <- users
Here is a scenario that i wish to be able to do. Let two victoria metrics instances A and B.
stop instance A : ok
wait for some time : ok
start instance A : ok
stop instance B : not ok
the gap is there. So i wish i could backfill/override instance A with data from instance B during the gap. so even if there's a new rolling restart we shall not see the gap.
This would help achieve a very simple and naive HA
I want to do offline analysis of a Kubernetes cluster so I generated a Prometheus snapshot based on this article, compressed it and moved to my local machine. Initially, I used a local Prometheus instance and it worked great, I can locally access all the cluster time series and perform the queries. The day after when I woke up, I uncompressed the snapshot again and launched a new local Prometheus instance but I can no more see the time series, everything returns empty results. Based on the Prometheus server log, it looks like it runs a garbage collector on old data so I tricked the system time and the GC is no more triggered but the same problem, no results.
I decided to use VictoriaMetrics, so I uncompressed the Prometheus snapshot again and loaded it into VictoMetrics using vmctl as explained in the docs. It was loaded successfully and it shows the correct min and max UNIX timestamps. I can get all the labels correctly but querying the time series returns empty results again. I have no idea what's the reason behind this behavior. I have many compressed tarballs of Prometheus snapshots that worked fine the night before with the local Prometheus instance but now they don't work neither with Prometheus nor VictoriaMetrics.
Here is the vmctl output:
$ export SNAP=20200424T010256Z-26811953682f91a2/
$ ./vmctl prometheus --prom-snapshot=$SNAP --vm-concurrency 1 --vm-batch-size=200000 --prom-concurrency 3
Prometheus import mode
Prometheus snapshot stats:
blocks found: 1;
blocks skipped: 0;
min time: 1587688778083 (2020-04-24T02:39:38+02:00);
max time: 1587690175941 (2020-04-24T03:02:55+02:00);
samples: 8391388;
series: 37061.
Filter is not taken into account for series and samples numbers.
Found 1 blocks to import. Continue? [Y/n] Y
1 / 1 [------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 0 p/s
2020/04/24 20:52:47 Import finished!
2020/04/24 20:52:47 VictoriaMetrics importer stats:
time spent while waiting: 4.852377614s;
time spent while importing: 3.291038472s;
total datapoints: 8391388;
datapoints/s: 2549769.04;
total bytes: 159.6 MB;
bytes/s: 48.5 MB;
import requests: 84;
import requests retries: 0;
2020/04/24 20:52:47 Total time: 8.155349577s
$ curl -g "http://localhost:8428/api/v1/query?query=apiserver_request_count" && echo
{"status":"success","data":{"resultType":"vector","result":[]}}
I would like to import data into a VictoriaMetrics instance, which is secured by by basic auth.
I'm missing a flag to set the username and password.
The HTTP Url encoding of basic authentication doesn't work on any Victoriametrics URL, so the import in currently not possible.
Additionally the Victoriametrics error message is not displayed, only the stats are empty.
e.g.
# vmctl influx --influx-addr https://localhost:8086 --influx-user .... --influx-password ... --influx-database migtest --vm-addr http://user:password@localhost:8432
InfluxDB import mode
2020/07/24 10:32:37 Exploring scheme for database "migtest"
2020/07/24 10:32:37 fetching fields: command: "show field keys"; database: "migtest"; retention: "autogen"
2020/07/24 10:32:37 found 3 fields
2020/07/24 10:32:37 fetching series: command: "show series"; database: "migtest"; retention: "autogen"
2020/07/24 10:32:37 found 2 series
Found 3 timeseries to import. Continue? [Y/n] Y
3 / 3 [-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s
2020/07/24 10:32:39 Import finished!
2020/07/24 10:32:39 VictoriaMetrics importer stats:
time spent while waiting: 0s;
time spent while importing: 0s;
total samples: 0;
samples/s: NaN;
total bytes: 0 B;
bytes/s: -9223372036854775808 B;
import requests: 0;
import requests retries: 0;
2020/07/24 10:32:39 Total time: 2.050544054s
Hi all!
I'm trying to import my historic Prometheus database (InfluxDB 1.8.0) to VictoriaMetrics:
vmctl version:
root@XXXXXXXXXX:/opt# ./vmctl-linux-amd64 --version
vmctl version 0.3.0, rev. 4217ae5, built at 2020-10-14_10:54:18
Importing:
./vmctl-linux-amd64 influx --influx-addr http://1.2.3.4:8086 --influx-user admin --influx-password XxXxX --influx-database prometheus
InfluxDB import mode
2020/11/16 20:28:47 Exploring scheme for database "prometheus"
2020/11/16 20:28:47 fetching fields: command: "show field keys"; database: "prometheus"; retention: "autogen"
2020/11/16 20:28:48 found 1000 fields
2020/11/16 20:28:48 fetching series: command: "show series"; database: "prometheus"; retention: "autogen"
2020/11/16 20:29:07 found 2106087 series
2020/11/16 20:29:08 explore query failed: can't find field keys for measurement "jvm_gc_collection_seconds_count"
Info about jvm_gc_collection_seconds_count
> use prometheus;
Using database prometheus
> select * from jvm_gc_collection_seconds_count limit 10;
name: jvm_gc_collection_seconds_count
time __name__ gc instance job monitor value
---- -------- -- -------- --- ------- -----
1577750404513000000 jvm_gc_collection_seconds_count ConcurrentMarkSweep XXXXXX-3 cassandra example 211188
1577750404513000000 jvm_gc_collection_seconds_count ParNew XXXXXX-3 cassandra example 16341106
1577750410520000000 jvm_gc_collection_seconds_count ConcurrentMarkSweep XXXXXX-2 cassandra example 159725
1577750410520000000 jvm_gc_collection_seconds_count ParNew XXXXXX-2 cassandra example 15038508
1577750416120000000 jvm_gc_collection_seconds_count ConcurrentMarkSweep XXXXXX-7 cassandra example 167201
1577750416120000000 jvm_gc_collection_seconds_count ParNew XXXXXX-7 cassandra example 17843084
1577750416810000000 jvm_gc_collection_seconds_count ConcurrentMarkSweep XXXXXX-10 cassandra example 184970
1577750416810000000 jvm_gc_collection_seconds_count ParNew XXXXXX-10 cassandra example 18190132
1577750430441000000 jvm_gc_collection_seconds_count ConcurrentMarkSweep XXXXXX-8 cassandra example 182340
1577750430441000000 jvm_gc_collection_seconds_count ParNew XXXXXX-8 cassandra example 17290695
show field keys from jvm_gc_collection_seconds_count;
name: jvm_gc_collection_seconds_count
fieldKey fieldType
-------- ---------
value float
Am I missing something?
Thanks!
In case of using multitenancy with accountid and projectid in the cluster version, we are not able to use vmctl to import historical data from an external prometheus to a tenant (accountid:projectid) on our vmdb cluster.
But, according to the vmctl doc, and the tests we have done, it seems that vmctl does not support accountid:projectid values for the flag -vm-account-id and the flag -vm-project-id does not exist
A workaound proposed in the general slack was to specify import url with flag: --vm-addr=vmhostname:port_number/insert/0:1/prometheus and omit flag --vm-account-id
Unfortunately it is not helpful. As soon as we specify vm-addr, the vmctl try to send an health status to the url provided, so vmhostname:port_number/insert/0:1/prometheus/health generate an error 400.
Is-it possible to add an option in vmctl as --vm-project-id ? Or a successful workaround :)
I tried to migrate from influxdb v1.8 to victoriametrics using docker vmctl v1.81.2 on an ARM MacBook M1.
Connection is fine and the process starts but then aborts with following error:
docker run victoriametrics/vmctl:v1.81.2 influx --influx-addr http://192.168.178.41:8086 --influx-user grafana --influx-database openhab_db --influx-chunk-size=500 --vm-addr http://192.168.178.41:8428 --vm-batch-size 2000
InfluxDB import mode
2022/09/21 10:35:10 Exploring scheme for database "openhab_db"
2022/09/21 10:35:10 fetching fields: command: "show field keys"; database: "openhab_db"; retention: "autogen"
2022/09/21 10:35:11 found 721 fields; skipped 116 non-numeric fields
2022/09/21 10:35:11 fetching series: command: "show series"; database: "openhab_db"; retention: "autogen"
2022/09/21 10:35:11 found 831 series
Found 721 timeseries to import. Continue? [Y/n] panic: EOF
goroutine 1 [running]:
main.prompt({0x400027ebd0, 0x29})
github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/utils.go:19 +0x1bc
main.(*influxProcessor).run(0x40002aed20, 0x0, 0x6b?)
github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/influx.go:47 +0xf4
main.main.func2(0x4000248240?)
github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/main.go:110 +0x494
github.com/urfave/cli/v2.(*Command).Run(0x4000248240, 0x40000b8240)
github.com/urfave/cli/[email protected]/command.go:173 +0x5a8
github.com/urfave/cli/v2.(*App).RunContext(0x40001a7340, {0xa8ca90?, 0x4000028030}, {0x40000204e0, 0xd, 0xd})
github.com/urfave/cli/[email protected]/app.go:384 +0xba0
github.com/urfave/cli/v2.(*App).Run(...)
github.com/urfave/cli/[email protected]/app.go:253
main.main()
github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/main.go:227 +0xb6c
I also tried with different batch and chunk size limits and older versions of vmctl - but nothing worked - always ending with this error.
Does anybody know how to fix the problem?
Hi all!!!
I'm getting problems importing metrics from influxdb:
root@GRFMONLBBE002:/opt# ./vmctl-linux-amd64 influx --influx-addr http://x.x.x.x:8086 --vm-addr http://y.y.y.y:8428 --influx-user admin --influx-password Pmqfqha --influx-database prometheus --influx-filter-series "where job='dell_s4810' and instance='z.z.z.z' "
InfluxDB import mode
2020/12/14 11:26:08 Exploring scheme for database "prometheus"
2020/12/14 11:26:08 fetching fields: command: "show field keys"; database: "prometheus"; retention: "autogen"
2020/12/14 11:26:09 found 3447 fields
2020/12/14 11:26:09 fetching series: command: "show series where job='dell_s4810' and instance='x.x.x.x' "; database: "prometheus"; retention: "autogen"
2020/12/14 11:26:09 found 5713 series
Found 5713 timeseries to import. Continue? [Y/n] y
2020/12/14 11:26:11 influx error: request failed for "ifAlias"."value": response error for select "value" from "ifAlias" where "name"='ifAlias' and "ifAlias"='' and "ifDescr"='Loopback 2' and "instance"='z.z.z.z' and "job"='dell_s4810': error parsing query: found BADSTRING, expected identifier, string, number, bool at line 1, char 71
Influx query:
select "value" from "ifAlias" where "name"='ifAlias' and "ifAlias"='' and "ifDescr"='Loopback 2' and "instance"='z.z.z.z' and "job"='dell_s4810'
if you need something elese just let me know
Kr.
InfluxDB 2 has been out for a while and I think vmctl does not properly support that.
The result I'm seeing is:
InfluxDB import mode
2021/01/01 10:02:49 failed to create influx client: ping failed: <!doctype html><html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>InfluxDB 2.0</title><base href="/"><link rel="shortcut icon" href="/favicon.ico"></head><body><div id="react-root" data-basepath=""></div><script src="/5e93c5f5aa.js"></script></body></html>
Very briefly looking at the codebase, I suppose vmctl would need to use the new version of the official influxdb client or bring in the changes some other way.
Alternatively, it would be good to mention that >2.0 is not supported.
If anyone else faces the same issue maybe this Python script is good enough alternative
https://github.com/jonppe/influx_to_victoriametrics
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.