influx error: response error: error parsing query: found DURATION, expected identifier, string, number, bool about vmctl

[morozovsk]

I upgraded my vm.
New error after 0.62%:

2020/01/23 10:32:01 influx error: request failed for "api_metrics"."duration": response error: error parsing query: found KEY, expected identifier, string, number, bool at line 1, char 88

I'd also appreciate any database examples I can experiment with ter an InfluxQL query

data for api_metrics:

select * from api_metrics limit 10
select * from api_metrics order by time desc limit 10
name: api_metrics
time category host key value

---

1520522701858510817 favicon_404 localhost 373
1520522701861698995 favicon_400 localhost 63
1520522701863788884 favicon_200 localhost 31
1520522701866800074 queue localhost waiting 60
1520522761957422589 favicon_404 localhost 547
1520522761960368024 favicon_400 localhost 60
1520522761963671923 favicon_200 localhost 39
1520522761966864947 queue localhost waiting 74
1520522821917470663 favicon_404 localhost 393
1520522821921713622 favicon_400 localhost 55

1579783981881089933 news localhost per_country 247
1579783981762802891 news localhost countries_provider1 93
1579783981528546704 news localhost per_country_provider2 128
1579783981504546531 news localhost countries_provider1 4
1579783981474137155 favicon localhost waiting 0
1579783981459636429 queue localhost waiting 96

SHOW FIELD KEYS
name: api_actions
fieldKey fieldType

---

duration float
value float

name: api_metrics
fieldKey fieldType

---

value float

So duration doesn't exist in api_metrics but in api_actions.
So "request failed for api_metrics.duration" doesn't have sense.

[hagen1778]

Closing this due to inactivity. Please feel free to reopen if it doesn't work for you.

[hagen1778]

Hi @morozovsk!
Pls see commit 78d4b93 which quotes field and measurement names to avoid conflicts. It is already in master

[morozovsk]

Now I have this one:

2020/01/21 22:19:35 influx error: response doesn't contain field "time"

[hagen1778]

Thanks for looking! I guess, that happens because of empty result returned from influx. Or response does not contain the time field. I've added some context to error logging in following commit - 8fc88ea. Could you pls test it and see which exact time series failing?

[morozovsk]

1102 / 15153602 [>] 0.01% 514 p/s2020/01/22 12:47:04 unexpected response code from "http://telegraf:telegraf@localhost:8428": 400
2020/01/22 12:47:04 unexpected response code from "http://telegraf:telegraf@localhost:8428": 400
1102 / 15153602 [>] 0.01% 514 p/s2020/01/22 12:47:04 unexpected response code from "http://telegraf:telegraf@localhost:8428": 400
2020/01/22 12:47:04 unexpected response code from "http://telegraf:telegraf@localhost:8428": 400
2020/01/22 12:47:04 unexpected response code from "http://telegraf:telegraf@localhost:8428": 400
2020/01/22 12:47:04 unexpected response code from "http://telegraf:telegraf@localhost:8428": 400
2020/01/22 12:47:04 unexpected response code from "http://telegraf:telegraf@localhost:8428": 400

2020/01/22 13:32:43 unexpected error when performing request to "http://telegraf:telegraf@localhost:8428": Post http://telegraf:***@localhost:8428/api/v1/import: dial tcp: lookup localhost: device or resource busy
2020/01/22 13:32:43 unexpected response code from "http://telegraf:telegraf@localhost:8428": 401
2020/01/22 13:32:43 unexpected error when performing request to "http://telegraf:telegraf@localhost:8428": Post http://telegraf:***@localhost:8428/api/v1/import: dial tcp: lookup localhost: device or resource busy

[hagen1778]

This looks not related to previous errors. The status code 400 says it is a bad request, but the reason is unclear. I've added more context to errors in following commit 36f23c7. Could you please test it? I'd also appreciate any database examples I can experiment with. Before this I tested only the one generated by https://github.com/timescale/tsbs.

[morozovsk]

2020/01/23 10:00:12 Import process failed for:
api_actions_value{action="a",host="api"} for timestamps range 1541690058176 - 1541690058176
api_actions_value{action="a",host="api"} for timestamps range 1541690058176 - 1541690058176
api_actions_value{action="a",host="api"} for timestamps range 1541690058176 - 1541690058176
api_actions_duration{action="abtest",host="localhost"} for timestamps range 1534929781727 - 1543171682147
with error: import request error for "http://localhost:8428": bad request: unexpected response code 400: unsupported path requested: "/api/v1/import"

But 8428 - is port of vm and /api/v1/import was added only in v1.30.6 but I have:

vm version: 1.28.0

So I need to upgrade my vm.

[hagen1778]

Thanks, I'll try to reproduce this and get back to you.

[hagen1778]

Ok, the reason was that tag key conflicts with Influx service names. I quoted tags keys and values in commit 5792e88. It also improves logging for failed chunked responses

[morozovsk]

I started new process of vm on different port:

/usr/bin/victoria-metrics-prod -storageDataPath=/tmp/victoria-metrics-data -retentionPeriod 1200 -loggerLevel ERROR -httpListenAddr :18428

And I also started vmctl:

./bin/vmctl influx --influx-database telegraf --influx-user telegraf --influx-password telegraf --vm-addr http://localhost:18428
InfluxDB import mode
2020/01/24 16:42:16 Exploring scheme for database "telegraf"
2020/01/24 16:42:16 fetching fields: command: "show field keys"; database: "telegraf"; retention: "autogen"
2020/01/24 16:42:16 found 836 fields
2020/01/24 16:42:16 fetching series: command: "show series "; database: "telegraf"; retention: "autogen"
Found 11421432 timeseries to import. Continue? [Y/n] y
2020/01/24 19:14:32 10.17% 519 p/s

10% done but size of /tmp/victoria-metrics-data/ doesn't change:

du -hs /tmp/victoria-metrics-data/
140K /tmp/victoria-metrics-data/

Is it normal situation?

/usr/bin/influxd 140% CPU 1.1 GB RAM
./bin/vmctl 28% CPU 1.6GB RAM

[hagen1778]

I guess no. Have you looked on VM metrics? You may want to use dashboards to track ingestion and other metrics https://grafana.com/orgs/victoriametrics.
So far I can assume that there are a lot of non-existing combinations for found timeseries and fields. Application fetches fields via show field keys and series via show series. Then it fetches every unique combination of fields and series. It allows to save some memory, but looks like it also results in empty query fetches from Influx.

[morozovsk]

11421432 / 11421432 [--------------------------------------------------------] 100.00% 432 p/s
2020/01/25 00:02:49 Import finished!
2020/01/25 00:02:49 VictoriaMetrics importer stats:
time spent while waiting: 0s;
time spent while importing: 6.324273ms;
total datapoints: 0;
datapoints/s: 0.00;
total bytes: 0 B;
bytes/s: 0 B;
import requests: 2;
import requests retries: 0;
2020/01/25 00:02:49 Total time: 7h20m32.469757416s

[hagen1778]

Is it possible to get a sample of your database to perform the tests? I'd really appreciate if you can send me some to [email protected]

[hagen1778]

I guess I know the reason why importing failed for you:

> select "duration" from "api_metrics" where "key"="1"
>
> select "duration" from "api_metrics" where "key"='1'
name: api_metrics
time                duration
----                --------
1579820438057765000 1.2

It turns out that values in Influx should be always quoted with single '. Otherwise, the result is always blank. You may try the latest commit 6a0ffd3 to see if it works for you. I'd recommend to apply query filters and import only small amount of timeseries in order to verify that import works. Thank you for your help!

[morozovsk]

New error:

50970 / 65929 [-------------------------------->_____________________] 77.31% 1 p/s
2020/01/27 08:59:53 influx error: request failed for "postgresql"."blk_read_time": response error for "select "blk_read_time" from "postgresql" where "db"='api' and "host"='localhost' and "server"='host\'": error parsing query: found host', expected identifier, string, number, bool at line 1, char 90

An example of problem records:

postgresql,db=api,host=localhost,server=host=localhost\ user=api\ blk_read_time=0 1577829659000000000

[morozovsk]

Is it possible to get a sample of your database to perform the tests? I'd really appreciate if you can send me some to [email protected]

Done.

[hagen1778]

Hello @morozovsk! Sorry for delayed response - got a busy week. I managed to import the example you provided. Please see latest commits for change details. Could you pls try to import again?