Invalid database timestamp. about vnstat

[vergoh]

I've now managed to reproduce the situation and can confirm that there's a bug in the --importdb functionality of version 1.16 when handling the btime value after reading the exported file. The bug results in the database being written with a btime value of zero which will then cause the database to fail validation when accessed later.

The good news is that a workaround can be used to fix the problem after import. If the used import command is for example

vnstat -i eth0 --importdb eth0_export.txt --force

then follow that command with

vnstat -i eth0 -u --force

That will cause the database validation to be ignored for one update which will also update the btime value to a non-zero value and essentially fix the invalidity in the database.

[vergoh]

Was there some reason why 1.16 wasn't able to directly use the database files created by 1.12?
Does the exported database contain lines starting with created, updated and btime? That error message suggests one of those is missing or gets translated to a value of zero.

[Misiek304]

I just updated the system on my router today and I used vnstat 1.16. I just started vnstat and the syslog was flooded with these errors:

Sat Feb  4 16:20:05 2017 daemon.err vnstatd[2200]: Error: eth0: Invalid database timestamp.
Sat Feb  4 16:20:05 2017 daemon.err vnstatd[2200]: Error: eth0: Invalid database timestamp.
Sat Feb  4 16:20:05 2017 daemon.err vnstatd[2200]: Error: Unable to use database "/mnt/sda3/vnstat/database/eth0" or backup database "/mnt/sda3/vnstat/database/.eth0".

version;3
active;1
interface;wlan0
nick;
created;0
updated;1483649105
totalrx;2219
totaltx;9277
currx;1962075809
curtx;2726490298
totalrxk;189
totaltxk;766
btime;1483639159

I guess you're right.
created is 0 since 5th January 2017

I corrected the values but vnstat still is showing the same error.

[vergoh]

1.16 introduced database validation before file write and when loading a database after I got one report of a situation where the cached database got somehow corrupted in memory which then resulted in data getting corrupted in the written database itself. I never managed to replicate that situation so I can only assume it might have been some sort of random rare hardware glitch. It's possible that you've encountered something similar if your previous backups before 5th January 2017 show created with a non-zero value. That created value is never updated after the database creation.

The solution itself is simple as you can just copypaste the created value from an older backup (the value should always have stayed the same) to the latest one and then proceed with the import.

[Misiek304]

Yeah, I did that and I'm still getting those errors.
I copied created, updated and btime values from the only database that is working to the others that don't and I I'm still getting the errors even though those three values are the same in working an non-working database.

Non-working:

version;3
active;1
interface;wlan0
nick;wlan0
created;1459292119
updated;1486226573
totalrx;0
totaltx;0
currx;10279
curtx;964007
totalrxk;0
totaltxk;9
btime;1486220883
d;0;1486226510;0;0;0;9;1
d;1;0;0;0;0;0;0
d;2;0;0;0;0;0;0
d;3;0;0;0;0;0;0
d;4;0;0;0;0;0;0
d;5;0;0;0;0;0;0
d;6;0;0;0;0;0;0
d;7;0;0;0;0;0;0
d;8;0;0;0;0;0;0
d;9;0;0;0;0;0;0
d;10;0;0;0;0;0;0
d;11;0;0;0;0;0;0
d;12;0;0;0;0;0;0
d;13;0;0;0;0;0;0
d;14;0;0;0;0;0;0
d;15;0;0;0;0;0;0
d;16;0;0;0;0;0;0
d;17;0;0;0;0;0;0
d;18;0;0;0;0;0;0
d;19;0;0;0;0;0;0
d;20;0;0;0;0;0;0
d;21;0;0;0;0;0;0
d;22;0;0;0;0;0;0
d;23;0;0;0;0;0;0
d;24;0;0;0;0;0;0
d;25;0;0;0;0;0;0
d;26;0;0;0;0;0;0
d;27;0;0;0;0;0;0
d;28;0;0;0;0;0;0
d;29;0;0;0;0;0;0
m;0;1486226510;0;0;0;9;1
m;1;0;0;0;0;0;0
m;2;0;0;0;0;0;0
m;3;0;0;0;0;0;0
m;4;0;0;0;0;0;0
m;5;0;0;0;0;0;0
m;6;0;0;0;0;0;0
m;7;0;0;0;0;0;0
m;8;0;0;0;0;0;0
m;9;0;0;0;0;0;0
m;10;0;0;0;0;0;0
m;11;0;0;0;0;0;0
t;0;0;0;0;0;0;0
t;1;0;0;0;0;0;0
t;2;0;0;0;0;0;0
t;3;0;0;0;0;0;0
t;4;0;0;0;0;0;0
t;5;0;0;0;0;0;0
t;6;0;0;0;0;0;0
t;7;0;0;0;0;0;0
t;8;0;0;0;0;0;0
t;9;0;0;0;0;0;0
h;0;0;0;0
h;1;0;0;0
h;2;0;0;0
h;3;0;0;0
h;4;0;0;0
h;5;0;0;0
h;6;0;0;0
h;7;0;0;0
h;8;0;0;0
h;9;0;0;0
h;10;0;0;0
h;11;0;0;0
h;12;0;0;0
h;13;0;0;0
h;14;0;0;0
h;15;0;0;0
h;16;0;0;0
h;17;1486226573;0;9
h;18;0;0;0
h;19;0;0;0
h;20;0;0;0
h;21;0;0;0
h;22;0;0;0
h;23;0;0;0

Working:

version;3
active;1
interface;eth0.2
nick;eth0.2
created;1459292119
updated;1486226573
totalrx;10346395
totaltx;22231027
currx;21903194
curtx;11796893
totalrxk;90
totaltxk;232
btime;1486220883
d;0;1486162801;5972;45101;38;97;1
d;1;1486076402;29167;63777;522;186;1
d;2;1485990002;2584;105;325;39;1
d;3;1485903601;19691;58955;25;507;1
d;4;1485823994;12473;62890;50;10;1
d;5;1485730801;52669;90977;440;310;1
d;6;1485644402;35836;92430;432;173;1
d;7;1485558002;33723;93567;305;786;1
d;8;1485471601;38262;90014;340;435;1
d;9;1485385204;40695;95162;619;879;1
d;10;1485298802;39736;94373;360;946;1
d;11;1485212405;90738;93687;225;96;1
d;12;1485126002;30091;96055;726;725;1
d;13;1485039602;9561;96242;71;242;1
d;14;1484953206;86880;94860;998;633;1
d;15;1484866804;77647;93643;95;804;1
d;16;1484780403;85636;94839;496;760;1
d;17;1484694003;22618;93892;926;669;1
d;18;1484607602;56216;94942;266;98;1
d;19;1484521201;85759;93039;615;493;1
d;20;1484434801;33480;92633;767;425;1
d;21;1484348402;10848;93278;282;767;1
d;22;1484262000;16200;94664;386;561;1
d;23;1484175602;22002;94465;983;541;1
d;24;1484089203;15028;94189;421;50;1
d;25;1484003964;31075;40147;380;284;1
d;26;1483916402;14083;83023;723;487;1
d;27;1483830002;12907;91542;371;571;1
d;28;1483743602;15157;93672;889;367;1
d;29;1483657202;18599;94454;313;574;1
m;0;1485903601;57414;167938;910;829;1
m;1;1483225202;1104406;2805178;918;448;1
m;2;1480546802;1001114;2818150;851;18;1
m;3;1477954802;823784;2739205;592;613;1
m;4;1475272804;1209202;2686183;389;736;1
m;5;1472680804;1456596;2670484;982;172;1
m;6;1470002403;1450351;2696132;988;948;1
m;7;1467324004;1069576;2688830;53;35;1
m;8;1464732003;827032;1408550;699;850;1
m;9;1462053600;686764;821288;223;430;1
m;10;1459461602;630927;708626;769;368;1
m;11;1459292119;29221;20457;908;929;1
t;0;1474408809;233125;99305;377;944;1
t;1;1474322404;179699;97674;259;60;1
t;2;1475186403;129167;91208;667;465;1
t;3;1470780003;128100;85910;216;706;1
t;4;1475877605;106630;94542;399;388;1
t;5;1479855604;100843;92836;107;494;1
t;6;1485212405;90738;93687;225;96;1
t;7;1481583600;89010;93933;527;490;1
t;8;1471298404;96015;85845;457;1000;1
t;9;1484953206;86880;94860;998;633;1
h;0;1486166392;296448;4062777
h;1;1486169996;136781;3935600
h;2;1486173588;115847;3769317
h;3;1486177186;1077282;3764502
h;4;1486180792;1839288;3873399
h;5;1486184387;129092;3823860
h;6;1486187997;105278;3215986
h;7;1486191585;128263;3696047
h;8;1486195189;444487;3341126
h;9;1486198793;129799;2626901
h;10;1486202396;225731;2895054
h;11;1486205985;336861;3849036
h;12;1486209006;1150652;3337278
h;13;0;0;0
h;14;0;0;0
h;15;0;0;0
h;16;0;0;0
h;17;1486226573;36;27
h;18;1486144791;412669;4096621
h;19;1486148394;255781;1419315
h;20;1486151997;117918;795921
h;21;1486155587;174620;4089678
h;22;1486159191;343497;4108656
h;23;1486162795;340422;4074136

Any suggestions on how to fix this issue?

[vergoh]

What command are you using for the database import and is the daemon stopped while importing?

[Misiek304]

Actually I'm wrong. Both of those files fail to work but here's the thing. The database file from eth0.2 interface is working but the exportdb file created from that database and then imported back to vnstat is giving the timestamp issue.

I'm lost. Is it possible that there is something wrong with timestamp verification?

[Misiek304]

What command are you using for the database import and is the daemon stopped while importing?

vnstat -i eth0.2 --force --importdb "/mnt/sda3/vnstat/exportdb/exportdb_eth0.2_(2017-02-04)";

and yes, the vnstat is stopped while doing this.

Update1: I think I'm just gonna give up.
For some reason the normal database file is working but the exportdb file after import is showing errors. This is weird.

Update2: OK, so now I'm almost sure there is something wrong with vnstat itself.
I just let it create new databases and then I used exportdb function and importdb to recreate databases from the backup. Guess what...

Error: eth0.2: Invalid database timestamp.
Error: Unable to open backup database "/mnt/sda3/vnstat/database/.eth0.2": No such file or directory

Those where new files created a few second ago by vnstat itself. Is there a way to disable those timestamps verification? Please keep in mind that my router doesn't have RTC clock, maybe that's the issue.

Where you maybe able to conform the issue?

[vergoh]

Now I'm a little confused. The first error you've mentioned shows the interface as being 'wlan0' but then it's 'eth0' in another error and then the exported file points to 'wlan0' and now there's also 'eth0.2'. The daemon process is also called 'vnstatd', not 'vnstat'.

What's the actual setup you are having, do all interfaces cause errors and what are the commands + their outputs you are executing?

[Misiek304]

I have 4 or 5 interfaces configured so the name isn't really relevant. It's br-lan, eth0, eth0.1, eth0.2 and wlan0. Only eth0.2 database is working correctly with 1.16, right after switching from 1.12. The rest of the databases are "Invalid database timestamp."

So when I restore the previous databases that show correctly their results when typing "vnstat" (those databases are from January 5th) and then backup them again with --exportdb and restore them with --importdb all of them stops working. The created, updated and btime values you mentioned are correct.

The same happens with new databases. For test I created 5 new databases for interfaces mentioned above. Then I --exportdb and --importdb them and vnstat failed again with Invalid database timestamp. Which is ridiculous since those databases where created just a few minutes before.

I was able to go back to 1.12 and everything works fine there. I couldn't test this with 1.15 cause the packet is not available. I don't think it matters though since according to you this "timestemp" stuff was introduced in 1.16.

Sorry if my posts have been a bit messy but like I said the name of interface does not matter. If there is something I could help you with or clarify something then please let me know.

For the actual setup please see this:
(I currently don't have the exact config from 1.16 since I reverted to 1.12 but this is probably identical, rest of the config is default anyway)

config vnstat
	list interface 'br-lan'
	list interface 'eth0'
	list interface 'eth0.1'
	list interface 'wlan0'
	list interface 'eth0.2'

# vnStat 1.12 config file
##

# default interface
Interface "eth0.2"

# location of the database directory
DatabaseDir "/mnt/sda3/vnstat/database"

# locale (LC_ALL) ("-" = use system locale)
Locale "-"

# on which day should months change
MonthRotate 1

# date output formats for -d, -m, -t and -w
# see 'man date' for control codes
DayFormat    "%x"
MonthFormat  "%b '%y"
TopFormat    "%x"

# characters used for visuals
RXCharacter       "%"
TXCharacter       ":"
RXHourCharacter   "r"
TXHourCharacter   "t"

# how units are prefixed when traffic is shown
# 0 = IEC standard prefixes (KiB/MiB/GiB/TiB)
# 1 = old style binary prefixes (KB/MB/GB/TB)
UnitMode 0

# output style
# 0 = minimal & narrow, 1 = bar column visible
# 2 = same as 1 except rate in summary and weekly
# 3 = rate column visible
OutputStyle 3

# used rate unit (0 = bytes, 1 = bits)
RateUnit 1

# maximum bandwidth (Mbit) for all interfaces, 0 = disable feature
# (unless interface specific limit is given)
MaxBandwidth 1000

# interface specific limits
#  example 8Mbit limit for 'ethnone':
MaxBWethnone 8
MaxBWeth0.2 120
MaxBWwlan0 300

# how many seconds should sampling for -tr take by default
Sampletime 5

# default query mode
# 0 = normal, 1 = days, 2 = months, 3 = top10
# 4 = exportdb, 5 = short, 6 = weeks, 7 = hours
QueryMode 0

# filesystem disk space check (1 = enabled, 0 = disabled)
CheckDiskSpace 1

# database file locking (1 = enabled, 0 = disabled)
UseFileLocking 1

# how much the boot time can variate between updates (seconds)
BootVariation 15

# log days without traffic to daily list (1 = enabled, 0 = disabled)
TrafficlessDays 1


# vnstatd
##

# switch to given user when started as root (leave empty to disable)
DaemonUser ""

# switch to given user when started as root (leave empty to disable)
DaemonGroup ""

# how often (in seconds) interface data is updated
UpdateInterval 15

# how often (in seconds) interface status changes are checked
PollInterval 5

# how often (in minutes) data is saved to file
SaveInterval 5

# how often (in minutes) data is saved when all interface are offline
OfflineSaveInterval 10

# force data save when interface status changes (1 = enabled, 0 = disabled)
SaveOnStatusChange 1

# enable / disable logging (0 = disabled, 1 = logfile, 2 = syslog)
UseLogging 2

# create dirs if needed (1 = enabled, 0 = disabled)
CreateDirs 1

# update ownership of files if needed (1 = enabled, 0 = disabled)
UpdateFileOwner 1

# file used for logging if UseLogging is set to 1
LogFile "/mnt/sda3/log/vnstat/vnstat.log"

# file used as daemon pid / lock file
PidFile "/var/run/vnstat/vnstat.pid"


# vnstati
##

# title timestamp format
HeaderFormat "%x %H:%M"

# show hours with rate (1 = enabled, 0 = disabled)
HourlyRate 0

# show rate in summary (1 = enabled, 0 = disabled)
SummaryRate 1

# layout of summary (1 = with monthly, 0 = without monthly)
SummaryLayout 1

# transparent background (1 = enabled, 0 = disabled)
TransparentBg 0

# image colors
CBackground     "FFFFFF"
CEdge           "AEAEAE"
CHeader         "606060"
CHeaderTitle    "FFFFFF"
CHeaderDate     "FFFFFF"
CText           "000000"
CLine           "B0B0B0"
CLineL          "-"
CRx             "92CF00"
CTx             "606060"
CRxD            "-"
CTxD            "-"