High memory usage about blazer

[kurin]

And it's killed with an out-of-memory error? Are these a few huge files, or many small files?

You can confirm that you're using the file buffer by examining the directory that corresponds to os.TempDir() (or whatever you've set explicitly on the writer). For example, in Linux: inotifywait -m -e create -e delete /tmp should show you many files of the form /tmp/blazer1234345, which are created and destroyed.

I would like to get a full stack dump if you can capture one. Given that you're getting OOM errors (a) more quickly with smaller chunksizes, and (b) with the file buffer, I suspect that there is a goroutine leak somewhere, but I can't tell if it's in restic or blazer.

[austinginder]

It's mostly small WordPress website files with an occasional large file here and there scattered in the upload directories. Correct it's getting killed due to out of memory Killed47] 8.25% 2.964 MiB/s 51.722 GiB / 627.106 GiB 448984 / 8804986 items 0 errors ETA 55:12:41 with the following viewable using dmesg.

[1699096.104747] Out of memory: Kill process 26902 (restic) score 638 or sacrifice child
[1699096.105290] Killed process 26902 (restic) total-vm:2118824kB, anon-rss:860564kB, file-rss:0kB

I see lots of timeouts in the logs and also noticing the transfer speed is significantly slower then S3. With S3 I can get 20MiB/s however B2 is getting around 2MiB/s.

Ok I upgraded to latest commit of Restic and enabled file buffering. Everything else is default code. Looks like it might be file buffering as I see the following with inotifywait. Still it's not continuously making new tmp files like I would expect and the memory usage is much higher thing if I backup to S3.

/tmp/ CREATE blazer502882299
/tmp/ DELETE blazer502882299
/tmp/ CREATE blazer945928478
/tmp/ DELETE blazer945928478
/tmp/ CREATE restic-temp-pack-696469989
/tmp/ DELETE restic-temp-pack-696469989
/tmp/ CREATE blazer279276032
/tmp/ DELETE blazer279276032
/tmp/ CREATE blazer295254367
/tmp/ DELETE blazer295254367

As far as the full stack dump would using strace work? If so I'll email you the results.

[kurin]

Looking quickly at these debug logs, your process was killed around 8:04:30, but the most recent HTTP interaction was over two minutes earlier. I'll take a closer look later, but I suspect the memory blowup is in restic and not blazer.

However, the upload speed is probably B2 and blazer. Backblaze recommends that users establish multiple connections, because apparently they don't have load balancers or something. blazer can do this automatically for large files that get split into many chunks, which can be uploaded sequentially, but if you're uploading smaller files serially, you're only going to be talking to one endpoint at a time.

[kurin]

Do you still see memory pressure if you don't build a debug binary?

[austinginder]

Yes I do. How far it gets is somewhat random but it does consistently fail. Just tried without a debug build and ended with Killed03] 5.59% 3.717 MiB/s 35.073 GiB / 627.105 GiB 382535 / 8804947 items 0 errors ETA 45:18:30

[austinginder]

Yes so think the root issue is Restic. Just confirmed I'm having similar memory issues backing up locally. Looks like it's already an issue. Until that's resolved attempting to improve Blazer probably isn't worth it.