cbPool storage exhausted about go-yara

[hillu]

@ozanh You're right; the static scan thread limit went away in YARA 4.0 and YR_MAX_THRERAD serves no other purpose than API compatibility.

This means that for practical the number of concurrent YARA threads may be way higher than before – so perhaps it makes sense to be able to grow the callback data structure on demand.

[lpendergrass]

I might not entirely understand here... but would hardcoding this at 256 cause any potential constraints elsewhere if scanning with a large number of rules within a goroutine?

https://github.com/hillu/go-yara/blob/master/util.go#L9

[ozanh]

Hi @LancePendergrass
As I use the go-yara heavily, I had realized cbPool limit and took some actions to prevent crashes. Under normal circumstances, one cannot easily exceed the cbPool limit but I limit the number of yara scanner goroutines to min(64, 2 * number of CPU). We use go-yara in thousands of machines around the world and I haven't been informed about any bug related to this.

[lpendergrass]

Okay limiting to runtime.NumCPU() and leveraging a similar pattern to below seems to have worked. Thanks.

https://stackoverflow.com/a/25306241

[hillu]

@lpendergrass By default, YARA concurrency is limited to 32 threads. If you limit your program to that value, you still should not run out of cbpool values.

If anyone has a good suggestion how to document the limit (and what those panics actually mean: PRs are welcome, of course.

[ozanh]

@hillu After my quick text search in yara source code, I couldn't find a limit for yaralib concurrency. YR_MAX_THREADS or ERROR_TOO_MANY_SCAN_THREADS aren't applicable for the library, but are used in yara cli and sandbox folders. Let me know if I missed something and let me know if it can be added to go-yara doc as a concurrency best practice.

[lpendergrass]

Dynamic growth of that cap sounds like the way to go but in lieu of that it might help to document it somewhere or indicate this in the exception message.

https://github.com/hillu/go-yara/blob/master/cbpool.go#L56

[hillu]

Sure. If somebody wants to take a shot at it, I'm open to PRs.

[hillu]

As of 3099294, dbPool has been replöaced with cgo.Handle (or an equivalent implementation that is used on older Go versions).

Closing this issue. Feel free to reopen if the issue comes up again.