Comments (7)
Hello,
I was searching for alternatives to mattn/go-sqlite3 and came across this library. I really like it so far!
Yay, thanks!
But because I don't just want to make my Go programs a lot slower, I thought about doing some benchmarks first. (I'm new to benchmarking in Go!) You can see them here: https://git.jlel.se/jlelse/GoSqliteBench/src/commit/12db3eac1fd3c0d32e078e9a2d8feb917e83df84/SqliteBench_test.go
Unfortunately the results look like this:
$ go test -bench=. goos: linux goarch: amd64 pkg: git.jlel.se/jlelse/GoSqliteBench cpu: Intel(R) Core(TM) i5-4590S CPU @ 3.00GHz Benchmark_Zombiezen/Queries-4 149104 7141 ns/op Benchmark_Mattn/Queries-4 269184 4256 ns/op PASS ok git.jlel.se/jlelse/GoSqliteBench 3.274s
Am I doing something wrong? Why is this library so much slower?
Your benchmark seems like it's a pretty direct comparison in operations. I'd be curious to see whether it makes a difference to move the dbpool.Get
outside the loop: that would help narrow down whether it's a slowness with the pool or whether it's coming from modernc.org/sqlite
.
You may be interested in looking at the upstream performance issues:
from go-sqlite.
Thanks for the links. Makes sense, it's probably due to the C to Go translation. After all +67% isn't too bad after all. But sure, I will try it without the DB pool.
from go-sqlite.
I added some more benchmarks: https://git.jlel.se/jlelse/GoSqliteBench/src/commit/7171467de1ad2da8390436d91e4e404af7db45d4/SqliteBench_test.go
This are the results:
$ go test -bench=.
goos: linux
goarch: amd64
pkg: git.jlel.se/jlelse/GoSqliteBench
cpu: Intel(R) Core(TM) i5-4590S CPU @ 3.00GHz
Benchmark_Zombiezen/Parallel_Queries_with_Pool-4 157695 7046 ns/op
Benchmark_Zombiezen/Sequential_Queries_with_Pool-4 99048 10946 ns/op
Benchmark_Zombiezen/Sequential_Queries_without_Pool-4 191665 5821 ns/op
Benchmark_Mattn/Parallel_Queries-4 260556 4338 ns/op
Benchmark_Mattn/Sequential_Queries-4 79332 15115 ns/op
PASS
ok git.jlel.se/jlelse/GoSqliteBench 7.088s
from go-sqlite.
I'm surprised the sequential case performs worse for both packages. You might want to use the Go CPU profiler on those benchmarks to try to get a sense for what's the bottleneck.
from go-sqlite.
$ go test -bench=Zombiezen/Parallel -run=^$ -benchmem -cpuprofile profile.out
goos: linux
goarch: amd64
pkg: git.jlel.se/jlelse/GoSqliteBench
cpu: Intel(R) Core(TM) i5-4590S CPU @ 3.00GHz
Benchmark_Zombiezen/Parallel_Queries_with_Pool-4 148962 7203 ns/op 408 B/op 16 allocs/op
PASS
ok git.jlel.se/jlelse/GoSqliteBench 2.221s
$ go tool pprof profile.out
File: GoSqliteBench.test
Build ID: 6bb7c167ac23edc994844c5df646faf161429ff7
Type: cpu
Time: Sep 7, 2021 at 8:52am (CEST)
Duration: 2.21s, Total samples = 2.51s (113.79%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 1040ms, 41.43% of 2510ms total
Dropped 126 nodes (cum <= 12.55ms)
Showing top 10 nodes out of 204
flat flat% sum% cum cum%
250ms 9.96% 9.96% 340ms 13.55% sync.(*Mutex).lockSlow
220ms 8.76% 18.73% 560ms 22.31% sync.(*Mutex).Lock (inline)
180ms 7.17% 25.90% 230ms 9.16% sync.(*Mutex).Unlock
90ms 3.59% 29.48% 460ms 18.33% modernc.org/sqlite/lib.Xsqlite3BtreeMovetoUnpacked
60ms 2.39% 31.87% 60ms 2.39% runtime.futex
50ms 1.99% 33.86% 60ms 2.39% modernc.org/sqlite/lib.pcache1FetchNoMutex
50ms 1.99% 35.86% 50ms 1.99% runtime.lock2
50ms 1.99% 37.85% 50ms 1.99% runtime.unlock2
50ms 1.99% 39.84% 60ms 2.39% syscall.Syscall
40ms 1.59% 41.43% 40ms 1.59% modernc.org/libc.(*TLS).Alloc
Looks like the mutex functions introduce some overhead.
from go-sqlite.
An alternative to the mutex in the pool struct might be to use a sync.Map.
from go-sqlite.
Just a heads up sync.Map is just a wrapper for a mutex and map. Would a RWMutex help? What's the ratio of read to write ?
from go-sqlite.
Related Issues (20)
- Missing Columns information HOT 1
- The best way to effectively do exec many?
- postgres style $1/$2 are treated as named arguments HOT 3
- how to use SetCollation HOT 1
- ability to customize connection creatd by sqlitex.Pool HOT 1
- Support the VFS API HOT 1
- error creating a transaction: sqlitex.Exec: sqlite: clear bindings: interrupted
- Pool.Close blocks indefinitely when a panic happens during iteration of statement rows
- Add a Pool interface? HOT 2
- BlockOnBusy backoff period is long HOT 5
- Undefined symbols in sqlite v.1.29.0 HOT 2
- Support pointer-passing
- JSONB support
- sqlitex.Pool.Get should be able to return an error HOT 3
- Extented error codes aren't reported properly HOT 1
- Document pattern of checking for zero rows HOT 6
- file:memory does not appear to work HOT 4
- INSERT multiple VALUES? HOT 2
- Stmt.findBindName shoud be public HOT 3
- Unexpected error: "migrate database: sqlite: step: database is locked"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go-sqlite.