talariadb / talaria Goto Github PK
View Code? Open in Web Editor NEWTalariaDB is a distributed, highly available, and low latency time-series database for Presto
License: MIT License
TalariaDB is a distributed, highly available, and low latency time-series database for Presto
License: MIT License
panic: Base level can't be zero.
goroutine 935 [running]:
github.com/dgraph-io/badger/v3.(*levelsController).fillTablesL0ToLbase(0x400091e4d0, 0x4000837770, 0x4000837400)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/levels.go:1168 +0x610
github.com/dgraph-io/badger/v3.(*levelsController).fillTablesL0(0x400091e4d0, 0x4000837770, 0x1d3c40c)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/levels.go:1229 +0x30
github.com/dgraph-io/badger/v3.(*levelsController).doCompact(0x400091e4d0, 0x2, 0x0, 0x3ff3333333333333, 0x3ff028509ce1b939, 0x0, 0x0, 0x0, 0x0, 0x40004c80a8, ...)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/levels.go:1505 +0x1b0
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor.func2(0x0, 0x3ff3333333333333, 0x3ff028509ce1b939, 0x0, 0x0, 0x0, 0x0, 0x40004c8048, 0x3, 0x3, ...)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/levels.go:465 +0x58
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor.func3(0x40007e3f30)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/levels.go:488 +0xf8
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor(0x400091e4d0, 0x2, 0x40007c65a0)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/levels.go:517 +0x28c
created by github.com/dgraph-io/badger/v3.(*levelsController).startCompact
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/levels.go:354 +0x7c
Proposing to use
https://github.com/imjohnbo/ok-to-test
Any commiter with write access to repo can comment on the MR from forked repos ok-to-test
,
will make the tokens available for the integration test-workflows like docker builds.
2022/08/08 07:17:26 [error] Received err: file does not exist for table 13775352. Cleaning up...
panic: file does not exist for table 13775352
goroutine 1 [running]:
github.com/kelindar/talaria/internal/storage/disk.Open({0x40003c8bba, 0x5}, {0x40003c8cb0, 0x8}, {0x2208f80, 0x4000f12500}, {0x40003c9178, 0x0, 0x0, 0x40003c9160, ...})
/go/src/talaria/internal/storage/disk/disk.go:55 +0x11c
main.openTable({0x40003c8cb0, 0x8}, {{0x40003c9178, 0x0, 0x0, 0x40003c9160, 0x0, 0x40003c9170, 0x0, {0x0, ...}}, ...}, ...)
/go/src/talaria/main.go:124 +0x10c
main.main()
/go/src/talaria/main.go:68 +0xcd8
In this encoding part, we have too many allocations:
talaria/client/golang/encoding.go
Line 36 in 2a1cffe
We can optimise it like this(Just an example)
type encoder struct {
// ... other fields ...
batchPool *sync.Pool
}
func newEncoder() *encoder {
e := new(encoder)
e.batchPool = &sync.Pool{
New: func() interface{} {
return &pb.Batch{Events: make([]*pb.Event, 0, batchSize)}
},
}
return e
}
// Encode implements formatter interface
func (e *encoder) Encode(events []Event) *pb.Batch {
e.Lock()
defer e.Unlock()
e.next = 0
e.dictionary = make(map[string]uint32, len(events))
e.batch = e.batchPool.Get().(*pb.Batch)
e.batch.Events = e.batch.Events[:0]
// Write the events
for _, ev := range events {
encoded := e.encodeEvent(ev)
e.batch.Events = append(e.batch.Events, encoded)
}
// Write the interned strings
e.writeDictionary()
e.batchPool.Put(e.batch)
return e.batch
}
2022/05/19 06:15:55 opening badger with options {Dir:/data/logs ValueDir:/data/logs SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0x4000b0c020 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:65536 LevelSizeMultiplier:3 TableSizeMultiplier:2 MaxLevels:25 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:268435456 IndexCacheSize:0 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:5000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:1 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:true NamespaceOffset:-1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
panic: while opening memtables error: while opening fid: 2 error: while updating skiplist error: mremap size mismatch: requested: 20 got: 134217728
goroutine 1 [running]:
github.com/kelindar/talaria/internal/storage/disk.Open(0x4000fa0143, 0x5, 0x1d2b516, 0x4, 0x2006498, 0x4000b5c0c0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/talaria/internal/storage/disk/disk.go:55 +0x114
github.com/kelindar/talaria/internal/table/log.New(0x4000b0c390, 0x1fdef80, 0x400039c1b0, 0x2006498, 0x4000b5c0c0, 0x4000ad4020)
/go/src/talaria/internal/table/log/log.go:42 +0xc0
main.main()
/go/src/talaria/main.go:54 +0x5c8
The config was designed to be live-loaded. However, it is a local variable inside the main function
here
https://github.com/talariadb/talaria/pull/86/files
This MR introduced sink
configs, which means to abstract writers, however it does not compatable with old config files.
2022/05/17 07:22:49 [error] ServerError: target=storage/writer/writer.go.158, reason=Internal Server Error, msg=stream: writer was not configured
2022/05/17 07:22:49 [error] ServerError: target=table/timeseries/timeseries.go.124, reason=Internal Server Error, msg=error reading from uri due to scheme is not supported
In fact, this behaviour is intended, the table does has not scheme setting and writer configured.
right now the computed columns are defined on a global scope. If any table has the column specified in its schema definition, it is computed and furnished. We might have a slight advantage when we scope this at a table definition level and avoid un-necessary computations on a multi-table ingestion scenario.
When instance ingesting SQS msg and compact to ORC file, it crashed after 20mins.
2022/06/03 07:11:57 [error] writeRequests: Unable to truncate file: "/data/eventlog/000001.vlog" error: mremap size mismatch: requested: 1074437932 got: 2147483646
2022/06/03 07:11:57 [error] ServerError: target=storage/disk/disk.go.160, reason=Internal Server Error, msg=unable to append due to Unable to truncate file: "/data/eventlog/000001.vlog" error: mremap size mismatch: requested: 1074437932 got: 2147483646
2022/06/03 07:13:14 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:13:14 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:13:14 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:14:14 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:14:14 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:14:14 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:15:14 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:15:14 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:15:14 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:16:14 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:16:14 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:16:14 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:17:15 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:17:15 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:17:15 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:18:15 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:18:15 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:18:15 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:19:15 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:19:15 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:19:15 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:20:15 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:20:15 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:20:15 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:21:16 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:21:16 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:21:16 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:22:06 [error] writeRequests: Unable to truncate file: "/data/eventlog/000002.vlog" error: mremap size mismatch: requested: 1074096225 got: 2147483646
2022/06/03 07:22:06 [error] ServerError: target=storage/disk/disk.go.160, reason=Internal Server Error, msg=unable to append due to Unable to truncate file: "/data/eventlog/000002.vlog" error: mremap size mismatch: requested: 1074096225 got: 2147483646
panic: runtime error: slice bounds out of range [1074096225:0]
goroutine 218270 [running]:
github.com/dgraph-io/badger/v3.(*valueLog).write.func2(0x403137e330, 0x403137e330, 0x40107e2120)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:826 +0x1bc
github.com/dgraph-io/badger/v3.(*valueLog).write(0x40003b33f8, 0x4083065f40, 0x1, 0xa, 0x0, 0x0)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:884 +0x388
github.com/dgraph-io/badger/v3.(*DB).writeRequests(0x40003b3200, 0x4083065f40, 0x1, 0xa, 0x1d94600, 0x0)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:816 +0x74
github.com/dgraph-io/badger/v3.(*DB).doWrites.func1(0x4083065f40, 0x1, 0xa)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:887 +0x4c
created by github.com/dgraph-io/badger/v3.(*DB).doWrites
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:940 +0x350
The way bytes is being copied is not thread safe https://github.com/kelindar/talaria/blob/master/internal/storage/flush/flush.go#L109 since buffer is being reused.
// Bytes returns a slice of length b.Len() holding the unread portion of the buffer.
// The slice is valid for use only until the next buffer modification (that is,
// only until the next call to a method like Read, Write, Reset, or Truncate).
// The slice aliases the buffer content at least until the next buffer modification,
// so immediate changes to the slice will affect the result of future reads.
func (b *Buffer) Bytes() []byte { return b.buf[b.off:] }
In flush, if another thread runs buffer might be written, which overwrites the output that is used to generate the orc file.
output := buffer.Bytes()
buffer.Reset()
s.memoryPool.Put(buffer)
return s.generateFileName(blocks[0]), output
Hi,
I am testing and review CPE's parodus client coonnection with talaria, and packet capture showed http 400 error. Just wonerring if it is one of the problem with tr1d1um issue xmidt-org/tr1d1um#498
Here is the talaria log when CPE's parodus client "say hello" to talaria, seems fine.
{"key":"debug","ts":"2024-04-09T10:56:58+08:00","message":"accepted connection","serverName":"talaria","bindAddress":":6400","listenNetwork":"tcp","listenAddress":"[::]:6400","remoteAddress":"192.168.1.1:56888"}
But packet capture showed 400 error, does it matter?
Where 192.168.1.1 is my CPE's IP, and 192.168.1.132 is WebPA server IP (talaria+scytale+tr1d1um).
Metric server(table name).ingest.count
is only recorded the events ingested in the batch with same partition key.
talaria/internal/server/server_ingest.go
Line 81 in cbdbb69
I want to add a exact events metric instead count here:
talaria/internal/server/server_ingest.go
Line 63 in cbdbb69
Genji is a sql store based on badger dB. Been getting very high perf when using it on projects because itโs embedded and because itโs badgerdb and not SQLite.
would you consider adding it as a sink store ?
https://github.com/genjidb/genji
genji has no HA aspects but I think itโs possible in the future using raft badger / pebble implementations out there
I am considering adding benthos to allow scriptable transformations also.
IMHO, ingest.count
is not the same a ingest.row.count
Could you help with this? @jeffreylean
Currently, if the specified schema for column is JSON and input is JSON, value will be marshaled twice and will not work.
// convertToJSON converts an ORC map/list/struct to JSON
func convertToJSON(value interface{}) (string, bool) {
switch vt := value.(type) {
case []orctype.MapEntry:
remap := make(map[string]interface{}, len(vt))
for _, v := range vt {
remap[fmt.Sprintf("%v", v.Key)] = v.Value
}
value = remap
case orctype.Struct:
case []interface{}:
case interface{}:
default:
return "", false
}
b, err := json.Marshal(value)
if err != nil {
return "", false
}
return string(json.RawMessage(b)), true
}
We can add support for the case where input is already json and ignore return accordingly.
169.254.169.254
is a magic IP(Link-local address) used by most cloud services (AWS & Azure) for metadata.
This line of code introduced in the latest MR will set the URL http://169.254.169.254/metadata/identity/oauth2/token
to get the metadata and send HTTP requests when refreshing. The Azure library did this.
However, when you run the Talaria on AWS, there is no such URL/info provided by it; it will be a timeout as there is no route.
Skip the auth here when Talaria is not running on Azure; directly uses the Env to auth.
Talaria version: 1.1.17
We use two deployments of talaria. One for ingestion and the other one for the real-time serving of events. The output of the first are orc files which is the input to the second.
We are facing the issue in which the output orc file from the first system produces different results when read by the second talaria deployment and when read by spark.
For example for this 2021-01-04-10-45-00--mcd--b5de9c69-e92d-4369-9278-b594b6c73f27.orc.zip (first unzip and then test), when read by talaria for real-time serving returns output 450
for column val
. But the same file when read by spark SQL returns output 0.0
. The expected value is 0.0
. This specific row can be found with the filters event == "bs.calculationServerGBCConfigID" and "8515ec5141fd48758a96f0e6cdb282a1" == bch.
Sample code to read the ORC file in go
import (
"fmt"
"log"
"github.com/crphang/orc"
)
func main() {
r, err := orc.Open("2021-01-04-10-45-00--mcd--b5de9c69-e92d-4369-9278-b594b6c73f27.orc")
if err != nil {
fmt.Printf("%+v\n", err)
}
defer r.Close()
// Create a new Cursor reading the provided columns.
c := r.Select("req", "event", "val", "bch")
// Iterate over each stripe in the file.
for c.Stripes() {
// Iterate over each row in the stripe.
for c.Next() {
// Retrieve a slice of interface values for the current row.
event := c.Row()[1]
//req := c.Row()[0]
bch := c.Row()[3]
if event == "bs.calculationServerGBCConfigID" {
if "8515ec5141fd48758a96f0e6cdb282a1" == bch {
fmt.Printf("%+v %+v\n", c.Row())
}
}
}
}
if err := c.Err(); err != nil {
log.Fatal(err)
}
}
`
Sample code to read the file in spark
df.createOrReplaceTempView("stalkerprocessordata")
spark.sql("""
select * from stalkerprocessordata
where event = 'bs.calculationServerGBCConfigID'
and bch = '8515ec5141fd48758a96f0e6cdb282a1'
""").show(false)`
If we abort talaria and restart it, we may have issues. It happened frequently in the new version, I reported the case here: #92
And we would better wait until it consumes all data in badgerDB if possible(in doubt, need your review)
// WriteBlock writes a one or multiple blocks to the underlying writer.
func (s *Flusher) WriteBlock(blocks []block.Block, schema typeof.Schema) error {
if s.writer == nil || len(blocks) == 0 {
return nil
}
// Merge the blocks based on the specified merging function
// buffer, err := s.merge(blocks, schema)
// if err != nil {
// return err
// }
// Generate the file name and write the data to the underlying writer
return s.writer.Write(s.generateFileName(blocks[0]), blocks)
}
In Flusher, we should return an error when the writer is nil, otherwise, the badgerDB data will be deleted without writing to any writter.
It would be better to keep them in badgerDB
2022/07/13 06:44:44 received signal terminated, start graceful shutdown
2022/07/13 06:44:44 Cancel ctx done
2022/07/13 06:44:44 Close gossip done
2022/07/13 06:44:44 server: wait exit...
2022/07/13 06:44:44 GRPC GracefulStop done, it will wait all request finished
2022/07/13 06:44:44 Close table nodes done, it will compact all if compact is enable
2022/07/13 06:44:44 Close table logs done, it will compact all if compact is enable
2022/07/13 06:44:44 Compact start closing, will compact all data
unexpected fault address 0xffff1c04d30f
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xffff1c04d30f pc=0x18897b8]
goroutine 994 [running]:
runtime.throw({0x1f02823, 0x5})
/usr/local/go/src/runtime/panic.go:1198 +0x54 fp=0x40007354c0 sp=0x4000735490 pc=0xb94544
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:742 +0x1e4 fp=0x4000735500 sp=0x40007354c0 pc=0xbadc64
github.com/google/flatbuffers/go.GetInt32(...)
/go/src/talaria/vendor/github.com/google/flatbuffers/go/encode.go:89
github.com/google/flatbuffers/go.GetSOffsetT(...)
/go/src/talaria/vendor/github.com/google/flatbuffers/go/encode.go:126
github.com/google/flatbuffers/go.(*Table).GetSOffsetT(...)
/go/src/talaria/vendor/github.com/google/flatbuffers/go/table.go:139
github.com/google/flatbuffers/go.(*Table).Offset(0x40005ba020, 0x4)
/go/src/talaria/vendor/github.com/google/flatbuffers/go/table.go:15 +0x58 fp=0x4000735530 sp=0x4000735510 pc=0x18897b8
github.com/dgraph-io/badger/v3/fb.(*TableIndex).Offsets(0x40005ba020, 0x40007356d0, 0xeb1)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/fb/TableIndex.go:30 +0x34 fp=0x4000735560 sp=0x4000735530 pc=0x188c994
github.com/dgraph-io/badger/v3/table.(*Table).offsets(0x4016c7c300, 0x40007356d0, 0xeb1)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/table/table.go:531 +0x44 fp=0x4000735590 sp=0x4000735560 pc=0x189e5f4
github.com/dgraph-io/badger/v3/table.(*Table).block(0x4016c7c300, 0xeb1, 0x1)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/table/table.go:556 +0x1e0 fp=0x40007357a0 sp=0x4000735590 pc=0x189e800
github.com/dgraph-io/badger/v3/table.(*Iterator).next(0x4005a569a0)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/table/iterator.go:331 +0xe0 fp=0x40007357f0 sp=0x40007357a0 pc=0x1899c70
github.com/dgraph-io/badger/v3/table.(*Iterator).next(0x4005a569a0)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/table/iterator.go:348 +0x204 fp=0x4000735840 sp=0x40007357f0 pc=0x1899d94
github.com/dgraph-io/badger/v3/table.(*Iterator).Next(0x4005a569a0)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/table/iterator.go:406 +0x40 fp=0x4000735860 sp=0x4000735840 pc=0x189a170
github.com/dgraph-io/badger/v3/table.(*node).next(0x4002f189e0)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/table/merge_iterator.go:83 +0x50 fp=0x4000735880 sp=0x4000735860 pc=0x189b4d0
github.com/dgraph-io/badger/v3/table.(*MergeIterator).Next(0x4002f189a0)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/table/merge_iterator.go:157 +0x30 fp=0x40007358f0 sp=0x4000735880 pc=0x189b990
github.com/dgraph-io/badger/v3/table.(*node).next(0x4002f18b00)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/table/merge_iterator.go:79 +0x5c fp=0x4000735910 sp=0x40007358f0 pc=0x189b4dc
github.com/dgraph-io/badger/v3/table.(*MergeIterator).Next(0x4002f18b00)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/table/merge_iterator.go:157 +0x30 fp=0x4000735980 sp=0x4000735910 pc=0x189b990
github.com/dgraph-io/badger/v3/table.(*node).next(0x4002f18bf0)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/table/merge_iterator.go:79 +0x5c fp=0x40007359a0 sp=0x4000735980 pc=0x189b4dc
github.com/dgraph-io/badger/v3/table.(*MergeIterator).Next(0x4002f18bb0)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/table/merge_iterator.go:157 +0x30 fp=0x4000735a10 sp=0x40007359a0 pc=0x189b990
github.com/dgraph-io/badger/v3.(*Iterator).parseItem(0x4002f18c60)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/iterator.go:666 +0x500 fp=0x4000735b00 sp=0x4000735a10 pc=0x18bb290
github.com/dgraph-io/badger/v3.(*Iterator).prefetch(0x4002f18c60)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/iterator.go:739 +0xa8 fp=0x4000735b40 sp=0x4000735b00 pc=0x18bbbb8
github.com/dgraph-io/badger/v3.(*Iterator).Seek(0x4002f18c60, {0x4001ab82f0, 0x10, 0x10})
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/iterator.go:780 +0x338 fp=0x4000735ba0 sp=0x4000735b40 pc=0x18bbf38
github.com/kelindar/talaria/internal/storage/disk.(*Storage).Range.func1(0x4000636000)
/go/src/talaria/internal/storage/disk/disk.go:187 +0x1bc fp=0x4000735cc0 sp=0x4000735ba0 pc=0x18f1a9c
github.com/dgraph-io/badger/v3.(*DB).View(0x4000459b00, 0x4036d39d60)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/txn.go:806 +0xac fp=0x4000735d00 sp=0x4000735cc0 pc=0x18e2f3c
github.com/kelindar/talaria/internal/storage/disk.(*Storage).Range(0x4000e9d080, {0x4001ab82f0, 0x10, 0x10}, {0x4001ab8300, 0x10, 0x10}, 0x4003106000)
/go/src/talaria/internal/storage/disk/disk.go:179 +0xe4 fp=0x4000735db0 sp=0x4000735d00 pc=0x18f18b4
github.com/kelindar/talaria/internal/storage/compact.(*Storage).Compact(0x4002f3e8c0, {0x21eda58, 0x400004a058})
/go/src/talaria/internal/storage/compact/compact.go:99 +0x290 fp=0x4000735eb0 sp=0x4000735db0 pc=0x18f40e0
github.com/kelindar/talaria/internal/storage/compact.(*Storage).Compact-fm({0x21eda58, 0x400004a058})
/go/src/talaria/internal/storage/compact/compact.go:87 +0x40 fp=0x4000735f00 sp=0x4000735eb0 pc=0x18f50a0
github.com/kelindar/talaria/internal/storage/compact.compactEvery.func1({0x21eda58, 0x400004a058})
/go/src/talaria/internal/storage/compact/compact.go:58 +0x58 fp=0x4000735f50 sp=0x4000735f00 pc=0x18f3bb8
github.com/grab/async.(*task).run.func1(0x4002ddd450, {0x21eda58, 0x400004a058}, 0x40005322a0)
/go/src/talaria/vendor/github.com/grab/async/task.go:139 +0x3c fp=0x4000735fb0 sp=0x4000735f50 pc=0x100acac
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000735fb0 sp=0x4000735fb0 pc=0xbcc534
created by github.com/grab/async.(*task).run
/go/src/talaria/vendor/github.com/grab/async/task.go:138 +0x14c
goroutine 1 [chan receive]:
main.main()
/go/src/talaria/main.go:108 +0x1064
goroutine 6 [select]:
go.opencensus.io/stats/view.(*worker).start(0x40002ac700)
/go/src/talaria/vendor/go.opencensus.io/stats/view/worker.go:276 +0x98
created by go.opencensus.io/stats/view.init.0
/go/src/talaria/vendor/go.opencensus.io/stats/view/worker.go:34 +0x80
goroutine 118 [chan receive, 1 minutes]:
github.com/golang/glog.(*loggingT).flushDaemon(0x2da36c0)
/go/src/talaria/vendor/github.com/golang/glog/glog.go:882 +0x70
created by github.com/golang/glog.init.0
/go/src/talaria/vendor/github.com/golang/glog/glog.go:410 +0x2a8
goroutine 913 [select, 7 minutes]:
github.com/dgraph-io/badger/v3.(*vlogThreshold).listenForValueThresholdUpdate(0x40003aafc0)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/value.go:1172 +0xd4
created by github.com/dgraph-io/badger/v3.Open
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/db.go:380 +0x123c
goroutine 299 [select]:
github.com/DataDog/datadog-go/statsd.(*sender).sendLoop(0x4000492c40)
/go/src/talaria/vendor/github.com/DataDog/datadog-go/statsd/sender.go:92 +0xb0
created by github.com/DataDog/datadog-go/statsd.newSender
/go/src/talaria/vendor/github.com/DataDog/datadog-go/statsd/sender.go:46 +0xf8
goroutine 300 [select]:
github.com/DataDog/datadog-go/statsd.(*Client).watch(0x4000a02800)
/go/src/talaria/vendor/github.com/DataDog/datadog-go/statsd/statsd.go:384 +0x80
github.com/DataDog/datadog-go/statsd.newWithWriter.func1(0x4000a02800)
/go/src/talaria/vendor/github.com/DataDog/datadog-go/statsd/statsd.go:351 +0x50
created by github.com/DataDog/datadog-go/statsd.newWithWriter
/go/src/talaria/vendor/github.com/DataDog/datadog-go/statsd/statsd.go:349 +0xa88
goroutine 301 [select]:
github.com/DataDog/datadog-go/statsd.(*Client).telemetry(0x4000a02800)
/go/src/talaria/vendor/github.com/DataDog/datadog-go/statsd/statsd.go:399 +0x8c
github.com/DataDog/datadog-go/statsd.newWithWriter.func2(0x4000a02800, 0x4000642090)
/go/src/talaria/vendor/github.com/DataDog/datadog-go/statsd/statsd.go:357 +0x5c
created by github.com/DataDog/datadog-go/statsd.newWithWriter
/go/src/talaria/vendor/github.com/DataDog/datadog-go/statsd/statsd.go:354 +0xac4
goroutine 927 [select]:
github.com/grab/async.Repeat.func2({0x21eda58, 0x400004a058})
/go/src/talaria/vendor/github.com/grab/async/repeat.go:24 +0xb0
github.com/grab/async.(*task).run.func1(0x4000a8ec80, {0x21eda58, 0x400004a058}, 0x4000eabce0)
/go/src/talaria/vendor/github.com/grab/async/task.go:139 +0x3c
created by github.com/grab/async.(*task).run
/go/src/talaria/vendor/github.com/grab/async/task.go:138 +0x14c
goroutine 216 [select, 1 minutes]:
github.com/grab/async.Repeat.func2({0x21eda58, 0x400004a058})
/go/src/talaria/vendor/github.com/grab/async/repeat.go:24 +0xb0
github.com/grab/async.(*task).run.func1(0x4000a8e0f0, {0x21eda58, 0x400004a058}, 0x4000d243c0)
/go/src/talaria/vendor/github.com/grab/async/task.go:139 +0x3c
created by github.com/grab/async.(*task).run
/go/src/talaria/vendor/github.com/grab/async/task.go:138 +0x14c
goroutine 1896 [select]:
github.com/grab/async.(*task).run(0x4001aba050, {0x21eda58, 0x400004a058})
/go/src/talaria/vendor/github.com/grab/async/task.go:143 +0x1c8
created by github.com/grab/async.(*task).Run
/go/src/talaria/vendor/github.com/grab/async/task.go:100 +0x4c
goroutine 348 [chan receive, 3 minutes]:
github.com/kelindar/loader.(*watcher).Start.func1(0x4000a8e460, {0x21eda58, 0x400004a058})
/go/src/talaria/vendor/github.com/kelindar/loader/watcher.go:78 +0x70
created by github.com/kelindar/loader.(*watcher).Start
/go/src/talaria/vendor/github.com/kelindar/loader/watcher.go:77 +0x144
goroutine 351 [select]:
github.com/dgraph-io/badger/v3/y.(*WaterMark).process(0x4000e9d590, 0x4000e9d560)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/y/watermark.go:214 +0x1c4
created by github.com/dgraph-io/badger/v3/y.(*WaterMark).Init
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/y/watermark.go:72 +0x84
goroutine 352 [select, 7 minutes]:
github.com/dgraph-io/badger/v3/y.(*WaterMark).process(0x4000e9d5c0, 0x4000e9d560)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/y/watermark.go:214 +0x1c4
created by github.com/dgraph-io/badger/v3/y.(*WaterMark).Init
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/y/watermark.go:72 +0x84
goroutine 353 [select]:
github.com/dgraph-io/ristretto/z.(*AllocatorPool).freeupAllocators(0x400048ea50)
/go/src/talaria/vendor/github.com/dgraph-io/ristretto/z/allocator.go:385 +0x12c
created by github.com/dgraph-io/ristretto/z.NewAllocatorPool
/go/src/talaria/vendor/github.com/dgraph-io/ristretto/z/allocator.go:324 +0xc0
goroutine 386 [select]:
github.com/dgraph-io/ristretto.(*lfuPolicy).processItems(0x40003ab040)
/go/src/talaria/vendor/github.com/dgraph-io/ristretto/policy.go:67 +0x70
created by github.com/dgraph-io/ristretto.newPolicy
/go/src/talaria/vendor/github.com/dgraph-io/ristretto/policy.go:51 +0x158
goroutine 387 [select]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0x4000492fc0)
/go/src/talaria/vendor/github.com/dgraph-io/ristretto/cache.go:476 +0x10c
created by github.com/dgraph-io/ristretto.NewCache
/go/src/talaria/vendor/github.com/dgraph-io/ristretto/cache.go:213 +0x590
goroutine 389 [select, 1 minutes]:
github.com/dgraph-io/badger/v3.(*DB).updateSize(0x4000459b00, 0x400099c750)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/db.go:1171 +0x13c
created by github.com/dgraph-io/badger/v3.Open
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/db.go:331 +0xbf4
goroutine 1015 [chan receive, 7 minutes]:
github.com/kelindar/talaria/internal/scripting.(*Loader).watch.func1({0x21eda58, 0x400004a058})
/go/src/talaria/internal/scripting/loader.go:76 +0x50
github.com/grab/async.(*task).run.func1(0x4000c6a7d0, {0x21eda58, 0x400004a058}, 0x4000a28840)
/go/src/talaria/vendor/github.com/grab/async/task.go:139 +0x3c
created by github.com/grab/async.(*task).run
/go/src/talaria/vendor/github.com/grab/async/task.go:138 +0x14c
goroutine 1083 [select, 7 minutes]:
github.com/grab/async.(*task).run(0x4000c6a7d0, {0x21eda58, 0x400004a058})
/go/src/talaria/vendor/github.com/grab/async/task.go:143 +0x1c8
created by github.com/grab/async.(*task).Run
/go/src/talaria/vendor/github.com/grab/async/task.go:100 +0x4c
goroutine 1082 [chan receive, 2 minutes]:
github.com/kelindar/loader.(*watcher).Start.func1(0x4000c6a6e0, {0x21eda58, 0x400004a058})
/go/src/talaria/vendor/github.com/kelindar/loader/watcher.go:78 +0x70
created by github.com/kelindar/loader.(*watcher).Start
/go/src/talaria/vendor/github.com/kelindar/loader/watcher.go:77 +0x144
goroutine 1081 [select, 7 minutes]:
github.com/grab/async.(*task).run(0x4000c6a1e0, {0x21eda58, 0x400004a058})
/go/src/talaria/vendor/github.com/grab/async/task.go:143 +0x1c8
created by github.com/grab/async.(*task).Run
/go/src/talaria/vendor/github.com/grab/async/task.go:100 +0x4c
goroutine 1095 [IO wait, 7 minutes]:
internal/poll.runtime_pollWait(0xffff80953de8, 0x72)
/usr/local/go/src/runtime/netpoll.go:303 +0xb4
internal/poll.(*pollDesc).wait(0x4003099818, 0x72, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x4003099800)
/usr/local/go/src/internal/poll/fd_unix.go:402 +0x1f4
net.(*netFD).accept(0x4003099800)
/usr/local/go/src/net/fd_unix.go:173 +0x2c
net.(*TCPListener).accept(0x40030dcc48)
/usr/local/go/src/net/tcpsock_posix.go:140 +0x2c
net.(*TCPListener).Accept(0x40030dcc48)
/usr/local/go/src/net/tcpsock.go:262 +0x34
net/http.(*Server).Serve(0x4000b32000, {0x21db748, 0x40030dcc48})
/usr/local/go/src/net/http/server.go:3002 +0x364
net/http.(*Server).ListenAndServe(0x4000b32000)
/usr/local/go/src/net/http/server.go:2931 +0xb0
main.startHTTPServerAsync.func1(0x17ac)
/go/src/talaria/main.go:166 +0x240
created by main.startHTTPServerAsync
/go/src/talaria/main.go:155 +0x3c
goroutine 1005 [syscall]:
syscall.Syscall(0xd7, 0xfffefb42c000, 0x206b90, 0x0)
/usr/local/go/src/syscall/asm_linux_arm64.s:9 +0x10
github.com/dgraph-io/ristretto/z.munmap({0xfffefb42c000, 0x206b90, 0x206b90})
/go/src/talaria/vendor/github.com/dgraph-io/ristretto/z/mmap_linux.go:71 +0x78
github.com/dgraph-io/ristretto/z.Munmap(...)
/go/src/talaria/vendor/github.com/dgraph-io/ristretto/z/mmap.go:31
github.com/dgraph-io/ristretto/z.(*MmapFile).Close(0x4000245220, 0xffffffffffffffff)
/go/src/talaria/vendor/github.com/dgraph-io/ristretto/z/file.go:194 +0x12c
github.com/dgraph-io/badger/v3.(*levelHandler).close(0x40118b67e0)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/level_handler.go:233 +0x108
github.com/dgraph-io/badger/v3.(*levelsController).cleanupLevels(0x4000493030)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/levels.go:208 +0x78
github.com/dgraph-io/badger/v3.(*levelsController).close(0x4000493030)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/levels.go:1564 +0x28
github.com/dgraph-io/badger/v3.(*DB).close(0x4000459b00)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/db.go:623 +0x46c
github.com/dgraph-io/badger/v3.(*DB).Close.func1()
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/db.go:529 +0x30
sync.(*Once).doSlow(0x4000459ee0, 0x400dd5dcd0)
/usr/local/go/src/sync/once.go:68 +0x108
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:59
github.com/dgraph-io/badger/v3.(*DB).Close(0x4000459b00)
/go/src/talaria/vendor/github.com/dgraph-io/badger/v3/db.go:528 +0x68
github.com/kelindar/talaria/internal/storage/disk.(*Storage).Close(0x4000e9d080)
/go/src/talaria/internal/storage/disk/disk.go:299 +0x50
github.com/kelindar/talaria/internal/storage.Close({0x400dd5dde8, 0x2, 0x2})
/go/src/talaria/internal/storage/storage.go:49 +0xc0
github.com/kelindar/talaria/internal/storage/compact.(*Storage).Close(0x4002f3e8c0)
/go/src/talaria/internal/storage/compact/compact.go:186 +0xe4
github.com/kelindar/talaria/internal/table/timeseries.(*Table).Close(0x4000b443c0)
/go/src/talaria/internal/table/timeseries/timeseries.go:78 +0x34
github.com/kelindar/talaria/internal/server.(*Server).Close(0x4000c6a230, {0x2208f80, 0x40003aa500})
/go/src/talaria/internal/server/server.go:164 +0x15c
main.main.func1({0x21d98b0, 0x2d4a5f8})
/go/src/talaria/main.go:84 +0x16c
main.onSignal.func1(0x40040ff800, 0x4004361040)
/go/src/talaria/main.go:149 +0x44
created by main.onSignal
/go/src/talaria/main.go:147 +0xc4
goroutine 1004 [syscall]:
os/signal.signal_recv()
/usr/local/go/src/runtime/sigqueue.go:169 +0x108
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:24 +0x20
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:151 +0x38
goroutine 1003 [chan receive, 7 minutes]:
github.com/kelindar/talaria/internal/table/timeseries.(*Table).loadStaticSchema.func1({0x21eda58, 0x400004a058})
/go/src/talaria/internal/table/timeseries/timeseries.go:136 +0x50
github.com/grab/async.(*task).run.func1(0x4000c6a1e0, {0x21eda58, 0x400004a058}, 0x40005338c0)
/go/src/talaria/vendor/github.com/grab/async/task.go:139 +0x3c
created by github.com/grab/async.(*task).run
/go/src/talaria/vendor/github.com/grab/async/task.go:138 +0x14c
goroutine 1080 [chan receive, 2 minutes]:
github.com/kelindar/loader.(*watcher).Start.func1(0x4002ddda90, {0x21eda58, 0x400004a058})
/go/src/talaria/vendor/github.com/kelindar/loader/watcher.go:78 +0x70
created by github.com/kelindar/loader.(*watcher).Start
/go/src/talaria/vendor/github.com/kelindar/loader/watcher.go:77 +0x144
goroutine 1278 [select]:
github.com/grab/async.Consume.func1({0x21eda58, 0x400004a058})
/go/src/talaria/vendor/github.com/grab/async/consume.go:34 +0x1b0
github.com/grab/async.(*task).run.func1(0x4001aba050, {0x21eda58, 0x400004a058}, 0x4000c884e0)
/go/src/talaria/vendor/github.com/grab/async/task.go:139 +0x3c
created by github.com/grab/async.(*task).run
Currently, we have Computed used for compute a new column, we can leverage the computed use for event filter.
This is the demo: #93
2022/06/03 07:09:19 [error] ServerError: target=storage/disk/disk.go.160, reason=Internal Server Error, msg=unable to append due to Unable to truncate file: "/data/eventlog/000001.vlog" error: mremap size mismatch: requested: 1073892944 got: 2147483646
2022/06/03 07:09:19 [error] writeRequests: Unable to truncate file: "/data/eventlog/000001.vlog" error: mremap size mismatch: requested: 1073892944 got: 2147483646
2022/06/03 07:09:22 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 44], Version : 3061, meta: 66, userMeta: 0 Error: ErrEOF: End of file
2022/06/03 07:09:22 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 44], Version : 3061, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1205847 Offset:1013888666}
2022/06/03 07:09:22 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
Just thinking if we embed data fusion into talaria(not really sure atm on how to do it) but it eliminates, us to provide output ports for trino/spark or any other tool.
Let me know your thoughts around this.
Datafusion was written in Rust.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1a01854]
goroutine 262480 [running]:
github.com/kelindar/talaria/internal/storage/writer/base.(*Writer).Encode(0x4030b02100, {0x1be0480, 0x401068de30})
/go/src/talaria/internal/storage/writer/base/base.go:89 +0x94
github.com/kelindar/talaria/internal/storage/writer/azure.(*MultiAccountWriter).Write(0x4030b0c000, {0x4000f1b480, 0x7f, 0x80}, {0x4011360000, 0x49, 0x87})
/go/src/talaria/internal/storage/writer/azure/azure.go:220 +0x80
github.com/kelindar/talaria/internal/storage/writer/multi.(*Writer).Write.func1()
/go/src/talaria/internal/storage/writer/multi/multi.go:50 +0x60
golang.org/x/sync/errgroup.(*Group).Go.func1(0x40113a02a0, 0x40103007d0)
/go/src/talaria/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go
Currently i see you're using v0.3.0, that version has a couple of bugs i'd recommend it gets upgraded.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.