easy way to tail starting at current time about gtm

[rwynn]

Can you try this...

func MongoTimestampNow() bson.MongoTimestamp {
// according to http://docs.mongodb.org/manual/reference/bson-types/
// a mongo internal timestamp is a 64 bit value where:
// the first 32 bits are a time_t value (seconds since the Unix epoch)
// the second 32 bits are an incrementing ordinal for operations within a given second.
return bson.MongoTimestamp(time.Now().Unix() << 32)
}

which could then be used as an option to tail...
ops, errs := gtm.Tail(session, &gtm.Options(MongoTimestampNow(), nil))

[rgarcia]

Yeah I tried that but it doesn't produce any output. I've verified in the mongo console that db.oplog.rs.count() is increasing, but no ops are getting reported by gtm.

[rwynn]

Tried to replicate the issue but I wasn't able to... it got output that is. Here is the code I used:

package main

import "labix.org/v2/mgo"
import "labix.org/v2/mgo/bson"
import "github.com/rwynn/gtm"
import "fmt"
import "time"

func MongoTimestampNow() bson.MongoTimestamp {
return bson.MongoTimestamp(time.Now().Unix() << 32)
}

func main() {
session, err := mgo.Dial("localhost")
if err != nil {
panic(err)
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
ops, errs := gtm.Tail(session, &gtm.Options{MongoTimestampNow(), nil})
for {
select {
case err = <-errs:
fmt.Println(err)
case op := <-ops:
msg := fmt.Sprintf("Got op <%v> for object <%v>",
op.Operation, op.Id)
fmt.Println(msg)
}
}
}

My replica set is setup exactly as described at http://docs.mongodb.org/manual/tutorial/deploy-replica-set-for-testing/. I created a test collection in a test database, did a couple inserts and deletes after starting my go program above and I was able to see each op logged to the console.

What versions of Mongo and mgo are you using?

[rgarcia]

Revisited this problem and finally figured it out: if you're querying on the ts field, you should set the OPLOG_REPLAY flag, otherwise you risk scanning the entire oplog: https://groups.google.com/d/msg/mongodb-user/E7BSv624nBg/IL9AFcsQWesJ. You can do this in mgo using LogReplay.

[rwynn]

Thanks, Rafael. I will try that option out and see how it goes. I was not able to get a query to work on the ts field directly, and as you can see from the code resorted to a $where query. If you can find a way to avoid $where using mgo for this type of query please let me know.

[rgarcia]

Here's what we came up with: https://github.com/Clever/mgotail. It uses a direct query on the ts field plus the LogReplay cursor option.

[rwynn]

Fix for default query has been committed. Utilizes the last timestamp in the oplog to build initial query. Added LogReplay query optimization.