Coder Social home page Coder Social logo

datastore's Introduction

Datastore Wrapper Go Documentation CI

(AppEngine | Cloud) Datastore wrapper for Go 👉

Simple. Happy. Respect standard library.

$ go get -u go.mercari.io/datastore/v2

see v2 doc at first.

Feature

DO

  • Wrap google.golang.org/appengine/datastore and cloud.google.com/go/datastore
    • keep key behavior
    • align to cloud.google.com/go/datastore first
  • Re-implement datastore package
  • Re-implement datastore.SaveStruct & LoadStruct
    • Ignore unmapped property
    • Add PropertyTranslator interface
      • Convert types like mytime.Unix to time.Time and reverse it
      • Rename property like CreatedAt to createdAt or created_at and reverse it
  • Re-implement PropertyLoadSaver
    • Pass context.Context to Save & Load method
  • Add retry feature to each RPC
    • e.g. Retry AllocateID when it failed
  • Add middleware layer
    • About...
      • Local Cache
      • AE Memcache
      • Logging
      • Retry
      • etc...
    • Easy to ON/OFF switching
  • Add some useful methods
    • aedatastore/TransactionContext

DON'T

  • have utility functions
  • support firestore

Restriction

  • aedatastore package
    • When using slice of struct, MUST specified datastore:",flatten" option.
      • original (ae & cloud) datastore.SaveStruct have different behaviors.
      • see aeprodtest/main.go /api/test3

Committers

Contribution

Please read the CLA below carefully before submitting your contribution.

https://www.mercari.com/cla/

Setup environment & Run tests

  • requirements
    • gcloud sdk
      • gcloud components install app-engine-go
      • gcloud components install beta cloud-datastore-emulator
  1. Testing in local
$ ./setup.sh # exec once
$ ./serve.sh # exec in background
$ ./test.sh

License

Copyright 2017 Mercari, Inc.

Licensed under the MIT License.

datastore's People

Contributors

dependabot[bot] avatar izumisy avatar kailuo-wwweb3 avatar sinmetal avatar timakin avatar tyamagu2 avatar vvakame avatar zchee avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

datastore's Issues

invalid cache created on Cloud Datastore

AEDatastore+AEMemcacheからCloudDatastore+AEMemcacheに移行しようとして気がついた。
なにかをGetして、Txに突入してPutし、Txから抜けてGetすると古いEntityが取れる… という問題がある。

Gob encoding panics when trying to write to cache after reading from datastore.

I added a rediscache middleware in front of the cloud datastore and I tried to write and read a struct with an embedded struct in it. The program successfully writes the payload in both the cache and datastore. But it panics (https://github.com/mercari/datastore/blob/master/clouddatastore/key.go#L29) when trying to write the data into the redis cahce when encountering a cache miss.
After some debugging, I think the issue was that the library wraps an Entity around the embedded struct with a nil key. It worked on write because the nil key was catched at https://github.com/mercari/datastore/blob/master/clouddatastore/convert.go#L10 since the key was a {*datastore.key}{nil}, but on read, the key was a {*datastore.keyImpl}{nil} so the nil checking returns false and keep proceeding to the panic point.

implement migrator

  • common
    • *datastore.Key to datastore.Key
    • *datastore.Query to datastore.Query
    • *datastore.Iterator to datastore.Iterator
    • datastore.TransactionOptions does not supported
  • AEDatastore
    • update https://github.com/favclip/qbg . support datastore wrapper
    • replace err.(appengine.MultiError) to err.(datastore.MultiError) carefully.
      • We can't find it on compile time.
    • don't use appengine.BlobKey
    • google.golang.org/appengine/datastore.Done to google.golang.org/api/iterator.Done
    • key.IntID() to key.ID()
    • key.StringID() to key.Name()
    • Custor handing
      • zero struct or nil
  • CloudDatastore
    • *datastore.Commit to datastore.Commit
  • goon
    • *goon.Goon to *boom.Boom
    • goon.FromContext(ctx) to ds, _ := aedatastore.FromContext(ctx); boom.FromClient(ctx, ds)

Panic occurs when Parent Key is nil

type Item struct {
	Parent datastore.Key `boom:"parent"`
	ID        string    `boom:"id"`
	Contents  []string  `json:"contents"`
	CreatedAt time.Time `json:"createdAt"`
	UpdatedAt time.Time `json:"updatedAt"`
	SchemaVersion int `json:"-"`
}

func hoge() {
	bm := boom.FromClient(ctx, client)
	item := &Item{
		Parent: nil,
		ID:       form.ID,
		Contents: form.Contents,
	}
        bm.Put(item) // panic !!!
}

interface conversion: interface is nil, not datastore.Key

Why isn't this library supporting `DistinctOn` method?

Why isn't this library supporting DistinctOn method?

// Query represents a datastore query.
type Query interface {
	Ancestor(ancestor Key) Query
	EventualConsistency() Query
	Namespace(ns string) Query
	Transaction(t Transaction) Query
	Filter(filterStr string, value interface{}) Query
	Order(fieldName string) Query
	Project(fieldNames ...string) Query
	Distinct() Query
	// NOT IMPLEMENTED ON APPENGINE DistinctOn(fieldNames ...string) *Query
	KeysOnly() Query
	Limit(limit int) Query
	Offset(offset int) Query
	Start(c Cursor) Query
	End(c Cursor) Query

	Dump() *QueryDump
}

With redis and cloud.google.com/go/datastore in v1.4.0 or later, put fail.

When using redis and cloud.google.com/go/datastore which is after V1.4.0, if the time.Time type member of the struct is uninitialized, the struct cannot be put to datastore.
Without redis, put work well.
Using cloud.google.com/go/datastore V1.3.0, put work well.
(Few days ago I updated go.mod, then my unittest failed. datastrore V1.3.0 is OK, V1.4.0 or later is fail.)

Below is my Test code.

package main

import (
        orgdatastore "cloud.google.com/go/datastore"
        "context"
        "fmt"
        "github.com/gomodule/redigo/redis"
        "go.mercari.io/datastore"
        "go.mercari.io/datastore/clouddatastore"
        "go.mercari.io/datastore/dsmiddleware/rediscache"
        "log"
        "net"
        "time"
)
type Entity struct {
        Tp time.Time //About cloud.google.com/go/datastore, in v1.4.0 or later, redis will result in an error unless it is explicitly initialized. No error occurs to v1.3.0
        P  int
}

var ename string
var pname string
var eid string

func main() {
        loc, err := time.LoadLocation("Asia/Tokyo")
        if err != nil {
                loc = time.FixedZone("Asia/Tokyo", 9*60*60)
        }
        time.Local = loc
        ename = "Entity"
        pname = "my-project"
        eid = "eid"
        //pattern_originalDatastore()
        pattern_mercariDatastore(true, false)
        pattern_mercariDatastore(true, true)
}

func pattern_mercariDatastore(withRedis, doInitTimeMember bool) {
        dbkind := fmt.Sprintf("mercari_redis_%v_timeInit_%v", withRedis, doInitTimeMember)
        fail := false
        c := context.Background()
        client := getMercariClient(c, withRedis)
        defer client.Close()

        k := client.NameKey(ename, eid, nil)
        e := new(Entity)
        if err := client.Get(c, k, e); err != nil {
                //This is first get, then err shold not be nil.
                //log.Printf("%s Get Error %v\n", dbkind, err)
        } else {
                log.Printf("%s initialize miss\n", dbkind)
                fail = true
        }

        e.P = 50
        if doInitTimeMember { //cloud.google.com/go/datastore in v1.4.0 or later, this is necessary.
                e.Tp = time.Now()
        }
        if _, err := client.Put(c, k, e); err != nil {
                log.Printf("%s Put Error %v\n", dbkind, err)
                fail = true
        }
        e2 := new(Entity)
        if err := client.Get(c, k, e2); err != nil {
                log.Printf("%s Put Fail %v\n", dbkind, err)
                fail = true
        }
        if e2.P != 50 {
                log.Printf("%s Put Fail 50, now %d\n", dbkind, e2.P)
                fail = true
        }
        e2.P = 100
        client.Put(c, k, e2)

        e3 := new(Entity)
        if err := client.Get(c, k, e3); err != nil {
                log.Printf("%s Put Fail %v\n", dbkind, err)
                fail = true
        }
        if e3.P != 100 {
                log.Printf("%s Put100 Fail \n", dbkind)
                fail = true
        }
        client.Delete(c, k)
        if fail {
                log.Printf("----- %s Fail -----", dbkind)
        } else {
                log.Printf("----- %s OK -----", dbkind)
        }
}

func pattern_originalDatastore() {
        dbkind := "original"
        fail := false
        c := context.Background()
        client, _ := orgdatastore.NewClient(c, pname)
        defer client.Close()

        k := orgdatastore.NameKey(ename, eid, nil)
        e := new(Entity)
        if err := client.Get(c, k, e); err != nil {
                //This is first get, then err shold not be nil.
                //log.Printf("%s Get Error %v\n", dbkind, err)
        } else {
                log.Printf("%s initialize miss\n", dbkind)
                fail = true
        }

        e.P = 50
        if _, err := client.Put(c, k, e); err != nil {
                log.Printf("%s Put Error %v\n", dbkind, err)
                fail = true
        }
        e2 := new(Entity)
        if err := client.Get(c, k, e2); err != nil {
                log.Printf("%s Put Fail %v\n", dbkind, err)
                fail = true
        }
        if e2.P != 50 {
                log.Printf("%s Put Fail 50, now %d\n", dbkind, e2.P)
                fail = true
        }
        e2.P = 100
        client.Put(c, k, e2)

        e3 := new(Entity)
        if err := client.Get(c, k, e3); err != nil {
                log.Printf("%s Put Fail %v\n", dbkind, err)
                fail = true
        }
        if e3.P != 100 {
                log.Printf("%s Put100 Fail \n", dbkind)
                fail = true
        }
        client.Delete(c, k)

        if fail {
                log.Printf("----- %s Fail -----", dbkind)
        } else {
                log.Printf("----- %s OK -----", dbkind)
        }
}

func getMercariClient(c context.Context, withredis bool) datastore.Client {
        orgClient, _ := orgdatastore.NewClient(c, "my-project")
        client, _ := clouddatastore.FromClient(c, orgClient)
        dial, err := net.Dial("tcp", "localhost:6379")
        if err != nil {
                log.Printf("getMercariClient %v", err)
        }
        if withredis {
                conn := redis.NewConn(dial, 100*time.Millisecond, 100*time.Millisecond)
                //defer conn.Close()
                mw := rediscache.New(conn,
                        rediscache.WithLogger(func(ctx context.Context, format string, args ...interface{}) {
                                log.Printf(format, args...)
                        }))
                rediscachemw := mw
                client.AppendMiddleware(rediscachemw)
        }
        return client
}

aedatastore cannot bind embedded pointer of struct

Thank you for awesome package.

I tried to get embedded struct but failed. Show the cases below (ignore error).
The difference of the cases is Bar is struct or pointer of struct.
Is this a bug?

failure case

The embedded Bar is null.

type Foo struct {
	_kind string `boom:"kind"`
	ID string `boom:"id"`
	Bar *Bar `datastore:",flatten"`
}

type Bar struct {
	Name string
}

func get() {
	client, _ := aedatastore.FromContext(ctx)
	bm := boom.FromClient(ctx, client)
	foo := Foo{ID:"1"}
	bm.Get(&foo)
        b, _ := json.Marshal(foo)
        fmt.Print(b)
        // => {"ID":"1","Bar":null}
}

success case

type Foo struct {
	_kind string `boom:"kind"`
	ID string `boom:"id"`
	Bar Bar `datastore:",flatten"`
}

type Bar struct {
	Name string
}

func get() {
	client, _ := aedatastore.FromContext(ctx)
	bm := boom.FromClient(ctx, client)
	foo := Foo{ID:"1"}
	bm.Get(&foo)
        b, _ := json.Marshal(foo)
        fmt.Print(b)
        // => {"ID":"1","Bar":{"Name":"bar1"}}
}

Support ReadOnly transaction

boom does not provide a transaction context to use taskqueue with transaction.

Boom does not provide a transaction context to use taskqueue with transaction.
If we want to use taskqueue with transaction, we need transaction context.

Currently, mercari/datastore has aedatastore.TransactionContext(t datastore.Transaction) context.Context. but boom.Transaction does not implement datastore.Transaction and boom.Transaction does not provide getter for datastore.Transaction.

So we can't use taskqueue with transaction when we use boom.

Failed on 'go get go.mercari.io/datastore'

Hi

I'm experiencing failed on go get. Following is the command which I ran and its result.

$ go get go.mercari.io/datastore
package go.mercari.io/datastore: unrecognized import path "go.mercari.io/datastore" (https fetch: Get /datastore?go-get=1: stopped after 10 redirects)

My environments:

  • Running on: Travis CI
  • OS: Linux (Ubuntu 14.04.5 LTS)
  • go: go version go1.8 linux/amd64

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.