Coder Social home page Coder Social logo

gcp-scala-datastore's Introduction

gcp-scala-datastore is a simple wrapper library for Google Cloud Datastore.

The library provides asynchronous API and supports the following types of fields in model classes:

  • Byte
  • Int
  • Long
  • Float
  • Double
  • String
  • Boolean
  • java.util.Date
  • com.google.cloud.datastore.DateTime
  • com.google.cloud.datastore.LatLng
  • com.google.cloud.datastore.Blob
  • java.time.LocalDateTime*
  • java.time.ZonedDateTime*
  • java.time.OffsetDateTime*
  • scala.Option[T] where T is one of types listed above
  • any other type that inherits scala.Serializable. In this case, such fields are converted into Blob in Google Cloud Console.

* java.time.* classes are converted into String at the moment. Thus, they do not work with comparison operators, i.e. |<|, |>|, |==| etc.

Usage of the gcp-scala-datastore

To be stored in Cloud Datastore a model class must inherit io.applicative.datastore.BaseEntity and must have id field of type Long or String.

import io.applicative.datastore.{BaseEntity, DatastoreService}
import io.applicative.datastore.util.reflection._
import io.applicative.datastore.query._

import scala.concurrent.Future

// Sample model class
case class Item(id: Long, name: String, price: Double, size: Int, brand: Option[String]) extends BaseEntity

val item = Item(1, "foo", 20.0, 2, None)

// Save
DatastoreService.add[Item](item)

// Save with autogenerated id
for {
  key <- DatastoreService.newKey[Item]()
  user <- DatastoreService.add[Item](Item(key.id.get, "foo", 20.0, 2, None))
} yield user

// Update
DatastoreService.update[Item](item.copy(brand = Some("bar")))

// Delete
DatastoreService.delete[Item](List(1L))

// Get one by id
DatastoreService.get[Item](1)

// or
select[Item] asSingle

// Select
val items: Future[List[Item]] = select[Item] where "size" |>| 23 and "price" |<=| 200.2 ascOrderBy "size" descOrderBy "price" asList
val items2: Future[List[Item]] = select[Item]
                                  .where("size" |>| 23)
                                  .and("price" |<| 200.2)
                                  .ascOrderBy("size")
                                  .descOrderBy("price")
                                  .asList
val singleItem: Future[Option[Item]] = select[Item] where "name" |==| "foo" asSingle

Indexes

By default, Cloud Datastore automatically predefines an index for each property of each entity kind(see https://cloud.google.com/datastore/docs/concepts/indexes for more details).
If you want to exclude any of your properties from the indexes, just add the annotation @excludeFromIndexes.

import io.applicative.datastore.BaseEntity
import io.applicative.datastore.util.reflection.excludeFromIndexes

case class Item(id: Long, name: String, price: Double, size: Int, brand: Option[String], @excludeFromIndexes description: String) extends BaseEntity

Custom Datastore Kind

By default, each model class has kind that consists of class' package and class' name. Let's consider a simple case class Foo

package com.foo.bar

import io.applicative.datastore.BaseEntity

case class Foo(id: Long) extends BaseEntity

In this case, the kind for class Foo will be com.foo.bar.Foo.

It is allowed to set up a custom kind by annotating a class with io.applicative.datastore.reflection.Kind annotation:

package com.foo.bar

import io.applicative.datastore.BaseEntity
import io.applicative.datastore.reflection.Kind

@Kind(value = "JustFoo")
case class Foo(id: Long) extends BaseEntity

In this case, the kind for the class Foo will be JustFoo

Installation using sbt

In order to install this package you will need set an extra resolver in build.sbt:

resolvers ++= Seq(
  "applctv-bintray" at "https://dl.bintray.com/applctv/gcp-scala-datastore/"
)

And then you can add it as a normal sbt dependency:

libraryDependencies ++= Seq(
  "io.applicative" %% "datastore-scala-wrapper" % "1.0-rc10"
)

gcp-scala-datastore's People

Contributors

a-panchenko avatar cortop avatar danosipov avatar jdugan1024 avatar slve avatar ticofab avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

gcp-scala-datastore's Issues

Support ListValue

The current implementation stores a list (e.g., Seq) as a BlobValue. Interestingly, it can save and read out an entity if only using this library. But if read out using Google's Java SDK directly, getList() will throw an exception:

com.google.cloud.datastore.Blob cannot be cast to java.util.List
java.lang.ClassCastException: com.google.cloud.datastore.Blob cannot be cast to java.util.List
	at com.google.cloud.datastore.BaseEntity.getList(BaseEntity.java:615)

And regardless of library, it won't index in the datastore in order to query for its values.

Make Datastore overridable

Currently, DatastoreService is a singleton object that creates a private reference to Google's Datastore object. That means it can't be changed. But for testability, it would be great to replace it with a test version using LocalDatastoreHelper:
https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-clients/google-cloud-datastore/src/main/java/com/google/cloud/datastore/testing/LocalDatastoreHelper.java

So probably, it would be a good idea to either make DatastoreService.cloudDataStore public, or better yet, make DatastoreService a class that can be instantiated with a caller-provided Datastore.

Use default field value when reading entity

Suppose you define an entity as follows:

case class Foo(bar: String = "default-bar") extends BaseEntity

Because bar is not an Option, it has to be defined, i.e., not null. But since the spirit of NoSql databases is to be schema-less, perhaps in the past an instance of Foo was created without the bar property set. In such as case, when fetching this entity, gcp-scala-datastore will throw an exception:

No such property bar
com.google.cloud.datastore.DatastoreException: No such property status
	at com.google.cloud.datastore.DatastoreException.throwInvalidRequest(DatastoreException.java:78)
	at com.google.cloud.datastore.BaseEntity.getValue(BaseEntity.java:505)
	at com.google.cloud.datastore.BaseEntity.getString(BaseEntity.java:527)
	at io.applicative.datastore.util.reflection.ReflectionHelper.getValue(ReflectionHelper.scala:200)
	at io.applicative.datastore.util.reflection.ReflectionHelper.$anonfun$setActualFieldValues$2(ReflectionHelper.scala:187)
	at io.applicative.datastore.util.reflection.ReflectionHelper.$anonfun$setActualFieldValues$2$adapted(ReflectionHelper.scala:167)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at io.applicative.datastore.util.reflection.ReflectionHelper.setActualFieldValues(ReflectionHelper.scala:167)
	at io.applicative.datastore.util.reflection.ReflectionHelper.datastoreEntityToInstance(ReflectionHelper.scala:160)
	at io.applicative.datastore.util.reflection.ReflectionHelper.datastoreEntityToInstance$(ReflectionHelper.scala:158)
	at io.applicative.datastore.DatastoreService$.datastoreEntityToInstance(DatastoreService.scala:14)
	at io.applicative.datastore.DatastoreService$.iter$1(DatastoreService.scala:139)
	at io.applicative.datastore.DatastoreService$.$anonfun$runQueryForList$1(DatastoreService.scala:144)

It would great if this library could catch such exceptions, and fill in the property value with the case class default, i.e., "default-bar".

Working with new transaction

When trying to work with new transaction. It returns an Future[Transaction], then it forces me to work with gcd package directly. Including creating a new keys and the BaseEntity is almost rendered useless in such scenario.

I am not sure if I am describing the situation correctly or if someone can please correct me when I am doing something wrong.

Custom field name

The @Kind annotation is useful to specify a different Kind name than case class name. It would also be useful to rewrite field names by annotating the case class property name, e.g.,

@Kind(value = "LegacyFooKind")
case class Foo(@Field("value = "LegacyBarProperty") bar: String)

Is there a way to query for keys only?

I need to be able to a query for matching keys only. That is, I don't want to fetch entities matching a query, just the the keys of those entities. That is a much faster operation, as the DataStore just read the keys off the index, and doesn't need to do a subsequent fetch of each entity. Is that possible with the current implementation of gcp-scala-datastore?

Determination of primary constructor is incorrect

In ReflectionHelper, there's a couple places where it tries to figure out the primary constructor (i.e., default case class constructor), such as in instanceToDatastoreEntity. It assumes that the primary constructor is the first in the list returned by class.getConstructors. However, that is not a correct assumption. The order is non-deterministic and can change each time the application is run:

https://stackoverflow.com/questions/34289801/how-is-the-order-of-the-array-what-class-getconstructors-returns-in-java

An example case class might look like this:

case class Foo(id: Long, bar: String) extends BaseEntity {
   def this(baz: Baz) = this (baz.id, baz.bar)
}

The result is that if it chooses a non-primary constructor then the annotates dictionary may not be populated, and it will throw an exception:

key not found: 0
java.util.NoSuchElementException: key not found: 0
	at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:101)
	at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:99)
	at io.applicative.datastore.util.reflection.ReflectionHelper.$anonfun$instanceToDatastoreEntity$3(ReflectionHelper.scala:63)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at scala.collection.TraversableLike.map(TraversableLike.scala:237)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
	at io.applicative.datastore.util.reflection.ReflectionHelper.instanceToDatastoreEntity(ReflectionHelper.scala:61)
	at io.applicative.datastore.util.reflection.ReflectionHelper.instanceToDatastoreEntity$(ReflectionHelper.scala:44)
	at io.applicative.datastore.DatastoreService$.instanceToDatastoreEntity(DatastoreService.scala:14)
	at io.applicative.datastore.DatastoreService$.$anonfun$add$1(DatastoreService.scala:48)

Here is a possible solution:
https://stackoverflow.com/questions/16496411/using-the-new-reflection-api-how-to-find-the-primary-constructor-of-a-class

Support for collections of values in any given field

Google Datastore allows for storing a collection of values against a single property in an entity. This is supported in the Google client library, and also in the Spotify async client library, but I don't believe it's supported by this project, from what I can see.

It would be neat to be able to define a traversable type property in my entity case class, and have ReflectionHelper.instanceToDatastoreEntity correctly convert the value.

e.g.

case class Photo(id: Long, name: String, tags: List[String], ...)

Example in the Spotify library: https://github.com/spotify/async-datastore-client/blob/master/src/main/java/com/spotify/asyncdatastoreclient/Value.java#L124-L136

If there's already a way of doing this using your library, please let me know.

Kind regards,
Steven

Support 'IN' operator

It would be great to support IN operator, where the right hand side is a list of values. Since the Datastore client doesn't natively support that, this library would need to split into multiple queries, and then merge the results. In fact, the original App Engine Datastore client does just that:
https://cloud.google.com/appengine/docs/standard/java/datastore/queries#filters

As an aside, the Python ndb library also supports IN operator:
https://googleapis.dev/python/python-ndb/latest/query.html#
In code it calls it a DisjunctionNode:
https://github.com/googleapis/python-ndb/blob/master/google/cloud/ndb/query.py#L637-L650

This should also work on a list value field: #35

Example in README throws java.lang.InternalError: Malformed class name

Hello. This package looks useful and I appreciate the effort that has gone into it so far. Thanks!

I'm trying the example in the README in an sbt console with 1.0-rc8, however it is throwing java.lang.InternalError: Malformed class name. I've included a transcript below. Any idea what's happening?

I am able to use the Java example from the same console and it works fine. I also included the crude translation into Scala of that example below.

Thanks for any insight you can shed on this issue.

> console
[info] Starting scala interpreter...
[info]
Welcome to Scala 2.11.11 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77).
Type in expressions for evaluation. Or try :help.

scala> import io.applicative.datastore.{BaseEntity, DatastoreService}
import io.applicative.datastore.{BaseEntity, DatastoreService}

scala> import io.applicative.datastore.query._
import io.applicative.datastore.query._

scala>

scala> import scala.concurrent.Future
import scala.concurrent.Future

scala> import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.ExecutionContext.Implicits.global

scala>

scala> // Sample model class

scala> case class Item(id: Long, name: String, price: Double, size: Int, brand: Option[String]) extends BaseEntity
defined class Item

scala>

scala> val item = Item(1, "foo", 20.0, 2, None)
item: Item = Item(1,foo,20.0,2,None)

scala>

scala> // Save

scala> DatastoreService.add[Item](item)
res0: scala.concurrent.Future[Item] = Future(<not completed>)
java.lang.InternalError: Malformed class name
	at java.lang.Class.getSimpleName(Class.java:1330)
	at java.lang.Class.getCanonicalName(Class.java:1399)
	at io.applicative.datastore.DatastoreService$.io$applicative$datastore$DatastoreService$$getKind(DatastoreService.scala:198)
	at io.applicative.datastore.DatastoreService$$anonfun$add$1.apply(DatastoreService.scala:46)
	at io.applicative.datastore.DatastoreService$$anonfun$add$1.apply(DatastoreService.scala:44)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
	at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

scala>

Java example:

// Imports the Google Cloud client library
import com.google.cloud.datastore.Datastore;
import com.google.cloud.datastore.DatastoreOptions;
import com.google.cloud.datastore.Entity;
import com.google.cloud.datastore.Key;

val datastore = DatastoreOptions.getDefaultInstance().getService();

val kind = "Task";
val name = "sampletask1";
val taskKey = datastore.newKeyFactory().setKind(kind).newKey(name);

val task = Entity.newBuilder(taskKey).set("description", "Buy milk").set("priority", 1).build()

datastore.put(task);

This client is not truly asynchronous

As I understand your implementation is just wraps the sync calls with Future. Client will block nevertheless while calling the api. Readme is misleading in that sense.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.