circe / circe-golden

Golden testing for Circe

circe

circe is a JSON library for Scala (and Scala.js).

Please see the guide for more information about why circe exists and how to use it.

Community

Adopters

Are you using circe? Please consider opening a pull request to list your organization here:

• Abacus
• Anduin Transactions
• Apollo Agriculture
• AutoScout24
• Babylon Health
• Banno inside of Jack Henry
• BBC
• Broad Institute
• Chartboost (sending hundreds of thousands of messages per second on our Ad Exchange)
• CiBO Technologies (using circe to (de)serialize data in support of a sustainable revolution in agriculture)
• ClearScore
• Codacy
• Colisweb
• Connio (creating and managing digital twins with Circe and Akka)
• Coya
• Criteo (collecting 200.000 events per second from our banners)
• Dassault Systèmes
• Datalogue
• Dreamlines
• DriveTribe
• Earnest
• Elastic
• Emmy Sharing
• FOLIO
• GLNGN Server
• Gympass
• Gutefrage
• Hello Soda
• HERE Technologies
• Hireproof
• HolidayCheck
• ImmobilienScout24
• Immozentral
• Indix
• ITV
• Kinoplan
• Latitude Financial Services
• MatchesFashion
• Merit
• Metacommerce
• Mobile GmbH
• MomentFeed
• New Work
• Nike
• Ocado Technology
• On Air Entertainment
• Onfocus
• Opt Technologies
• OVO Energy
• Panaseer
• Permutive
• Prezi
• Project September (using circe to exchange and store data within the platform and serve data using GraphQL with Sangria)
• Raster Foundry
• Ravel Law (using circe to (de)serialize data for search, analytics, and visualization of tens of millions of legal opinions)
• REA Group - realestate.com.au
• Reonomy
• Resilient plc
• Sky
• Snowplow Analytics
• SoundCloud (transforming 200,000,000 JSON events every hour in MapReduce ETLs)
• Spotify (using circe for JSON IO in Scio)
• SpotX
• Stripe
• Stylight
• TabMo (parsing more than 100k events per second with Akka Stream and Spark)
• Tinkoff
• The Guardian
• Threat Stack
• Tranzzo
• Twilio (sending many, many millions of messages a day with Circe and Akka)
• Uberall
• VEACT
• WeWork
• Whisk
• Wide Angle Analytics
• Zalando
• Zendesk

Other circe organization projects

Please get in touch on Gitter if you have a circe-related project that you'd like to discuss hosting under the circe organization on GitHub.

• circe-benchmarks: Benchmarks for comparing the performance of circe and other JSON libraries for the JVM.
• circe-config: A library for translating between HOCON, Java properties, and JSON documents.
• circe-derivation: Experimental generic derivation with improved compile times.
• circe-fs2: A library that provides streaming JSON parsing and decoding built on fs2 and Jawn.
• circe-iteratee: A library that provides streaming JSON parsing and decoding built on iteratee.io and Jawn.
• circe-jackson: A library that provides Jackson-supported parsing and printing for circe.
• circe-spray: A library that provides JSON marshallers and unmarshallers for Spray using circe.
• circe-yaml: A library that uses SnakeYAML to support parsing YAML 1.1 into circe's Json.

Related projects

The following open source projects are either built on circe or provide circe support:

• Actor Messenger: A platform for instant messaging.
• akka-http-json: A library that supports using circe for JSON marshalling and unmarshalling in Akka HTTP.
• akka-stream-json: A library that provides JSON support for stream based applications using Jawn as a parser with a convenience example for circe.
• Argus: Generates models and circe encoders and decoders from JSON schemas.
• Blackdoor JOSE: circe JSON support for blackdoor JOSE and JWT.
• borer: Allows circe encoders/decoders to be reused for CBOR (de)serialization.
• circe-debezium: Circe codecs for Debezium payload types
• circe-geojson: Circe support for GeoJSON (RFC 7946)
• circe-kafka: Implicit conversion of Encoder and Decoder into Kafka Serializer/Deserializer/Serde
• cornichon: A DSL for JSON API testing.
• Cosmos: An API for DCOS services that uses circe.
• crjdt: A conflict-free replicated JSON datatype in Scala.
• diffson: A Scala diff / patch library for JSON.
• elastic4s: A Scala client for Elasticsearch with circe support.
• Enumeratum: Enumerations for Scala with circe integration.
• Featherbed: A REST client library with circe support.
• Finch: A library for building web services with circe support.
• fintrospect: HTTP contracts for Finagle with circe support.
• fluflu: A Fluentd logger.
• Github4s: A GitHub API wrapper written in Scala.
• content-api-models: The Guardian's Content API Thrift models.
• http4s: A purely functional HTTP library for client and server applications.
• IdeaLingua: Staged Interface Definition and Data Modeling Language & RPC system currently targeting Scala, Go, C# and TypeScript. Scala codegen generates models and JSON codecs using circe.
• Iglu Schema Repository: A JSON Schema repository with circe support.
• jsactor: An actor library for Scala.js with circe support.
• jsoniter-scala-circe: A booster for faster parsing/printing to/from circe AST and decoding/encoding of java.time._ and BigInt types.
• jwt-circe: A JSON Web Token implementation with circe support.
• kadai-log: A logging library with circe support.
• msgpack4z-circe: A MessagePack implementation with circe support.
• ohNoMyCirce: Friendly compile error messages for shapeless's Generic, circe's Encoder & Decoder and slick's case class mapping.
• play-circe: circe support for Play!.
• pulsar4s: A Scala client for Apache-Pulsar with circe support.
• Rapture: Support for using circe's parsing and AST in Rapture JSON.
• roc: A PostgreSQL client built on Finagle.
• sangria-circe: circe marshalling for Sangria, a GraphQL implementation.
• scalist: A Todoist API client.
• scala-jsonapi: Scala support library for integrating the JSON API spec with Spray, Play! or Circe
• scala-json-rpc: JSON-RPC 2.0 library for Scala and Scala.js
• scalatest-json-circe: Scalatest matchers for Json with appropriate equality and descriptive error messages.
• Scio: A Scala API for Apache Beam and Google Cloud Dataflow, uses circe for JSON IO
• seals: Tools for schema evolution and language-integrated schemata (derives circe encoders and decoders).
• shaclex: RDF validation using SHACL or ShEx.
• Slick-pg: Slick extensions for PostgreSQL.
• sttp: Scala HTTP client.
• Synapses: A lightweight Neural Network library, for js, jvm and .net.
• telepooz: A Scala wrapper for the Telegram Bot API built on circe.
• Zenith: Functional HTTP library built on circe.

Examples

The following projects provide examples, templates, or benchmarks that include circe:

• https://github.com/alanphillips78/akka-http-microservice-blueprint
• https://github.com/bneil/fcs_boilerplate
• https://github.com/gvolpe/simple-http4s-api
• https://github.com/vitorsvieira/akka-http-circe-json-template
• https://github.com/stephennancekivell/some-jmh-json-benchmarks-circe-jackson
• https://github.com/pauljamescleary/scala-pet-store

Contributors and participation

circe is a fork of Argonaut, and if you find it at all useful, you should thank Mark Hibberd, Tony Morris, Kenji Yoshida, and the rest of the Argonaut contributors.

circe is currently maintained by Darren Gibson and Erlend Hamnaberg. After the 1.0 release, all pull requests will require two sign-offs by a maintainer to be merged.

The circe project is a typelevel affiliate project, and follow the Typelevel Code of Conduct

Please see the contributors' guide for details on how to submit a pull request.

License

circe is licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

circe-golden's Issues

Unstable encoding of maps

This is a fantastic tool, it is however troublesome to use as soon as you are encoding maps because the order of keys in generated JSON objects is not stable (also see circe/circe#556).

A way to work around this would be to allow passing an io.circe.Printer to use in io.circe.testing.golden.GoldenCodecLaws.printJson when building the GoldenCodecTests instance. This would allow users to pass Printer.spaces2SortKeys which would guarantee stable encoding and printing of json when maps are involved.

`NullPointerException` when trying to read resources in bloop

Evaluating Resources.inferRootDir is throwing a NPE when it's run in bloop - I believe this is because (in sbt) the initial new File(getClass.getResource("/").toURI) call returns something like

(project root)/target/scala-2.12/test-classes,

but in Bloop it's

(project root)/src/test/resources - the actual source directory.

So in the latter case, the loop:

while (current.ne(null) && current.getName != "target") {
  current = current.getParentFile
}

basically runs until it finds the root directory (/) and its parent, null (because there's usually no target in the path).

It would be great to support the case of running these tests through bloop, as it's often used by users of Metals to avoid duplicate compilations with sbt.

Maybe instead of the fragile classpath-based logic, we could have something compile-time, getting the path of the current file in a sourcecode-like fashion and moving upwards until we hit test, then just adding /resources?

Publish 0.4.0 to maven-central

Hi @zmccoy!

Looks like you've prepared 0.4.0 a while ago, would it be possible to publish it to maven-central?
Thanks in advance!

Use same seed every time you rebuild your golden files

Just making the plan in the README.md into a ticket (so I can watch it, and also ask the following)

Is there a workaround to configure the seed manually? I'm really rusty with scalacheck - as well as this library being sufficiently simple to obscure the "normal" seed-fixing techniques.

Backward-compatible changes fail the golden tests

When we evolve a data type in a backward-compatible way (e.g., by adding or removing an optional property), the golden tests fail because the Arbitrary generators produce different values.

Instead, it would be useful to have tests that support backward-compatible evolutions of data types.

We have implemented it internally at Bestmile, I can make a PR if you are interested. In essence, when we create a new version of the data type, we keep the old golden files and add new golden files. The law we test is the following:

      goldenExamples.flatMap {
        _.traverse { case (encodedString, maybeValue) =>
          maybeValue match {
            case Some(value) =>
              // The golden file contains the current version of the resource.
              // We check that:
              // - encoding the `value` produces the same `encodedString`
              // - decoding the `encodedString` produces the `value`
              parser.decode[A](encodedString)(decode).map { decoded =>
                val decoding = isEqToProp(decoded <-> value)
                val encoding = isEqToProp(printJson(encode(value)) <-> encodedString)
                decoding && encoding
              }.toTry
            case None =>
              // The golden file contains an old version of the resource.
              // We check that:
              // - we can still decode it
              // - encoding and then decoding it produces the same value as
              //   when decoding the old version
              (
                for {
                  decodedOldValue <- parser.decode[A](encodedString)(decode)
                  decodedNewValue <- decode.decodeJson(encode(decodedOldValue))
                } yield isEqToProp(decodedOldValue <-> decodedNewValue)
              ).toTry
          }
        }
      }

Doesn't seem to work with Enumeratum

Hi Travis,

Here's an example to reproduce the problem:

package com.example

import cats.Eq
import enumeratum.{CirceEnum, Enum, EnumEntry}
import enumeratum.EnumEntry.Uppercase
import io.circe.Codec
import io.circe.generic.semiauto._
import io.circe.testing.ArbitraryInstances
import io.circe.testing.golden.GoldenCodecTests
import org.scalacheck.{Arbitrary, Gen}
import org.scalacheck.Arbitrary.arbitrary
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.prop.Configuration
import org.typelevel.discipline.scalatest.FlatSpecDiscipline

object CurrencyGoldenTest {
  sealed abstract class Currency extends EnumEntry
  object Currency extends Enum[Currency] with Uppercase with CirceEnum[Currency] {
    override final val values = findValues

    final case object AUD extends Currency
  }

  final case class Price(value: Double, currency: Currency)
  object Price {
    implicit final val codec: Codec[Price] = deriveCodec
  }

  implicit val arbCurrency: Arbitrary[Currency] = Arbitrary(Gen.oneOf(Currency.values))
  implicit val arbPrice: Arbitrary[Price] = Arbitrary {
    for {
      v <- arbitrary[Double]
      c <- arbitrary[Currency]
    } yield Price(value = v, currency = c)
  }
}

class CurrencyGoldenTest extends AnyFlatSpec with FlatSpecDiscipline with Configuration with ArbitraryInstances {

  import CurrencyGoldenTest._

  implicit final val eqCurrency: Eq[Currency] = Eq.fromUniversalEquals
  implicit final val eqPrice: Eq[Price]       = Eq.fromUniversalEquals

  checkAll("GoldenCodec[Currency]", GoldenCodecTests[Currency].goldenCodec)
  checkAll("GoldenCodec[Price]", GoldenCodecTests[Price].goldenCodec)

}

When I run it, I have the following error:

NotSerializableException was thrown during property evaluation.
  Message: com.example.CurrencyGoldenTest$Currency$
  Occurred when passed generated values (

  )
ScalaTestFailureLocation: org.scalatestplus.scalacheck.CheckerAsserting$$anon$2 at (CurrencyGoldenTest.scala:45)
org.scalatest.exceptions.GeneratorDrivenPropertyCheckFailedException: NotSerializableException was thrown during property evaluation.
  Message: com.example.CurrencyGoldenTest$Currency$
  Occurred when passed generated values (

  )
	at org.scalatestplus.scalacheck.CheckerAsserting$$anon$2.indicateFailure(CheckerAsserting.scala:251)
	at org.scalatestplus.scalacheck.CheckerAsserting$$anon$2.indicateFailure(CheckerAsserting.scala:238)
	at org.scalatestplus.scalacheck.UnitCheckerAsserting$CheckerAssertingImpl.check(CheckerAsserting.scala:160)
 ...

I've added a breakpoint on the GeneratorDrivenPropertyCheckFailedException exception, and we can find the following message in the pos parameter of this type:

Please set the environment variable SCALACTIC_FILL_FILE_PATHNAMES to yes at compile time to enable this feature.

Any idea of how to mitigate that problem and/or where it can come from?

Thanks,
Jules

When you encode something as `Option[Option[String]]` it can not properly decode

Expected: Right(VacancyProfileF(1683177f-5edb-4a9c-8f41-ca1b9e9ac631,Some(PartTime),Some(None),Some(O)))
Recieved: Right(VacancyProfileF(1683177f-5edb-4a9c-8f41-ca1b9e9ac631,Some(PartTime),None,Some(O)))

final case class VacancyProfileF[F[_]](
  id: VacancyProfileId,
  vacancyType: F[VacancyType],
  title: F[Option[NonEmptyString]],
  description: F[NonEmptyString]
)

Where F is set op Option

As you encode Some(None) and expect that.. but it's isomorphic to None which you get back

The use case is, when F is set to cats.Id it's an insert when it's set to Option it's an update where you choose to update specific fields by supplying a Some of something. When there is a Option inside this data structure, like title you'll get a Option[Option[NonEmptyString]]. Setting the title to nothing is Some(None)

File encoding must be explicitly set when reading a file?

On my local machine all the tests runs fine, whenever I run it on my build server it complains with

java.nio.charset.MalformedInputException: Exception raised on property evaluation.> Exception: java.nio.charset.MalformedInputException: Input length = 1

I guess that's because the file contains chinese/arabic characters and must be read with a specific file encoding which is not the default of that operating system?

Release a new version?

Hi @travisbrown ,

Is it possible for you to release a new version, please?
I hit this fixed-in-master problem: #63

Thanks.
Jules 🙂

golden logic outside of the lib

Hi,

Great idea with this lib ! I'm not using it yet but I will recommend it to my team.

I saw in the README that you are concerned with the fragility of writing the golden files. You might want to delegate this to another library.

I maintain https://gitlab.com/tyrcho/approvals-scala and I'd be happy to help with the integration.

Here is a sample project demonstrating a possible integration : https://github.com/WriteThemFirst/approvals-demo-scala

nulls in generated Json break backwards compatiblity checks

Opening this to see if anyone has found a better solution to this.

Consider for example:

case class Person(name: String, address: Option[String])

object Person {

  implicit val decoderForPerson: Decoder[Person] = deriveDecodder[Person

  implicit val encoderForPerson: Encoder[Person] = deriveEncoder[Person]

  implicit val eqForPerson: Eq[Person] = Eq.fromUniversalEquals[Person]
}

The address field is optional and if we are unlucky then scalacheck will generate Person("some name", None)
which circe golden will serialize to

{
  "name": "some name",
  "address": null
}

Unfortunately this means that the address subtree is effectively untested for backwards compatiblity. For example,
the following breaking change to the codec for address still passes the golden tests

case class Person(name: String, address: Option[String])


object Person {

  implicit val decoderForPerson: Decoder[Person] = new Decoder[Person] {
    def apply(c: HCursor): Decoder.Result[Person] =
      for {
        name <- c.downField("name").as[String]
        address <- c.downField("address").as[Option[Value]]
      } yield Person(name, address.map(_.value))
  }

  implicit val encoderForPerson: Encoder[Person] = new Encoder[Person] {

    def apply(a: Person): Json = Json.obj(
      "name" -> a.name.asJson,
      "address" -> a.address.fold(Json.Null)(addr => Json.obj("value" -> addr.asJson))
    )

  }

  implicit val eqForPerson: Eq[Person] = Eq.fromUniversalEquals[Person]
}

case class Value(value: String)

object Value{
  implicit val decoderForValue: Decoder[Value] = deriveDecoder[Value]
}

I've hacked around it using circe-droste:

  val containsNulls: Algebra[JsonF, Boolean] = Algebra {
    case JsonF.NullF       => true
    case JsonF.ArrayF(xs)  => xs.exists(identity)
    case JsonF.ObjectF(xs) => xs.map(_._2).exists(identity)
    case _                 => false
  }

generator.filter(person => !cata(containsNulls).apply(person))

but was wondering if anyone had a better solution? Or if things would go terribly wrong if circe-golden did this by default?

Release a new version?

Hi Travis,

Is it possible for you to release a new version? The conflicts with the new version of disciple-scalatests are annoying and you already updated it in master.

Thanks.
Jules 🙂

Scala 3 support

Has there been any discussion, or planning for supporting scala 3?

Generic case classes and Resources.inferName don't work

There is no file written when I:

checkAll("GoldenCodec[PlanResult[AppointmentDelivery]]", GoldenCodecTests[PlanResult[AppointmentDelivery]].unserializableGoldenCodec)

final case class PlanResult[A](appointment: A, succeeded: Boolean)

Guess Resources.inferName doesn't work with that type