Coder Social home page Coder Social logo

waargonaut's Introduction

Build Status

Waargonaut

Flexible, precise, and efficient JSON decoding/encoding library. This package provides a plethora of tools for decoding, encoding, and manipulating JSON data.

Features

  • Fully RFC compliant, with property based testing used to ensure the desired invariants are preserved.

  • Encoders and Decoders are values, they are not tied to a typeclass and as such you are not tied to a single interpretation of how a particular type "should" be handled.

  • No information is discarded on parsing. Trailing whitespace, and any formatting whitespace (carriage returns etc) are all preserved.

  • A history keeping zipper is used for Decoding, providing precise control of how you decode your JSON data. With informative error messages if things don't go according to plan.

  • Flexible and expressive Decoder & Encoder functions let you parse and build the JSON structures you require, with no surprises.

  • BYO parsing library, the parser built into Waargonaut does not tie you to a particular parsing library. With the caveat that your parsing library must have an instance of CharParsing from the parsers package.

  • Generic functions are provided to make the creation of Encoders and Decoders are bit easier. However these are tied to typeclasses, so they do come with some assumptions.

  • Lenses, Prisms, and Traversals are provided to allow you to investigate and manipulate the JSON data structures to your hearts content, without breaking the invariants.

  • The awesome work on succinct data structures by John Ky and Haskell Works is used to power the decoder. Providing the same zipper capabilities and property based guarantees, but with all the speed and efficiency capabilities that succinct data structures have to offer.

Example

  • Data Structure:
data Image = Image
  { _imageWidth    :: Int
  , _imageHeight   :: Int
  , _imageTitle    :: Text
  , _imageAnimated :: Bool
  , _imageIDs      :: [Int]
  }
  • Encoder:
encodeImage :: Applicative f => Encoder f Image
encodeImage = E.mapLikeObj $ \img ->
    E.intAt "Width" (_imageWidth img)
  . E.intAt "Height" (_imageHeight img)
  . E.textAt "Title" (_imageTitle img)
  . E.boolAt "Animated" (_imageAnimated img)
  . E.listAt E.int "IDs" (_imageIDs img)
  • Decoder:
imageDecoder :: Monad f => D.Decoder f Image
imageDecoder = D.withCursor $ \curs -> do
  -- Move down into the JSON object.
  io <- D.down curs
  -- We need individual values off of our object,
  Image
    <$> D.fromKey "Width" D.int io
    <*> D.fromKey "Height" D.int io
    <*> D.fromKey "Title" D.text io
    <*> D.fromKey "Animated" D.bool io
    <*> D.fromKey "IDs" (D.list D.int) io

Zippers

Waargonaut uses zippers for its decoding which allows for precise control in how you interrogate your JSON input. Take JSON structures and decode them precisely as you require:

Input:
["a","fred",1,2,3,4]
Data Structure:
data Foo = Foo (Char,String,[Int])
Decoder:

The zipper starts the very root of the JSON input, we tell it to move 'down' into the first element.

fooDecoder :: Monad f => Decoder f Foo
fooDecoder = D.withCursor $ \cursor -> do
  fstElem <- D.down cursor

From the first element we can then decode the focus of the zipper using a specific decoder:

  aChar <- D.focus D.unboundedChar fstElem

The next thing we want to decode is the second element of the array, so we move right one step or tooth, and then attempt to decode a string at the focus.

  aString <- D.moveRight1 fstElem >>= D.focus D.string

Finally we want to take everything else in the list and combine them into a single list of Int values. Starting from the first element, we move right two positions (over the char and the string elements), then we use one of the provided decoder functions that will repeatedly move in a direction and combine all of the elements it can until it can no longer move.

  aIntList <- D.moveRightN 2 fstElem >>= D.rightwardSnoc [] D.int

Lastly, we build the Foo using the decoded values.

  pure $ Foo (aChar, aString, aIntList)

The zipper stores the history of your movements, so any errors provide information about the path they took prior to encountering an error. Making debugging precise and straight-forward.

Property Driven Development

This library is built to parse and produce JSON in accordance with the RFC 8259 standard. The data structures, parser, and printer are built to satify the Round Trip Property:

Which may be expressed using the following pseudocode:

parse . print = id

This indicates that any JSON produced by this library will be parsed back in as the exact data structure that produced it. This includes whitespace such as carriage returns and trailing whitespace. There is no loss of information.

There is also this property, again in pseudocode:

print . parse . print = print

This states that the printed form of the JSON will not change will be identical after parsing and then re-printing. There is no loss of information.

This provides a solid foundation to build upon.

NB: The actual code will of course return values that account for the possibility of failure. Computers being what they are.

TODO(s)

In no particular order...

  • improve/bikeshed encoding object api
  • gather feedback on tests/benchmarks that matter
  • provide testing functions so users can be more confident in their Encoder/Decoder construction
  • (feedback required) documentation in the various modules to explain any weirdness or things that users may consider to be 'missing' or 'wrong'.
  • (mostly) provide greater rationale behind lack of reliance in typeclasses for encoding/decoding
  • provide functions to add preset whitespace layouts to encoded json.

waargonaut's People

Contributors

benkolera avatar dalaing avatar emilypi avatar enemeth79 avatar gwils avatar luke-clifton avatar mankykitty avatar sheaf avatar treffynnon avatar tshinohara avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

waargonaut's Issues

Help: What is the best way to encode optional fields?

Given a record like this:

  data Property = SomeProperty {
      name :: Text,
      description :: Text
    }
    | FancyProperty {
      name :: Text,
      description :: Text

      total :: Maybe Int,
      average :: Maybe Int,
      required :: Maybe Bool
    }
    deriving (Show)

I then want to write a property encoder that looks something like this:

import qualified Waargonaut.Encode as WE

encodeProperty :: Applicative f => WE.Encoder f Property
encodeProperty = WE.mapLikeObj $ \jss ->
  WE.textAt "name" (name jss) .
  WE.textAt "description" (description jss) .
  WE.textAt "type" (encodePropertyType jss) .
  WE.boolAt "required" (required jss)   -- this field only exists in one of the Property record's constructors

encodePropertyType :: Property -> Text
encodePropertyType x = case x of
  SomeProperty {} -> "some"
  FancyProperty {} -> "fancy"

Help: Prettier

OK, so there are no tests here to cheat from and I found if I passed in an Int to the Prettier functions (as the docs currently read to me) they'd both reject it.

This is what I've come up with to get it working, but I figure I've probably taken a detour somewhere:

import qualified Waargonaut.Encode as WE
import qualified Waargonaut.Prettier as WEP
import qualified Waargonaut.Types as WT
import qualified Data.Text.Lazy as TL

encode :: [J.Schema] -> [TL.Text]
encode xs =
    let spaces = WEP.NumSpaces $ successor' $ successor' zero' -- this is fun with Natural instead of 2::Int
        indent = WEP.IndentStep $ successor' $ successor' zero'
        beautify = WEP.simpleEncodePretty WEP.ArrayOnly indent spaces WE.json'
        json = WE.asJson' encodeSchema <$> xs
    in runIdentity . beautify <$> json

To get a decent print out in GHCI I am then calling it with:

import qualified Data.Text.Lazy.IO as T

main :: IO ()
main = do
  params <- -- fetches data here of type J.Schema
  foldr (const . T.putStr) (pure ()) $ JS.encode params

J.Schema looks like this:

data Schema = Schema {
  id :: Text,
  schema :: Text,
  schemaDescription :: Text,
  schemaProperties :: Properties
} deriving (Show)

What's the best way of utilising the Prettier stuff and how does one overcome the Natural impediment?

Moving down into an empty object or list fails.

This isn't what I was expecting.

D.withCursor $ \c -> D.down c >> pure ()

will fail on "{}" or "[]". I was expecting the D.down to succeed, but then any D.moveRight to cause an error. Is this documented somewhere?

Default / example parser?

Would you be interested in a PR that included a few default parsing functions (similar to or exactly like https://github.com/qfpl/waargonaut/blob/master/test/Types/Common.hs#L255) along with docs related to them?

My team recently decided to try Waargonaut and it took us a little while to piece together what was needed to construct a parser and we thought we could help reduce that time for another team.

I'm asking because I'm not sure if this would seem to be stepping on the toes of "BYO parsing library" by suggesting a default implementation.

Changing compiler via nix doesn't apply overlays

The overlay in waargonaut-deps.nix only applies to the default haskellPackages and not to the one selected if the compiler is changed (by passing the compiler argument to (shell|default).nix.

To reproduce:

nix-shell --argstr compiler ghc862

Expected: dropped in a nix shell

Actual: error saying that generics-sop-0.3.2.0 cannot be built. The overlay specifies version 0.4.0.1 of generics-sop, so it appears the overlays are not working.

Building using Nix fails for GHC: 7.10.3 & 8.0.2

Build fails for the following reason:

Configuring concurrent-output-1.10.4...
Setup: At least the following dependencies are missing:
process >=1.6.0 && <1.7.0

Not sure if this is a straight forward fix as process doesn't exist in haskell.packages.ghc7103 due to being a boot library for GHC (https://ghc.haskell.org/trac/ghc/wiki/Commentary/Libraries/VersionHistory). So overriding it causes more terrible errors.

Waargonaut happily builds for these versions of GHC when nix isn't involved (see TravisCI).

My nix skills aren't enough for this, so help would very much be appreciated.

documentation incorrect

The haddock for simpleEncodePretty says that it prints to a ByteString. In fact, it prints to a Text.

Not available in Stackage

I am using the resolver 18.10, which is very recent, and it seems like this package is not available: https://www.stackage.org/lts-18.10/hoogle?q=waargonaut

It would be a lot easier to use for stack users, if it were available in Stackage. I have attempted to add the package using extra-deps and the versions that stack suggest, but possibly due to the lack of a solver, Stack suggests a broken configuration:

Dependency cycle detected in packages:
    [witherable,witherable-class,witherable,waargonaut,argus]

In the dependencies for witherable-class-0.0.1:
    witherable dependency cycle detected: witherable, witherable-class, witherable, waargonaut

Closing braces aren't checked. `[ }` is a valid list.

λ import Types.Common (parseBS)  -- From the test suite
λ import Waargonaut.Decode as D
λ D.runPureDecode (D.list D.int) parseBS $ D.mkCursor "[]"
Right []
λ D.runPureDecode (D.list D.int) parseBS $ D.mkCursor "[}"  -- <--- oops
Right []

Waargonaut.Decode.either's document or implementation maybe wrong

Hi, thank you for creating this library!
I just started learning it and I like it.

Reading through Waargonaut.Decode module, I realized that the either function attemps the Left decoder first, even though the document states "Right decoder is attempted first.".

For example:

> D.simpleDecode (D.either D.scientific D.int) parseBS "123"
Right (Left 123.0)

If the Right decoder is attempted first, the result should be Right (Right 123).

Fix CI for GHC 8.0.2

Currently CI always fails on GHC 8.0.2, which is annoying. Local builds are ok. It would be good to either fix it or remove GHC 8.0.2 support. My preference is the former.

The CI is conservative and builds with installed versions of boot dependencies. I think if we simply undid that, 8.0.2 would build again.

Encoder mapLikeObj API thoughts

In my opinion the current API for creating a 'map-like' JSON object using the
mapLikeObj and atKey functions is awkward and not as straight-forward as I would like.

In the current form, it looks like this:

personEncoder :: Applicative f => Encoder f Person
personEncoder = E.mapLikeObj $ \p ->
  E.atKey' "name" E.text (_personName p) .
  E.atKey' "age" E.int (_personAge p) .
  E.atKey' "address" E.text (_personAddress p) .
  E.atKey' "numbers" (E.list E.int) (_personFavouriteLotteryNumbers p)

With the atKey' functions being composed together to create the final object.
However the available generalisation over f complicates things a bit when both
the mapLikeObj function AND the atKey functions both have general and
Identity implementations. But neither mapLikeObj function works when used
with the generalised over f atKey function.

Also, the fact that these functions are composed isn't always obvious to newer users
of the library.

Also the general type of the atKey functions can produce errors that are difficult to untangle:

atKey :: (At t, IxValue t ~ Json, Applicative f) => Index t -> Encoder f a -> a -> t -> f t
atKey' :: (At t, IxValue t ~ Json) => Index t -> Encoder' a -> a -> t -> t

Contrast this to the glorious ease of decoding the equivalent object:

personDecoder2 :: Monad f => Decoder f Person
personDecoder2 = Person
  <$> D.atKey "name" D.text
  <*> D.atKey "age" D.int
  <*> D.atKey "address" D.text
  <*> D.atKey "numbers" (D.list D.int)

I would like to have something for creating 'map-like' objects that has a more
obvious interface as well as being more difficult to use incorrectly.

Maybe something like:

data ObjKV f a = OKV
  { _objkvKey :: Text
  , _objkvEncoder :: Encoder f v
  , _objkvGetter :: a -> (v ????)
  }

mkObj :: (Applicative f, Foldable g) => g (OKV f a) -> a -> Encoder f a

-- Then encoding a map-like object is more obviously:
mapLike :: Applicative f => Encoder f a
mapLike = E.encodeA . mkObj
  [ OKV "name" E.text  _personName
  , OKV "age" E.int _personAge
  , OKV "address" E.text _personAddress
  , OKV "numbers" (E.list E.int) _personFavouriteLotteryNumbers
  ]

I'm not sure, I hacked that out as an idea...

Try to keep the suggestions reasonable. Whilst I'm not against burning the
Encoder structure to the ground in the name of a far superior alternative, I
would expect a sufficiently compelling justification. You need more than
"I/we/this package/that package/bob/susan does it this way and I like it, so you
should do that."

Likewise, nerdsnipes like "Build a better representation in ATS2 with linear
types and just use the FFI" are ❤️ , but no.

Conversion of Scientific to JNumber computes an incorrect exponent

ghci> jNumberToScientific $ _JNumberScientific # ( fromFloatDigits @Double 5 )
Just 5.0
ghci> jNumberToScientific $ _JNumberScientific # ( fromFloatDigits @Double 0.5 )
Just 5.0

This seems to be because of an off-by-one error in _JNumberScientific. I don't understand why the testsuite doesn't catch this.

Fix in #85.

Discover worst-case key search scenario

The moveToKey function is a O(n) search from its current position to the desired key or an error from the end of the object. The succinct data structures make this incredibly fast and the movements are super efficient.

But how efficient this is remains to be measured. Create benchmarks for an increasing in size ordered key object and find out where we run into performance problems by looking for the last key.

Use of Natural (from Numeric.Natural) is unsafe

Waargonaut's movement code uses Natural from Numeric.Natural, which doesn't actually enforce that a number is natural via the type system, but rather throws an impure arithmetic exception when provided with negative input.

Reproduction:

λ> :i manyMoves
manyMoves ::
  Monad m => GHC.Natural.Natural -> (b -> m b) -> b -> m b
  	-- Defined at /Users/jkachmar/code/waargonaut/src/Waargonaut/Decode.hs:215:1
λ> manyMoves (-1) undefined undefined
*** Exception: arithmetic underflow

Still not available on Stackage

Hi! I'm unsuccessfully trying to build a project with recommended extra-deps.

stack.yaml:

resolver: lts-19.18

# some stuff ...

extra-deps:
  - waargonaut-0.8.0.2
  - digit-0.11@sha256:d3b42df305e64756505e46dabad32d08df3721b2f58e7c4741cfd6a73ccaaef2,3888
  - hoist-error-0.2.1.0@sha256:b7ccf72fe0dc2339a06956a0388115a33550cbe08089c7c2da310d1ef7aa7b9c,1353
  - lens-4.19.2@sha256:d4d704141d7c322bbfb746157b0709f3b966dfec92421f571c34069893af08cc,16062
  - natural-0.3.0.6@sha256:6149f7df782b5fa0d7e6685319fed73cacbe3efb0721acff4d2c34e0f6d92205,2042
  - witherable-0.3.5@sha256:9febcda439f514aec8f430bd3c75a7cda20b28978a8cbad1b2f8f097a3a16b42,1542
  - Cabal-3.2.1.0@sha256:8743076ec022296f9771d962000c9ca3a0fe02e68c37b992c63e382a675f791d,27482
  - template-haskell-2.16.0.0@sha256:71d8c2e92b712695cb4fd0be9cd1a41b05c25315e9748cfef1132bf4d02b6556,1917

package.yaml:

dependencies:
  - base >= 4.7 && < 5
  - aeson
  - bytestring
  - text
  - unordered-containers
  - containers
  - stm
  - wai
  - wai-logger
  - warp
  - servant
  - servant-server
  - websockets
  - wai-websockets
  - servant-websockets
  - wai-app-static
  - hashable
  - waargonaut
  - digit
  - hoist
  - lens
  - natural
  - witherable
  - Cabal
  - template-haskell

# some stuff ...

Without the last two (Cabal and template-haskell) got such message:

Error: While constructing the build plan, the following exceptions were encountered:

Dependency cycle detected in packages:
    [aeson,monoidal-containers,witherable,aeson,backend]

In the dependencies for backend-0.1.0.0:
    hoist needed, but the stack configuration has no specified version (no package with that name found, perhaps there is a typo in a package's build-depends or an
          omission from the stack.yaml packages list?)
    witherable dependency cycle detected: witherable, monoidal-containers, witherable, aeson, backend
needed since backend is a build target.

In the dependencies for lens-4.19.2:
    Cabal-3.4.1.0 from stack configuration does not match >=1.10 && <3.3  (latest matching version is 3.2.1.0)
    template-haskell-2.17.0.0 from stack configuration does not match >=2.4 && <2.17  (latest matching version is 2.16.0.0)
needed due to backend-0.1.0.0 -> lens-4.19.2

In the dependencies for monoidal-containers-0.6.2.0:
    aeson dependency cycle detected: aeson, monoidal-containers, witherable, aeson, backend
    witherable dependency cycle detected: witherable, monoidal-containers, witherable, aeson, backend
needed due to backend-0.1.0.0 -> monoidal-containers-0.6.2.0

In the dependencies for servant-0.19:
    aeson dependency cycle detected: aeson, monoidal-containers, witherable, aeson, backend
needed due to backend-0.1.0.0 -> servant-0.19

In the dependencies for servant-server-0.19.1:
    aeson dependency cycle detected: aeson, monoidal-containers, witherable, aeson, backend
needed due to backend-0.1.0.0 -> servant-server-0.19.1

In the dependencies for servant-websockets-2.0.0:
    aeson dependency cycle detected: aeson, monoidal-containers, witherable, aeson, backend
needed due to backend-0.1.0.0 -> servant-websockets-2.0.0

In the dependencies for waargonaut-0.8.0.2:
    witherable dependency cycle detected: witherable, monoidal-containers, witherable, aeson, backend
needed due to backend-0.1.0.0 -> waargonaut-0.8.0.2

In the dependencies for wai-extra-3.1.12.1:
    aeson dependency cycle detected: aeson, monoidal-containers, witherable, aeson, backend
needed due to backend-0.1.0.0 -> wai-extra-3.1.12.1

Dependency cycle detected in packages:
    [witherable,monoidal-containers,witherable,aeson,backend]

Unknown package: witherable-class

_optionsFieldName is called twice when generating an encoding using Waargonaut.Generic.gEncoder

When I use gEncoder to generate an encoder for a data type, setting an _optionsFieldName field name modifier function, that function appears to be called twice, which is a problem if it isn't idempotent. To reproduce:

{-# LANGUAGE DataKinds             #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE TemplateHaskell       #-}
{-# LANGUAGE TypeFamilies          #-}
module Test where

import Data.Text          (Text)
import Generics.SOP.TH    (deriveGeneric)
import Waargonaut.Encode  (Encoder)
import Waargonaut.Generic (GWaarg, JsonEncode (..), Options (..),
Tagged (..),
                           defaultOpts, gEncoder, untag)

data Foo = Foo
  { abc :: Text
  }
$(deriveGeneric ''Foo)

instance JsonEncode GWaarg Foo where
  mkEncoder = gEncoder (defaultOpts { _optionsFieldName  = tail })

fooEncoder :: Applicative f => Encoder f Foo
fooEncoder = untag (mkEncoder :: Applicative f => Tagged GWaarg
(Encoder f Foo))

Now to test in ghci:

λ simpleEncodeText fooEncoder (Foo "test")
"{"c":"test"}"

I expected to see:

λ simpleEncodeText fooEncoder (Foo "test")
"{"bc":"test"}"

Readme code would not compile

Ok, this is a pedantic issue, I'm not sure how you feel about it. The readme should rather say

The data structures, parser, and printer are built to comply with the following *pseudo* properties

Since parse . print is more akin to Right rather than id :-)

Error Reporting

Errors need to be more descriptive and precise. We don't have an error not finding a key on an object, for example. We have the history, but it'd be nice if the message was more expressive.

Also the history reporting hasn't been thoroughly battle tested so more than likely needs tweaking as well.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.