JSON::ParserError: unexpected characters after the JSON document () at line 30, column 2 [sparse.c:642] about oj_mimic_json

[michaelperret]

Any progress on this? Also seeing this behavior when installing action-text

Rails 6.0.3.2
ruby 2.7.0p0

[talhajunaid63]

Thanks for the quick reply. Sure let me do it.

[ohler55]

I'll look into it. It may be that the https response is a steam or has the read methods to act like a stream. Anyway, let me do some digging.

[ohler55]

@honeyryderchuck made the fix so using the latest https should work fine.

[ohler55]

Ok, will try running with your repo.

[ohler55]

I have not done anything in regard to this issue since I am not able to get the described environment working. If someone can give me a smaller test case I'm happy to work on a solution but trying to recreate the environment has not worked for me.

[ohler55]

thanks, will try to get to it this week.

[ohler55]

Created an issue on httpx: https://gitlab.com/os85/httpx/-/issues/237

[ohler55]

Any way you can show me what the string you are trying to parse looks like?

[swiknaba]

Not sure, what is being parsed. Sorry, my description is not very clear.

I've done:
Shell: Rails new app-name
Ad the oj-gems
Shell: install rich text ---> parsing error

So the error must happen inside the install script from Action-Text..

[ohler55]

How do I run the repo. I tried rails console and that was okay or sort of. It complained about yarn being out of date. I also tried bundle exec rails action_text:install.

[swiknaba]

bundle install should be enough. Maybe yarn install as well. I can check out the repo on a different machine tomorrow and try again (it's getting late in Europe^^).

[ohler55]

I managed to get past the yarn issue. You can turn off integrity checking. Anyway. What is going on is that Oj is given a stream to parse. That stream contains more than one JSON object so Oj complains when it sees the second object. That is expected behaviour so I need to figure out how that differs from the JSON gem.

If I have Oj return without an error the process continues but then runs into other errors later in the process.

Options I am considering are adding an option to not raise but just return and then turning on that option for the JSON gem mimicking.

[ohler55]

Sorry, I stopped it after not getting a response. I can pick it up again this weekend. To summarize and get my head back into the issue, you are parsing a stream that contains multiple JSON documents and you would like to see Oj return without errors after the first document is read. Is that correct?

[ohler55]

Are you able to describe what you expect? Was my description in the previous comment correct?

[swiknaba]

Options I am considering are adding an option to not raise but just return and then turning on that option for the JSON gem mimicking.

this sounds like a good idea to me. It seems to be wrong to just return an ignore errors, but if that is what the JSON gem is doing, then that's what should be expected from a "mimic-json" gem I guess?

[ohler55]

I guess the real question is what is returned, the first or the last JSON document in the string or file. As long as I have a sounding board I'll take a look at what makes the most sense and throw it up here for discussion.

Thanks

[ohler55]

okay, you repo has been cloned and I ran bundle install. I tried bundle exec rails action_text:install but even after installing the yarn gem I get complaints about yarn not being included in the bundle. If you have time would you mind updating your repo with what you would like to to work with and I'll give it another try.

[ramaboo]

Just ran into this today. Any solutions?

[ohler55]

I need more to go on to stand any chance of fixing what ever you are encountering. Some sample code or project would be great.

[talhajunaid63]

Just ran into this today, Any solutions? seeing this behavior when installing action-text

Rails: 6.1.4.1
Ruby: 3.0.2

[swiknaba]

I updated the repo linked above (on Jul 15, so already a while ago. Sry. forgot to ping you here about the update).
@talhajunaid63 maybe you can push a scaffold for a fresh rails app using rails 6.1 and Ruby 3?

[hhassan01]

Any update guys? Still stuck on this issue :/

[swiknaba]

I've further slimmed down the dummy app: https://github.com/swiknaba/testojmimic/blob/master/README.md

And also hacked together a docker image; you can download it here: https://hub.docker.com/layers/lud11/swiknaba/testojmimic/images/sha256-18822523defe24a7937a397acf6b0d0d5401604e4d0e527ab41dde45a120ddc7?context=repo

Here is a short demo of how to boot the docker image, including showing that the bug is present when using oj-mimic-json and is resolved by removing the gem: https://www.loom.com/share/3304286f50ff47c79228b58e0482eea4

I hope that helps the investigation :)

[ohler55]

I know this has been going on for a long time. At this point I still don't know what the JSON file looks like that is causing the issue. If you want to solve this and are willing to zoom maybe we can get on a call and try to see what is going on together. Reach me at [email protected].

[jgarth]

Hi all, hope I can add some relevant info to this. I encountered the same error when using HTTPX in combination with Oj. Upon investigation it turns out that given a ruby object body, which prints as #<HTTPX::Response::Body:20400 @state=memory @length=339>, I can do Oj.load(body.to_s), but not Oj.load(body). I can however do JSON.parse(body).

AFAICT, Oj checks if something is a ruby String here, and if it isn't, uses a different method of parsing. It doesn't call .to_s or inspect. Arguably it's not Oj's job to call to_s on anything before parsing it, either. Maybe the default JSON implementation does that. Either way, some 3rd party code depends on this behavior.

My C-fu is not strong enough to figure out what exactly Oj is doing after the call to oj_pi_sparse, so I'm not sure how to progress from here. But I can now give you a really simple reproduction case and hope it helps:

require 'httpx'
require 'oj'

# Works as intended
puts "Without Oj"
response = HTTPX.get('https://dummyjson.com/products/1')
puts JSON.parse(response.body)

# Raises "JSON::ParserError: unexpected characters after the JSON document…"
puts "With Oj"
Oj.mimic_JSON()
response = HTTPX.get('https://dummyjson.com/products/1')
puts JSON.parse(response.body)

Let me know if there's anything I can do to assist further, and thanks for making this gem @ohler55 🎉

[ohler55]

It turns out the https response does respond to the read method but it always returns the body as a full string so repeated calls to read always return a full string which makes the response appear to be an infinitely long string. Of course that is not how the read method is supposed to work. I'll have to look for a way to detect that broken behaviour and work around it.