Coder Social home page Coder Social logo

json's Introduction

JSON implementation for Ruby

CI

Description

This is a implementation of the JSON specification according to RFC 7159 http://www.ietf.org/rfc/rfc7159.txt . Starting from version 1.0.0 on there will be two variants available:

  • A pure ruby variant, that relies on the iconv and the stringscan extensions, which are both part of the ruby standard library.
  • The quite a bit faster native extension variant, which is in parts implemented in C or Java and comes with its own unicode conversion functions and a parser generated by the Ragel state machine compiler.

Both variants of the JSON generator generate UTF-8 character sequences by default. If an :ascii_only option with a true value is given, they escape all non-ASCII and control characters with \uXXXX escape sequences, and support UTF-16 surrogate pairs in order to be able to generate the whole range of unicode code points.

All strings, that are to be encoded as JSON strings, should be UTF-8 byte sequences on the Ruby side. To encode raw binary strings, that aren't UTF-8 encoded, please use the to_json_raw_object method of String (which produces an object, that contains a byte array) and decode the result on the receiving endpoint.

Installation

It's recommended to use the extension variant of JSON, because it's faster than the pure ruby variant. If you cannot build it on your system, you can settle for the latter.

Just type into the command line as root:

# rake install

The above command will build the extensions and install them on your system.

# rake install_pure

or

# ruby install.rb

will just install the pure ruby implementation of JSON.

If you use Rubygems you can type

# gem install json

instead, to install the newest JSON version.

There is also a pure ruby json only variant of the gem, that can be installed with:

# gem install json_pure

Compiling the extensions yourself

If you want to create the parser.c file from its parser.rl file or draw nice graphviz images of the state machines, you need Ragel.

Usage

To use JSON you can

require 'json'

to load the installed variant (either the extension 'json' or the pure variant 'json_pure'). If you have installed the extension variant, you can pick either the extension variant or the pure variant by typing

require 'json/ext'

or

require 'json/pure'

Now you can parse a JSON document into a ruby data structure by calling

JSON.parse(document)

If you want to generate a JSON document from a ruby data structure call

JSON.generate(data)

You can also use the pretty_generate method (which formats the output more verbosely and nicely) or fast_generate (which doesn't do any of the security checks generate performs, e. g. nesting deepness checks).

There are also the JSON and JSON[] methods which use parse on a String or generate a JSON document from an array or hash:

document = JSON 'test'  => 23 # => "{\"test\":23}"
document = JSON['test' => 23] # => "{\"test\":23}"

and

data = JSON '{"test":23}'  # => {"test"=>23}
data = JSON['{"test":23}'] # => {"test"=>23}

You can choose to load a set of common additions to ruby core's objects if you

require 'json/add/core'

After requiring this you can, e. g., serialise/deserialise Ruby ranges:

JSON JSON(1..10) # => 1..10

To find out how to add JSON support to other or your own classes, read the section "More Examples" below.

Serializing exceptions

The JSON module doesn't extend Exception by default. If you convert an Exception object to JSON, it will by default only include the exception message.

To include the full details, you must either load the json/add/core mentioned above, or specifically load the exception addition:

require 'json/add/exception'

More Examples

To create a JSON document from a ruby data structure, you can call JSON.generate like that:

json = JSON.generate [1, 2, {"a"=>3.141}, false, true, nil, 4..10]
# => "[1,2,{\"a\":3.141},false,true,null,\"4..10\"]"

To get back a ruby data structure from a JSON document, you have to call JSON.parse on it:

JSON.parse json
# => [1, 2, {"a"=>3.141}, false, true, nil, "4..10"]

Note, that the range from the original data structure is a simple string now. The reason for this is, that JSON doesn't support ranges or arbitrary classes. In this case the json library falls back to call Object#to_json, which is the same as #to_s.to_json.

It's possible to add JSON support serialization to arbitrary classes by simply implementing a more specialized version of the #to_json method, that should return a JSON object (a hash converted to JSON with #to_json) like this (don't forget the *a for all the arguments):

class Range
  def to_json(*a)
    {
      'json_class'   => self.class.name, # = 'Range'
      'data'         => [ first, last, exclude_end? ]
    }.to_json(*a)
  end
end

The hash key json_class is the class, that will be asked to deserialise the JSON representation later. In this case it's Range, but any namespace of the form A::B or ::A::B will do. All other keys are arbitrary and can be used to store the necessary data to configure the object to be deserialised.

If the key json_class is found in a JSON object, the JSON parser checks if the given class responds to the json_create class method. If so, it is called with the JSON object converted to a Ruby hash. So a range can be deserialised by implementing Range.json_create like this:

class Range
  def self.json_create(o)
    new(*o['data'])
  end
end

Now it possible to serialise/deserialise ranges as well:

json = JSON.generate [1, 2, {"a"=>3.141}, false, true, nil, 4..10]
# => "[1,2,{\"a\":3.141},false,true,null,{\"json_class\":\"Range\",\"data\":[4,10,false]}]"
JSON.parse json
# => [1, 2, {"a"=>3.141}, false, true, nil, 4..10]
json = JSON.generate [1, 2, {"a"=>3.141}, false, true, nil, 4..10]
# => "[1,2,{\"a\":3.141},false,true,null,{\"json_class\":\"Range\",\"data\":[4,10,false]}]"
JSON.parse json, :create_additions => true
# => [1, 2, {"a"=>3.141}, false, true, nil, 4..10]

JSON.generate always creates the shortest possible string representation of a ruby data structure in one line. This is good for data storage or network protocols, but not so good for humans to read. Fortunately there's also JSON.pretty_generate (or JSON.pretty_generate) that creates a more readable output:

 puts JSON.pretty_generate([1, 2, {"a"=>3.141}, false, true, nil, 4..10])
 [
   1,
   2,
   {
     "a": 3.141
   },
   false,
   true,
   null,
   {
     "json_class": "Range",
     "data": [
       4,
       10,
       false
     ]
   }
 ]

There are also the methods Kernel#j for generate, and Kernel#jj for pretty_generate output to the console, that work analogous to Core Ruby's p and the pp library's pp methods.

The script tools/server.rb contains a small example if you want to test, how receiving a JSON object from a webrick server in your browser with the JavaScript prototype library http://www.prototypejs.org works.

Speed Comparisons

I have created some benchmark results (see the benchmarks/data-p4-3Ghz subdir of the package) for the JSON-parser to estimate the speed up in the C extension:

 Comparing times (call_time_mean):
  1 ParserBenchmarkExt#parser   900 repeats:
        553.922304770 (  real) ->   21.500x
          0.001805307
  2 ParserBenchmarkYAML#parser  1000 repeats:
        224.513358139 (  real) ->    8.714x
          0.004454078
  3 ParserBenchmarkPure#parser  1000 repeats:
         26.755020642 (  real) ->    1.038x
          0.037376163
  4 ParserBenchmarkRails#parser 1000 repeats:
         25.763381731 (  real) ->    1.000x
          0.038814780
            calls/sec (  time) ->    speed  covers
            secs/call

In the table above 1 is JSON::Ext::Parser, 2 is YAML.load with YAML compatible JSON document, 3 is is JSON::Pure::Parser, and 4 is ActiveSupport::JSON.decode. The ActiveSupport JSON-decoder converts the input first to YAML and then uses the YAML-parser, the conversion seems to slow it down so much that it is only as fast as the JSON::Pure::Parser!

If you look at the benchmark data you can see that this is mostly caused by the frequent high outliers - the median of the Rails-parser runs is still overall smaller than the median of the JSON::Pure::Parser runs:

 Comparing times (call_time_median):
  1 ParserBenchmarkExt#parser   900 repeats:
        800.592479481 (  real) ->   26.936x
          0.001249075
  2 ParserBenchmarkYAML#parser  1000 repeats:
        271.002390644 (  real) ->    9.118x
          0.003690004
  3 ParserBenchmarkRails#parser 1000 repeats:
         30.227910865 (  real) ->    1.017x
          0.033082008
  4 ParserBenchmarkPure#parser  1000 repeats:
         29.722384421 (  real) ->    1.000x
          0.033644676
            calls/sec (  time) ->    speed  covers
            secs/call

I have benchmarked the JSON-Generator as well. This generated a few more values, because there are different modes that also influence the achieved speed:

 Comparing times (call_time_mean):
  1 GeneratorBenchmarkExt#generator_fast    1000 repeats:
        547.354332608 (  real) ->   15.090x
          0.001826970
  2 GeneratorBenchmarkExt#generator_safe    1000 repeats:
        443.968212317 (  real) ->   12.240x
          0.002252414
  3 GeneratorBenchmarkExt#generator_pretty  900 repeats:
        375.104545883 (  real) ->   10.341x
          0.002665923
  4 GeneratorBenchmarkPure#generator_fast   1000 repeats:
         49.978706968 (  real) ->    1.378x
          0.020008521
  5 GeneratorBenchmarkRails#generator       1000 repeats:
         38.531868759 (  real) ->    1.062x
          0.025952543
  6 GeneratorBenchmarkPure#generator_safe   1000 repeats:
         36.927649925 (  real) ->    1.018x 7 (>=3859)
          0.027079979
  7 GeneratorBenchmarkPure#generator_pretty 1000 repeats:
         36.272134441 (  real) ->    1.000x 6 (>=3859)
          0.027569373
            calls/sec (  time) ->    speed  covers
            secs/call

In the table above 1-3 are JSON::Ext::Generator methods. 4, 6, and 7 are JSON::Pure::Generator methods and 5 is the Rails JSON generator. It is now a bit faster than the generator_safe and generator_pretty methods of the pure variant but slower than the others.

To achieve the fastest JSON document output, you can use the fast_generate method. Beware, that this will disable the checking for circular Ruby data structures, which may cause JSON to go into an infinite loop.

Here are the median comparisons for completeness' sake:

 Comparing times (call_time_median):
  1 GeneratorBenchmarkExt#generator_fast    1000 repeats:
        708.258020939 (  real) ->   16.547x
          0.001411915
  2 GeneratorBenchmarkExt#generator_safe    1000 repeats:
        569.105020353 (  real) ->   13.296x
          0.001757145
  3 GeneratorBenchmarkExt#generator_pretty  900 repeats:
        482.825371244 (  real) ->   11.280x
          0.002071142
  4 GeneratorBenchmarkPure#generator_fast   1000 repeats:
         62.717626652 (  real) ->    1.465x
          0.015944481
  5 GeneratorBenchmarkRails#generator       1000 repeats:
         43.965681162 (  real) ->    1.027x
          0.022745013
  6 GeneratorBenchmarkPure#generator_safe   1000 repeats:
         43.929073409 (  real) ->    1.026x 7 (>=3859)
          0.022763968
  7 GeneratorBenchmarkPure#generator_pretty 1000 repeats:
         42.802514491 (  real) ->    1.000x 6 (>=3859)
          0.023363113
            calls/sec (  time) ->    speed  covers
            secs/call

Development

Release

Update the json.gemspec and json-java.gemspec.

rbenv shell 2.6.5
rake build
gem push pkg/json-2.3.0.gem

rbenv shell jruby-9.2.9.0
rake build
gem push pkg/json-2.3.0-java.gem

Author

Florian Frank mailto:[email protected]

License

Ruby License, see https://www.ruby-lang.org/en/about/license.txt.

Download

The latest version of this library can be downloaded at

Online Documentation should be located at

json's People

Contributors

burdettelamar avatar byroot avatar eregon avatar flori avatar headius avatar hsbt avatar jonleighton avatar jvshahid avatar k0kubun avatar kares avatar lorensr avatar mame avatar marcandre avatar mernen avatar mmacvicar avatar mrkn avatar msp-greg avatar nobu avatar nurse avatar olleolleolle avatar perlun avatar sho-h avatar sorah avatar tenderlove avatar vipulnsward avatar watson1978 avatar yui-knk avatar zenspider avatar znz avatar zverok avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

json's Issues

unexpected token when encoding and parsing 1.chr

Nevermind. This is a Rails bug:

Loading development environment (Rails 2.3.5)
>> JSON::dump(["foo\001bar"])
=> "[\"foo\001bar\"]"

However:

irb(main):001:0> require 'json'
=> true
irb(main):002:0> JSON::dump(["foo\001bar"])
=> "[\"foo\\u0001bar\"]"

Specify signature on to_json?

In absolute terms a splat signature can be quite a bit slower than a specific signature because it causes the creation of a new array on each call (as shown below). Normally it's not a big deal because you need to create a LOT of arrays to see the difference, but nonetheless I was wondering if you have considered putting a specific signature on the to_json method, in particular for the pure-ruby variant? Thanks.

require 'benchmark'

# % ruby signature_benchmark.rb 
#                                user     system      total        real
# with arg                   0.030000   0.000000   0.030000 (  0.033594)
# with default               0.050000   0.000000   0.050000 (  0.049114)
# with splat                 0.050000   0.000000   0.050000 (  0.045439)
#
Benchmark.bm(25) do |x|
  def one(arg=[])
  end

  def two(*)
  end

  n = 100000
  arg = []

  x.report "with arg" do
    n.times { one(arg) }
  end

  x.report "with default" do
    n.times { one }
  end

  x.report "with splat" do
    n.times { two }
  end
end

json/ext 1.5.0 on 32-bit ree-1.8.7 sometimes produces garbage output on Bignums

Here's a sample program that exhibits the problem:

ree-1.8.7@jsonbug1 steinbergd@davedev:~/dev/timing-bug$ cat test1.rb
require 'rubygems'
require 'json/ext'

puts "JSON Version: #{JSON::VERSION}"
1000000.times do |i|
s = 12345678900000.to_json
puts s
end
ree-1.8.7@jsonbug1 steinbergd@davedev:/dev/timing-bug$ ruby test1.rb | sort | uniq -c
2 �ï¿5678900000
999461 12345678900000
1 �ï¿5678900000
533 ï¿5678900000
3 @ï¿5678900000
1 JSON Version: 1.5.0
ree-1.8.7@jsonbug1 steinbergd@davedev:
/dev/timing-bug$ uname -a

Linux davedev 2.6.32-24-generic #41-Ubuntu SMP Thu Aug 19 01:12:52 UTC 2010 i686 GNU/Linux

Also on pastebin: http://pastebin.com/fpxF5Tzd

Switching the test program to json/pure fixes it. In addition, monkey patching Bignum.to_s like so:

class Bignum
def to_json
return self.to_s
end
end

also seems to fix it.

Add option to parse object keys as symbols

The app I work on has a heavily used JSON-based API and I'm pretty sure I'd get some efficiencies out of being able to get hashes straight from JSON.parse with symbolized keys... like:

{ :request_type => 'create', :entity_type => 'Ticket' ...etc }

If you could add that as an option, that'd be awesome...

json_pure 1.4.3 appears to break RFC spec?

Related to: http://github.com/flori/json/issues/closed#issue/12

On the front page of the ruby JSON project is the statement:

"Both variants of the JSON generator escape all non-ASCII an control characters with \uXXXX escape sequences, .."

which appears to contradict the sentence following that:

"This means that generated JSON text is encoded as UTF-8 "

If the generate should always escapes all non-ASCII stuff, then it should always produce ASCII, no? (According to my read of the RFC, this would be correct behaviour).

If the generated JSON text is (by default) encoded as UTF-8, a subset of that always qualifies as non-ASCII, no? (According to my read of the RFC, this would be incorrect behaviour).

I understand there's an option :ascii_only to accomplish the ASCII-only behaviour, but shouldn't that be the default and the UTF-8 generation be the thing that requires an option? Or am I (and cobbr2) missing something here?

1.4.3 from rubygems fails to create json with the message: NameError: uninitialized constant JSON::SAFE_STATE_PROTOTYPE

1.4.3 from rubygems fails to create json with the message:
NameError: uninitialized constant JSON::SAFE_STATE_PROTOTYPE

here is an example from irb using 1.4.3 from rubygems:

 Welcome to interactive ruby!
 irb --> require 'json'
     ==> true

 irb --> nil.to_json
 NameError: uninitialized constant JSON::SAFE_STATE_PROTOTYPE
    from (irb):2:in `to_json'
    from (irb):2
 irb --> "this".to_json
 NameError: uninitialized constant JSON::SAFE_STATE_PROTOTYPE
    from (irb):3:in `to_json'
    from (irb):3

and again, using json 1.1.9
irb --> require 'json'
==> true

 irb --> nil.to_json
     ==> "null"

 irb --> {:this => 'that'}.to_json
     ==> "{"this":"that"}"

Already initialized constants on Ruby 1.9.2

When I have json_pure in my project, every time I run rake something, it outputs this first:

/home/steven/.rvm/gems/ruby-1.9.2-p0/gems/json_pure-1.4.6/lib/json/common.rb:65: warning: already initialized constant State
/home/steven/.rvm/gems/ruby-1.9.2-p0/gems/json_pure-1.4.6/lib/json/common.rb:66: warning: already initialized constant SAFE_STATE_PROTOTYPE
/home/steven/.rvm/gems/ruby-1.9.2-p0/gems/json_pure-1.4.6/lib/json/common.rb:67: warning: already initialized constant FAST_STATE_PROTOTYPE
/home/steven/.rvm/gems/ruby-1.9.2-p0/gems/json_pure-1.4.6/lib/json/common.rb:74: warning: already initialized constant PRETTY_STATE_PROTOTYPE
/home/steven/.rvm/gems/ruby-1.9.2-p0/gems/json_pure-1.4.6/lib/json/pure.rb:76: warning: already initialized constant JSON_LOADED

Highly annoying.

Extrange error parsing a hash

require 'json'

test1 = { "userdata"=>"PK\001\004\024\023\01230\010\b\000{{5;\37" }
p test1.to_json
test2 = { "userdata"=>"PK\001\004\024\023\01230\010\b\000{{5;
\377" }
p test2.to_json

"{"userdata":"PK\u0001\u0004\u0014\u0013\n30\b\b\u0000{{5;~\u001f"}"
test.rb:6:in `to_json': partial character in source, but hit end (JSON::GeneratorError)
from test.rb:6

What can be happen with this data?

Thanks in advance

Add a customisable field to the state argument for use in to_json methods

It would be powerfull to be able to pass custormized options in the to_json params which would be available for reading by user defined to_json methods.
Actually state and depth params are propagated but there is no way to set any user defined options. This could be done by adding an option field to the state object from which user defined to_json methods would be able to read application specific options.

truncation in exception message

There seems to be something truncating exception messages. In this example I try to parse an invalid string, and check the length of the exception message.

require 'json'

s = '"' + '*' * 9000 + '"'

begin
  JSON::parse(s)
rescue JSON::ParserError => e
  puts e.message.length
  puts e.message[0..75]
  puts e.message[-75..-1]
end

begin
  raise JSON::ParserError, "618: unexpected token at '#{s}'"
rescue JSON::ParserError => e
  puts e.message.length
  puts e.message[0..75]
  puts e.message[-75..-1]
end

produces:

8191
618: unexpected token at '"*************************************************
***************************************************************************
9029
618: unexpected token at '"*************************************************
*************************************************************************"'

That's with json 1.1.9 on ruby 1.8.7. With json 1.2.0 on ruby 1.8.6 I get:

511
655: unexpected token at '"*************************************************
***************************************************************************
9029
618: unexpected token at '"*************************************************
*************************************************************************"'

So it seems like there are internal limits somewhere of 8192 and 512 (with NULL).

What's causing that truncation?

Encode versus. force_encode in pure json

I have been trying to upgrade a project to ruby 1.9.1 today and I think I found an issue in the pure code base.
I have a library that is giving me an already encoded byte string with encoding ASCII-8BIT. The first thing the pure library does is to check and see if its encoded with UTF-8 and if not call encode(). This explodes immediately: Encoding::UndefinedConversionError: "\xED" from ASCII-8BIT to UTF-8. If I call force_encoding before passing the string in it works and I can round trip data through the conversion and back again without an issue. Is there a reason "encode" was used there rather than "force_encode"? Another clue may be that the ext version fails on this string regardless of whether I forced the encoding or not. Below is a test string I have that has been failing like this.

"\n\x04test\x10\x00\x18\x02:\b\x1A\x020\x01 \x01(\x00H\xED\xE1\x8E\xE8\x8C%"

segmentation faut while using json in testing environment

I'm trying to run some tests:
rake test:functionals TEST=test/functional/abuse_controller_test.rb
and get error:
/home/uzzz/dev/project/vendor/rails/activesupport/lib/active_support/json/encoding.rb:79: [BUG] Segmentation fault

My configuration:
ruby 1.8.7 (2010-01-10 patchlevel 249) [i486-linux]
json 1.4.3
running on rails 2.3.8

json/ext handles isolated escaped surrogates inconsistently

Using json 1.4.3, Ruby 1.8.7p174 and 1.9.2-preview1.

When given a high surrogate with no low surrogate, it gives nil for the string:

JSON.parse(%q{[123,"x\uD834x"]}    # => [123, nil]

When given a low surrogate with no high surrogate, it treats it as a valid character:

JSON.parse %q{[123,"x\uDD5Ex"]}    # => [123, "x\355\265\236x"]

In both cases, json/pure throws JSON::GeneratorErrors due to iconv's strictness (IllegalSequence on OS X, InvalidCharacter on Linux).

Unicode Issue with Native Ext

Hi,

I came across the error JSON::GeneratorError: source sequence is illegal/malformed for valid utf-8.

Once I load "json/pure" it works fine.

Test case:

"\355\257\200".to_json # should fail
require 'json/pure'
"\355\257\200".to_json # should work.

Revert "simplified build structure"

The the new ext/ structure is worse than the old way. It uses non-standard extconf_*.rb names and combines multiple extensions into one folder.

76e017a
josh/json@993f261ccb8f911d2ae57e9db48ec7acd0187283

You should model the gem's structure the same way as the bundled version in ruby-core.

http://github.com/ruby/ruby/tree/trunk/ext/json

You should have 2 dirs for "generator/extconf.rb" and "parser/extconf.rb". Also you need to specify the install target in the Makefile. This fix was applied to ruby-core but never backported here.

ruby/ruby@3b1e5ba

(http://chneukirchen.github.com/rps/)

json 1.4.3 breaks to_json on subclasses of String

While the 1.4.3 seems to have fixed an issue with subclasses of Hash, it seems to have broken to_json entirely for subclasses of String, which previously worked ok.

Easy to replicate:

>> require 'json/ext'
=> true
>> JSON::VERSION
=> "1.4.3"
>> class X < String; end
=> nil
>> X.new("abc").to_json
SystemStackError: stack level too deep
    from (irb):4:in `to_json'
    from (irb):4

In our case this was showing up when serializing a Sequel::SQL::Blob value.

Error with Rails 2.3.2

I get this error issuing generated JSON to a rails app: (note, the return trip of JSON encoded hash comes back through just fine - I just can't seem to issue any JSON to the server), as follows:

D, [2010-03-26T17:36:48.079660 #13579] DEBUG -- : Retrieving WSDL from: http://localhost:3333/transactions/wsdl
D, [2010-03-26T17:36:48.268882 #13579] DEBUG -- : SOAP request: http://localhost:3333/transactions/api
D, [2010-03-26T17:36:48.269147 #13579] DEBUG -- : SOAPAction: /transactions/api/transfer_funds, Content-Type: text/xml;charset=UTF-8
D, [2010-03-26T17:36:48.279064 #13579] DEBUG -- : <env:Envelope xmlns:wsdl="urn:rBankTransactions" xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">env:Bodywsdl:transfer_funds{"from_acct":"0","to_acct":"1","amount":0.0}/wsdl:transfer_funds/env:Body/env:Envelope
D, [2010-03-26T17:36:48.532153 #13579] DEBUG -- : SOAP response (status 500):
D, [2010-03-26T17:36:48.532379 #13579] DEBUG -- : Internal protocol error: undefined method collect' for #<SOAP::SOAPString:0xb69bfe68> Backtrace: /usr/lib/ruby/gems/1.8/gems/actionwebservice-2.3.2/lib/action_web_service/protocol/soap_protocol.rb:61:indecode_request'
/usr/lib/ruby/gems/1.8/gems/actionwebservice-2.3.2/lib/action_web_service/protocol/soap_protocol.rb:45:in decode_action_pack_request' /usr/lib/ruby/gems/1.8/gems/actionwebservice-2.3.2/lib/action_web_service/protocol/discovery.rb:20:indiscover_web_service_request'
/usr/lib/ruby/gems/1.8/gems/actionwebservice-2.3.2/lib/action_web_service/protocol/discovery.rb:18:in each' /usr/lib/ruby/gems/1.8/gems/actionwebservice-2.3.2/lib/action_web_service/protocol/discovery.rb:18:indiscover_web_service_request'
/usr/lib/ruby/gems/1.8/gems/actionwebservice-2.3.2/lib/action_web_service/dispatcher/action_controller_dispatcher.rb:49:in dispatch_web_service_request' (eval):1:inapi'

if I send just straight text to test the basic protocols and delivery mechanism all works just dandy. It is only when I pass the result of JSON.generate() that it blows up. Any ideas on what is doing this and how to solve the issue?

Thanks in advance.

json_pure should be able to parse "null"

JSON.parse("null") should return nil instead of raising
"JSON::ParserError: 618: unexpected token at 'null' "
It seems like this affects any version.

json_ext parses "null" correctly.

json gem is broken on JRuby

The json gem uses a native extension, which means any attempts to even install it on JRuby fails.

The json-jruby gem acts as a drop-in replacement for the json gem on JRuby:

However, this doesn't help when there are gems which explicitly require "json" as a dependency. The simple solution here is to release json-jruby as "json" for the Java platform. That way gems that depend on json would install the regular json gem on all other platforms and json-jruby on the Java platform.

This would require no changes to the existing "json" gem. Instead all that's needed is for the existing json-jruby gem to be released as "json" for the Java platform on rubygems.org

1.4.5 does not reset nesting depth after an exception

>> c = []
>> c << c
>> c.to_json
JSON::NestingError: nesting of 20 is too deep
>> [].to_json
JSON::NestingError: nesting of 21 is too deep
>> [].to_json
JSON::NestingError: nesting of 22 is too deep
>> JSON::SAFE_STATE_PROTOTYPE.depth
22

One way around that would be to make #to_json (with no arguments) duplicate SAFE_STATE_PROTOTYPE, but we'd still fall for people who pass it manually, or create their own Generator::State configuration objects.

A few solutions:

  • Decrease the depth counter on an ensure block. Sounds expensive.
  • Separate configuration objects and generation state objects (which hold actual state; at the moment, depth is the only value). Might be possible to do that with no API changes.

JSON::ParserError: 665: unexpected token at 'true'

When trying to parse response from the twitter api the following happens:

http://gist.github.com/351755

Which seems to be a JSON problem.

Twitter API delivers "true" or "false".
http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-friendships-exists

Seems to be a problem between twitter's way to response (is it valid JSON?) and the JSON gem's way to parse it.

This is how to reproduce the problem with JSON itself:
http://gist.github.com/351769

Now the question is whether to fix it in the twitter or json gem (or tell twitter to change their api :-))

See also the issue created for the twitter gem:
http://github.com/jnunemaker/twitter/issues/#issue/31

Better test coverage

The library needs more extensive testing. Through bug reports on json-jruby, I've noticed the following parts are not covered:

  • Bignums
  • Non-UTF strings
  • Strings with isolated surrogates (see issue #38)
  • Generator::State getters and setters (settings are always being passed on initialization)
  • Strings whose internal buffer offset is nonzero (a rather specific implementation detail, but could be useful nonetheless), i.e.:
    str = 'xx["foobar"]'
    substr = str[2..-1]
    JSON.parse(substr) # confuses json-jruby 1.1.5

JSON.dump in pure_json fails

irb(main):003:0> JSON.dump([1,2,3])
NoMethodError: undefined method `generate' for #<JSON::Ext::Generator::State:0x000000024a3580>
    from /home/manveru/.gem/ruby/1.9.1/gems/json_pure-1.4.3/lib/json/common.rb:212:in `generate'
    from /home/manveru/.gem/ruby/1.9.1/gems/json_pure-1.4.3/lib/json/common.rb:336:in `dump'
    from (irb):3
    from /usr/bin/irb:12:in `<main>'

Add option to generate non-escaped UTF-8 characters

In my app, JSON-encoded strings are pre-cached in MySQL from a composite of multiple model objects, which can't afford encoding on-the-fly for n-records.

In that case, the table definition of MySQL had to be changed from VARCHAR(1000) to VARCHAR(6000) because previously a raw UTF-8 character was counted as 1 but now it's 6, by escaping in the form of "\uXXXX". It has some storage and I/O impact.

In addition, it makes debugging a little harder for public-facing web APIs, when you can read non-ASCII characters. It's much easier to find "日本語" rather than "\u65e5\u672c\u8a9e" in occasions that you can't use debug tools but have to read from raw transport logs.

I noticed that there's to_json_raw method in String, but there's no public method to use it.

I realized this JSON lib is much faster and I don't want to look back, so I'd much appreciate if you consider this as an option.

Please provide binaries for MinGW version of Ruby

Hello Florian,

First thanks for providing binaries for Ruby on Windows using rake-compiler.

As maintainer of RubyInstaller, the new replacement of One-Click, we are using MinGW (GCC) as compiler, which uses a different platform than current One-Click

For that purpose, I've added in rake-compiler a few versions ago the possibility to generate gems for both mswin32 and mingw32.

The change is simple, and requires alter the cross_platform definition from a string to an array of strings containing i386-mswin32 and i386-mingw32

Also, it is possible to generate fat-binaries which can be installed in both 1.8 and 1.9 versions of Ruby, and the changes required to make that happen are minimal.

Either way, we all appreciate your effort to support MinGW platform in the future.

Thank you.

RuntimeError: conversion undefined for byte sequence (maybe invalid byte sequence)

I encounter an error when trying to encode/decode a JSON string containing UTF-8 characters.

Here is a simple example running in irb1.9:

irb(main):001:0> require 'json'
=> true
irb(main):002:0> JSON::VERSION
=> "1.2.3"
irb(main):003:0> jsonstring = '{"name":"Eric Bréchemier"}'
=> "{\"name\":\"Eric Bréchemier\"}"
irb(main):004:0> jsonstring.encoding
=> #<Encoding:UTF-8>
irb(main):005:0> json = JSON.parse(jsonstring)
=> {"name"=>"Eric Br\xC3\xA9chemier"}
irb(main):006:0> JSON.generate(json)
RuntimeError: conversion undefined for byte sequence (maybe invalid byte sequence)
    from /var/lib/gems/1.9.0/gems/json-1.2.3/lib/json/common.rb:188:in `encode'
    from /var/lib/gems/1.9.0/gems/json-1.2.3/lib/json/common.rb:188:in `to_json'
    from /var/lib/gems/1.9.0/gems/json-1.2.3/lib/json/common.rb:188:in `to_json'
    from /var/lib/gems/1.9.0/gems/json-1.2.3/lib/json/common.rb:188:in `generate'
    from (irb):6
    from /usr/bin/irb1.9:12:in `<main>'

Is it a known issue? Any idea how I could avoid such errors?

json 1.4.3 introduces a change that causes SystemStackError with ohai

Users installing ohai today after the json 1.4.3 release are experiencing the following problem. This in turn breaks chef. The problem does not occur with json_pure, just json. Happens on OS X 10.5 (ruby 1.8.6), and Ubuntu 9.10 (ruby 1.8.7). The issue is specifically introduced in commit 82ed5b7.

/usr/lib/ruby/gems/1.8/gems/json-1.4.3/lib/json/common.rb:270:in `to_json': stack level too deep (SystemStackError)
    from /usr/lib/ruby/gems/1.8/gems/json-1.4.3/lib/json/common.rb:270:in `generate'
    from /usr/lib/ruby/gems/1.8/gems/json-1.4.3/lib/json/common.rb:270:in `pretty_generate'
    from /usr/lib/ruby/gems/1.8/gems/ohai-0.5.0/lib/ohai/system.rb:218:in `json_pretty_print'
    from /usr/lib/ruby/gems/1.8/gems/ohai-0.5.0/lib/ohai/application.rb:96:in `run_application'
    from /usr/lib/ruby/gems/1.8/gems/ohai-0.5.0/lib/ohai/application.rb:67:in `run'
    from /usr/lib/ruby/gems/1.8/gems/ohai-0.5.0/bin/ohai:40
    from /usr/bin/ohai:19:in `load'
    from /usr/bin/ohai:19

ubuntu910$ ruby --version
ruby 1.8.7 (2009-06-12 patchlevel 174) [i486-linux] 

osx105$ ruby --version
ruby 1.8.6 (2009-06-08 patchlevel 369) [universal-darwin9.0]

Parser does not support JSON string

Here's a test case:

irb(main):001:0> JSON.parse("foo".to_json)
JSON::ParserError: 705: unexpected token at '"foo"'
    from /usr/lib/ruby/gems/1.8/gems/json-1.4.6/lib/json/common.rb:146:in `parse'
    from /usr/lib/ruby/gems/1.8/gems/json-1.4.6/lib/json/common.rb:146:in `parse'
    from (irb):1

json and json_pure are same

Hello,

I can't seem to install json_pure gem. After checking the downloads from rubygems.org these even have the same MD5 checksums.

$ sudo gem install json_pure
Building native extensions.  This could take a while...
Successfully installed json-1.4.3
1 gem installed
Installing ri documentation for json-1.4.3...
Installing RDoc documentation for json-1.4.3...

After running this I still don't have access to json_pure gem.

As an example: http://github.com/datamapper/dm-serializer/issues/2 ; in this case, dm-serializer depends on json_pure which doesn't work as expected, since installing json_pure only appears to install json.

extconf.rb confuses itself on OS X

extconf.rb currently checks for ruby/re.h, which is the location of the re.h header on 1.9. Sadly on OS X though, this causes gcc to check for the re.h header in the system ruby.framework.

In my case, this meant that building json under Rubinius cause it to try and include the MRI version of re.h, which doesn't work.

The linked patch only checks for ruby/re.h (and ruby/encoding.h) if the ruby running extconf is 1.9, and thus it's expected to find the headers it wants at those locations. This fixes the build bug and json builds and runs cleanly on Rubinius.

Patch: http://gist.github.com/405577

require 'json' does NOT failover to json_pure

Usage notes say "JSON first tries to load the extension variant. If this fails, the pure variant is loaded and used."

But in my tests it seems that json_pure is not sufficient. I have a script that includes "require 'json'":

opensrf@dev-vm7:/$ sudo gem install json_pure
Successfully installed json_pure-1.4.6
1 gem installed
Installing ri documentation for json_pure-1.4.6...
Installing RDoc documentation for json_pure-1.4.6...
opensrf@dev-vm7:
/$ ./myscript.rb
/usr/local/lib/site_ruby/1.8/rubygems.rb:779:in report_activate_error': Could not find RubyGem json (>= 1.1.3) (Gem::LoadError) from /usr/local/lib/site_ruby/1.8/rubygems.rb:214:inactivate'
from /usr/local/lib/site_ruby/1.8/rubygems.rb:249:in activate' from /usr/local/lib/site_ruby/1.8/rubygems.rb:248:ineach'
from /usr/local/lib/site_ruby/1.8/rubygems.rb:248:in activate' from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:35:inrequire'
from ./edi_webrick.rb:10

$ uname -a
Linux dev-vm7 2.6.26-1-xen-amd64 #1 SMP Fri Mar 13 21:39:38 UTC 2009 x86_64 GNU/Linux

I am getting constants re-initialized when using autotest with rspec and rails environment.

This "warning" feedback is what happens when I run autotest from my bash shell. Otherwise I am not detecting any other issues.

~/.rvm/gems/ruby-1.9.2-p0@rails3/gems/json_pure-1.4.6/lib/json/common.rb:65: warning: already initialized constant State
~/.rvm/gems/ruby-1.9.2-p0@rails3/gems/json_pure-1.4.6/lib/json/common.rb:66: warning: already initialized constant SAFE_STATE_PROTOTYPE
~/.rvm/gems/ruby-1.9.2-p0@rails3/gems/json_pure-1.4.6/lib/json/common.rb:67: warning: already initialized constant FAST_STATE_PROTOTYPE
~/.rvm/gems/ruby-1.9.2-p0@rails3/gems/json_pure-1.4.6/lib/json/common.rb:74: warning: already initialized constant PRETTY_STATE_PROTOTYPE
~/.rvm/gems/ruby-1.9.2-p0@rails3/gems/json_pure-1.4.6/lib/json/pure.rb:76: warning: already initialized constant JSON_LOADED

JSON.parse("foo".to_json) gives unexpected token error

I think a standalone string value (without a key) is a valid JSON, and actually Facebook REST API returns auth token in this form, starting with a double quote, followed by auth token string, followed by a double quote. It'd be great if the gem could cover that case.

irb(main):002:0> JSON.parse("foo".to_json)
JSON::ParserError: 665: unexpected token at '"foo"'

to_json crashes in v1.4.6

This simple command throws an Exception. Ruby 1.8.7p174.

{:x=>3}.to_json

TypeError: wrong argument type JSON::Pure::Generator::State (expected Data)
from ./vendor/bundler_gems/gems/json-1.4.6/lib/json/pure/generator.rb:283:in to_json' from ./vendor/bundler_gems/gems/json-1.4.6/lib/json/pure/generator.rb:283:injson_transform'
from ./vendor/bundler_gems/gems/json-1.4.6/lib/json/pure/generator.rb:276:in each' from ./vendor/bundler_gems/gems/json-1.4.6/lib/json/pure/generator.rb:276:injson_transform'
from ./vendor/bundler_gems/gems/json-1.4.6/lib/json/pure/generator.rb:258:in `to_json'
from (irb):1

add/rails.rb incompatible with rails 2.3.x

require 'json/add/rails'
class Person < ActiveRecord::Base ; end
Person.new.to_json # => {'attributes' => ..., 'attributes_cache' => ..., ...}

This is clearly not the behaviour we would want. Why does this happen?

# active_record/base.rb
class ActiveRecord::Base
  def to_json
    super
  end
end
# active_support/json.rb
class Object
  def to_json(options = nil)
    ActiveSupport::JSON.encode(as_json(options))
  end
end

When requiring json/add/rails.rb after ActiveSupport, the to_json method is overridden and rails json serialization breaks. Is there a reason this file still exists? It seems to just copy functionality that is already present in ActiveSupport.

Encountered this issue using nanite, which requires this file in the json library.

warning: method redefined; discarding old recurse_proc

I'm getting the following warning running ./script/autospec on a Rails project, Ruby 1.9.1.

/Users/adam/.gem/ruby/1.9.1/gems/json-1.1.9/lib/json/common.rb:274: warning: method redefined; discarding old recurse_proc

I'm not really sure where this problem lies, since I don't see it if I rake spec. It seems like maybe the recurse_proc function has already been defined by the system JSON? I also can't easily get this to appear in irb. I'm raising the issue here in the hopes you will know what it relates to.

Note: it also happens if I uninstall the 'json' gem and leave the 'json_pure' gem. Currently, I have 2 gems which depend on 'json_pure', 'gemcutter' and 'rubyforge'.

JSON.parse JSON.unparse "foobar" doesn't work

$ ruby191 -v -rjson -e 'p JSON.parse JSON.unparse "foobar"'
ruby 1.9.1p129 (2009-05-12 revision 23412) [i686-linux]
json/common.rb:274: warning: method redefined; discarding old recurse_proc
json/common.rb:122:in parse': 574: unexpected token at '"foobar"' (JSON::ParserError) from json/common.rb:122:inparse'
from -e:1:in `

'

$ ruby -v -rjson -e 'p JSON.parse JSON.unparse "foobar"'
ruby 1.9.2dev (2009-09-28 trunk 25126) [i686-linux]
json/common.rb:273: warning: method redefined; discarding old recurse_proc
json/common.rb:122:in parse': 618: unexpected token at '"foobar"' (JSON::ParserError) from json/common.rb:122:inparse'
from -e:1:in `

'

Original post is [ruby-core:25829] string objects don't survive round-trip in yaml&json.
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/25829

RFC4627 doesn't allow such JSON, so I think JSON.unparse shouldn't return such JSON.
http://www.ietf.org/rfc/rfc4627.txt

Integer keys in hash not preserved

I was wondering why Integer keys in hashes are not preserved after a JSON generate/parse cycle?

Here an example:
require 'json'

hash = {7,"some value"}
puts hash[7]
hash.keys.each {|key| puts "Key is Integer? #{key.is_a?(Integer)}\nKey is String? #{key.is_a?(String)}"}

new_hash = JSON.parse(JSON.generate(hash))
puts new_hash[7]
new_hash.keys.each {|key| puts "Key is Integer? #{key.is_a?(Integer)}\nKey is String? #{key.is_a?(String)}"}`

This will output:
some value
Key is Integer? true
Key is String? false
nil
Key is Integer? false
Key is String? true

I check the tests and saw that this is apparently expected behavior,
as in test_json_generate.rb I find:
json = generate({1=>2})
assert_equal('{"1":2}', json)

I would be interested to know why the Integer keys cannot be preserved?
Thanks.

numbers larger than a 32 bit signed int cause a range error with ruby 1.9

Hi,
using the json gem version 1.4.0 with ruby 1.9.1 and 1.9.2 HEAD, I get a range error when attempting to convert an integer larger than a 32 bit signed integer to json:

irb(main):025:0> RUBY_VERSION
=> "1.9.2"
irb(main):026:0> {:foo => ((1 << 31) -1)}.to_json
=> "{\"foo\":2147483647}"
irb(main):027:0> {:foo => ((1 << 31) )}.to_json
RangeError: integer 2147483648 too big to convert to `int'
from (irb):27:in `to_json'
from (irb):27
from /Users/ddeleo/.rvm/rubies/mput-head/bin/irb:15:in `<main>'

And:

rb(main):003:0> RUBY_VERSION
=> "1.9.1"
irb(main):004:0> {:foo => ((1 << 31) -1)}.to_json
=> "{\"foo\":2147483647}"
irb(main):005:0> {:foo => ((1 << 31))}.to_json
RangeError: integer 2147483648 too big to convert to `int'
from (irb):5:in `to_json'
from (irb):5
from /Users/ddeleo/.rvm/rubies/ruby-1.9.1-p378/bin/irb:15:in `<main>'

Thanks,
Dan DeLeo

Custom class inheriting from Hash does not follow its to_json

I create a class that inherits from Hash and has a custom to_json method defined.

I create an instance, x, of this class. When I call x.to_json I get the right thing. But if x is embedded in a normal Hash it doesn't follow x's to_json method.

This had the correct behavior in 1.2.3 but is incorrect in 1.4.2.

Sample code below:

require "rubygems"
require "json"

class CustomHash < Hash
  def to_json(*a)
    {
      'json_class'   => self.class.name,
      'data'         => 'dummy data'
    }.to_json(*a)
  end

  def self.json_create(o)
    new
  end
end


x = CustomHash.new

puts x.to_json # => {"json_class":"CustomHash","data":"dummy data"}
puts ({"a" => x}.to_json)
  # in 1.2.3 => {"a":{"json_class":"CustomHash","data":"dummy data"}}
  # in 1.4.2 => {"a":{}}

Importing json makes to_json not return json

irb

require 'ostruct'

=> ["OpenStruct"]

test =OpenStruct.new

=> #

test.hi = 'hi'

=> "hi"

test.to_json

=> "{"table": {"hi": "hi"}}"

require 'json'

=> []

test.to_json

=> ""#<OpenStruct hi=\"hi\">""

Generate JSON with sorted Hash keys

It is OK to generate json with any order of keys in Hash according to spec, but it would be nice if we could force json genrator to emit keys in same order. When json generates keys in random order it's very hard to compare two versions of json with diff or similar tool. I solved this problem for me by forcing Hash always emit pairs by each method in same order:

class Hash
  def each(&proc)
    keys.sort{|a1, a2| a1.to_s <=> a2.to_s}.each do |k|
      proc.call(k, self[k])
    end
  end
end

But it would be nice to have such option in json library.

1.4.3 segfaults on hash sublcass


paul@scratch:current $ bundle exec irb
irb(main):001:0> require 'json'
=> true
irb(main):002:0> {}.to_json
=> "{}"
irb(main):003:0> class Foo < Hash; end
=> nil
irb(main):004:0> f = Foo.new
=> {}
irb(main):005:0> f[:foo] = "bar"
=> "bar"
irb(main):006:0> f.to_json
Segmentation fault
paul@scratch:current $ 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.