scalaj / scalaj-http Goto Github PK

Hi,

I seem to be getting a SocketTimeoutException every time I try to use param or params.

Here's my code:

println(Http("https://api.twitter.com/1/statuses/mentions.json").param("include_entities", "true").asString)

This also does not work:

println(Http("https://api.twitter.com/1/statuses/mentions.json?include_entities=true").asString)

Exception:

(run-main) java.net.SocketTimeoutException: connect timed out
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
at java.net.Socket.connect(Socket.java:529)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:564)
at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:395)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:272)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:329)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:172)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:158)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:133)
at scalaj.http.Http$$anonfun$1.apply(Http.scala:192)
at scalaj.http.Http$$anonfun$1.apply(Http.scala:192)
at scalaj.http.Http$Request.process(Http.scala:96)
at scalaj.http.Http$Request.apply(Http.scala:84)
at scalaj.http.Http$Request.asString(Http.scala:112)
at budogo.Runner$.main(Runner.scala:13)
at budogo.Runner.main(Runner.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
java.lang.RuntimeException: Nonzero exit code: 1

I'm using version: 0.2.9

If I leave out the parameters everything works fine. Any ideas?

Scala 2.9.1 has been out for a little while now. The installation instructions use a cross-build configuration, but that fails on Scala 2.9.1 because there aren't any 2.9.1 artifacts yet. I locked down to an older version for now, but it'd be nice to fix this on the maven repos.

I am trying to take scalaj-http into use in my Scala Play2 application. This is how I set my dependency for sbt:

"org.scalaj" % "scalaj-http_2.10" % "0.3.8"

This, however, yields the error message in the title:
[warn] [NOT FOUND ] junit#junit;4.10!junit.src (130ms)
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/junit/junit/4.10/junit-4.10-sources.src
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: FAILED DOWNLOADS ::
[warn] :: ^ see resolution messages for details ^ ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: junit#junit;4.10!junit.src
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
sbt.ResolveException: download failed: junit#junit;4.10!junit.src
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:114)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:114)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:102)
...

Running with 1.1.3 I started getting gzipped bytes from twitter again. switching back to 1.1.1 I get strings. Looks like the 1.1.3 somehow broke the content-header case insensitivity.

Hi there,

is it possible to do HTTP PUT commands? Something like:

Http.put("http://foo.com/add").data(""" {"some": "data"} """)

Regards,
Sven

First, thanks for you work !

When I try to use this lib with an incorrect URL, it blocks.
I can see, with Wireshark, the dns response error : No such name.
I have solved the problem by adding a try/catch (see below).
It's a little bit ugly, but it seems to work.

def exec[T](parser: (Int, Map[String, String], InputStream) => T): HttpResponse[T] = {
    val urlToFetch: URL = new URL(urlBuilder(this))
    try{
      proxyConfig.map(urlToFetch.openConnection).getOrElse(urlToFetch.openConnection) match {
        case conn: HttpURLConnection =>
          conn.setInstanceFollowRedirects(false)
          HttpOptions.method(method)(conn)
          if (compress) {
            conn.setRequestProperty("Accept-Encoding", "gzip,deflate")
          }
          headers.reverse.foreach{ case (name, value) =>
            conn.setRequestProperty(name, value)
          }
          options.reverse.foreach(_(conn))

          connectFunc(this, conn)
          try {
            toResponse(conn, parser, conn.getInputStream)
          } catch {
            case e: java.io.IOException if conn.getResponseCode > 0 =>
              toResponse(conn, parser, conn.getErrorStream)
          } finally {
            closeStreams(conn)
          }
      }
    }catch{
      case e: Exception =>
        HttpResponse(e.asInstanceOf[T], 502, Map())
    }
  }

It seems I can make GET requests successfully to a given site - except to "http://www.dilbert.com". Then I get a java.net.SocketTimeoutException, although the site seems to be responding to HTTP requests normally.

Please see below:

Welcome to Scala version 2.10.0 (Java HotSpot(TM) Client VM, Java 1.7.0_17).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import scalaj.http.Http
import scalaj.http.Http

scala> val html = Http("http://www.dilbert.com").asString
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)

But the site is up:

kypeli@iceberg:~$ curl -I http://www.dilbert.com
HTTP/1.1 200 OK
Date: Sun, 21 Jul 2013 09:26:08 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.2.10
Set-Cookie: PHPSESSID=3oeodcel2pacdgd7cpoel95vo5; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=5nvtfb4m3qsm1hiknsglo6lmq4; path=/
Set-Cookie: PHPSESSID=69t1p2kc5afs3de3gi75ouuoq2; path=/
Set-Cookie: PHPSESSID=tbfjgtkn6dk6mgogtdv6evl0p1; path=/
Set-Cookie: PHPSESSID=5punq0q7rsm8b9q4357l2lb8a1; path=/
Connection: close
Content-Type: text/html; charset=utf-8lias

And I can make requests to other sites:

scala> val html = Http("http://www.google.com").asString
html: String =
<!doctype html><title>Google</title><script>(function(){
window.google={kEI:"TKnrUYe4I-zZ0QXx7IGIDg",getEI:function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI},https:function(){return"https:"==window.location.

Hi...I'm wrapping my request inside a Future{blocking{ Http .... }} and I have a list of 200 urls for call freebase service

val foo=List[String].map(myHttpFutureMethod))

this return a List of Future[String]...so far so good
now when I do Await.result(Future.sequence(foo),10 minutes) I get

java.net.SocketTimeoutException: connect timed out
  at java.net.PlainSocketImpl.socketConnect(Native Method)
  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:589)
  at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
  at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
  at sun.net.www.http.HttpClient.New(HttpClient.java:308)
  at sun.net.www.http.HttpClient.New(HttpClient.java:326)
  at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1167)
  at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1146)
  at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:987)
  at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:985)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.security.AccessController.doPrivileged(AccessController.java:713)
  at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:984)
  at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:931)
  at scalaj.http.Http$$anonfun$1.apply(Http.scala:296)
  at scalaj.http.Http$$anonfun$1.apply(Http.scala:296)
  at scalaj.http.Http$Request.process(Http.scala:159)
  at scalaj.http.Http$Request.apply(Http.scala:141)
  at scalaj.http.Http$Request.asString(Http.scala:207)
  at $anonfun$YQL4$1$$anonfun$apply$1$$anonfun$apply$2.apply(<console>:29)
  at $anonfun$YQL4$1$$anonfun$apply$1$$anonfun$apply$2.apply(<console>:29)
  at scala.util.Try$.apply(Try.scala:191)
  at .retry(<console>:23)
  at $anonfun$YQL4$1$$anonfun$apply$1.apply(<console>:26)
  at $anonfun$YQL4$1$$anonfun$apply$1.apply(<console>:26)
  at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$2$$anon$4.block(ExecutionContextImpl.scala:48)
  at scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)
  at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$2.blockOn(ExecutionContextImpl.scala:45)
  at scala.concurrent.package$.blocking(package.scala:54)
  at $anonfun$YQL4$1.apply(<console>:25)
  at $anonfun$YQL4$1.apply(<console>:25)
  at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
  at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
  at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
  at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
  at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
  at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
  at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I test the code with only 10 url and works good...but when I use much more then fails
I've played with HttpOptions.connTimeout and HttpOptions.readTimeout and change the number of thread pools in Executors.newFixedThreadPool(N) but always fails, dont care what number I put

how could fix it??...thanks!!

Love this library and use it for everything, but I've hit a weird edge case that prevents me from using it in one particular case.

For reasons I won't go into, I have no DNS lookup available so I have to use the IP address, and the service I'm trying to reach is on a subdomain. I've deduced that the way around this using curl or some other posting tool is to set the host header to what would be the subdomain.domain.tld I'd like to reach.

Attempting to set the host header using the header method does not work, apparently this is a security feature of the underlying java library. Ok fair enough.

I tried the same trick with another library (Play's WS) and setting the host is also not allowed there, however, they provide a method called withVirtualHost("subdomain.domain.tld") and that does the magic.

I'd love it if something similar appeared in this library so I can ditch WS! :)

If I get some time I might have a go but it's unlikely :( so I thought I should capture this in an issue.

I need to extract some information from a 302 response before following it, but I can't find a method that allows me to not follow the redirect. Did I miss it or does it not currently exist?

Under certain circumstances a NPE is thrown when trying to get the response string:

java.lang.NullPointerException
at scalaj.http.Http$.tryParse(Http.scala:128)
at scalaj.http.Http$Request.process(Http.scala:95)
at scalaj.http.Http$Request.apply(Http.scala:78)
at scalaj.http.Http$Request.asString(Http.scala:106)

Seems that the conn.getInputStream is null. Is that an expected condition?

How could someone implement the data parts of a multipart ?

These are a seperate section from files.

For example my web service framework is using a method like multipartBody.dataParts("some key in here") to get data parts and multipartBody.file("filename") to get files.

To make more clear what I am talking about I will note this from official specificaiton: A form data part of a multipart looks like that at the end (if boundary is *****):

--*****
Content-Disposition: form-data; name="some name"\n
\n
value here\n
\n

First, thank you for this library, it has simplified my life greatly.

(Also, I was able to resolve my issue by using 0.3.12, so not a big problem for me)

The method HttpURLConnection.setFixedLengthStreamingMode(long contentLength) is only available on android-19 or above, previously only taking an 'int contentLength'. So, on android (scaloid) when sending a multipart post, the method cannot be found with that signature. I am not sure if it is possible to support both long and int signatures, but I thought I should bring this to your attention. (Http.scala, line 326)

Having set a val request = Http.postData(...
for instance when you call request.asHeadersAndParse(... then the request is actually executed in order to get the response.

Could you have an request.asHeadersAndParse method run asynchronously (a future or something) and also have a request.cancelConnection method to cancel the current connection.

I am not implying to interrupt a certain thread, only to terminate connection and get thrown an HttpException as you would normally get after connection or read timeout.

If the above is possible would be really useful.
Thanks
pligor

I see you can already have a body (for example json) in a request using "postData" method.
But If I want to perform a DELETE verb/method I start the command like that:

Http(link).option(HttpOptions.method("DELETE")).
header("Content-Type", "application/json"). ..... ?

How could I include the json body?
Thank you!

Hi I tried to invoke a cross domain request using scalaj which eventually failed.
is there any hack to make it work
please let us know
Thanks in Advance

I'm trying to use scalaj-http but I'm getting errors classfile version errors (looks like the artifacts on sonatype are v51). I'm deploying on macs that default to java 1.6.

Hey, your library seems to be awesome, but I can't find it for Scala 2.10.
Are you supporting it? Will it work?

Cheers,
A.

twitter is sending the content-encoding header in lower case and this isn't being detected by scalaj (unless I'm doing something wrong). This means that for twitter I am decoding manually but for facebook no problem.

Can it (should it) be made case insensitive?

This is the kind of code I am using

   request.apply {
      inputStream =>
        copyInputToOutputImperative(inputStream, tempOutstream, bufferSize);
    }

The input stream is empty!
How to I trigger the request?
Normally I would do request.responseCode or request.asString

But now I want the inputstream and also I want the response code to know if the inputstream has anything useful or not.
Thanks!

This sample program will keep many connections in CLOSE_WAIT:

import scalaj.http.{HttpOptions, Http}

object Runner {

  def main(args: Array[String]) {
    val data =
      """
        |{
        |"test": "test"
        |}
      """.stripMargin

    for (i <- 0 to 10000) {
      println(s"Making request $i...")

      val request = Http
        .postData("http://google.com", data)
        .header("Content-type", "application/json")
        .header("Connection", "close")

      request.responseCode
    }

  }

}

This may not fit the design of the library but I need to send a POST that has both query parameters and a body. I couldn't figure out how to accomplish this. I have managed query parameters on a POST but as soon as I add a body the query parameters disappear.

wanting something like this http://www.bla.com/url/path?param1=something&param2=somethingelse
with content-type=application/json in the body.

Thanks

SSL handshake exception always pops up while trying to do the following. Any idea would be appreciated.
val timeout = 5000
val path = "some curl-able path"
// Have set proper java opts etc
Http(path).option(HttpOptions.connTimeout(timeout)).option(HttpOptions.readTimeout(timeout)).responseCode

Wonder if I am missing anything?

I use scalaj-http in my functional tests to test a post to a service that I'm writing. I post the request, no problem. Then I check the status code by doing a request.responseCode. I then want to pick apart the response for an id returned in the response. I do a request.asString, only to get a second post to the service, which is an unwanted side effect.

As a result my test is failing due to the duplicate post. Am I doing something wrong when interacting with the scalaj-http api?

Please!

How can I replicate the effect of the following (apache http-core) code with scalaj-http?

        HttpPost httppost = new HttpPost(uri);
        HttpEntity entity = new StringEntity("my content", "utf-8");;
        httppost.setEntity(entity);

http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html
CTL-F entities

We're trying to move from dispatch to scalaj, and during a file upload against spray (and, later against play) we run into this issue:

java.io.IOException: insufficient data written
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.close(HttpURLConnection.java:3214)
at scalaj.http.Http$$anonfun$3.apply(Http.scala:399)
at scalaj.http.Http$$anonfun$3.apply(Http.scala:312)
at scalaj.http.Http$Request.process(Http.scala:159)
at scalaj.http.Http$Request.apply(Http.scala:141)
at scalaj.http.Http$Request.asString(Http.scala:190)
at rest.DocumentProcessorClient$$anonfun$1.apply(DocProcClient.scala:27)
at rest.DocumentProcessorClient$$anonfun$1.apply(DocProcClient.scala:21)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

This is consistent across Mac and PC, and both using a stream or a byte array. We're running scala 2.10.3, scalaj 0.3.14, and java 1.7.0_51. Our code looks like this:

val responseFuture = Future {
log.debug("Uploading file " + fileName)
val r = Http.multipart(putUrl, MultiPart(fileName, fileName, "application/none", byteArray))
.option(HttpOptions.connTimeout(10000))
.option(HttpOptions.readTimeout(60000))
(r.responseCode, r.asString)
}

However, note, we've also tried this outside of a future, same result.

On the server side, we see a java.nio.channels.ClosedChannelException when we try to send an Ok back, which could be related?

Any advice would be much appreciated!

These are currently set at:

Connection timeout = 0.1 seconds
Read timeout = 0.5 seconds

Surely these are unrealistically low?

Furthermore, your README says your defaults are set at 30s (connection timeout) and 60s (read timeout).

Hi, thanks for this lib!

This issue is related to a previous one: #38.
I explained then that I couldn't uplaod files of >2Go using multipart. That was fixed but there's still a problem:
the Long (length written) returned and handled by writeCallBack: Long => Unit is negative and decreasing after 2Go (2147483648 as Long) has been reached.
It should still be growing until reaching the file length.

I upgraded to version 1.1.4.

Hi guy.
how to implement this below:
GET /my_index/my_type/_search
{
"query": {
"match": {
"title": "QUICK!"
}
}
}

scala> val html = Http("http://www.questionablecontent.net").asString
scalaj.http.HttpException: 403: Forbidden
at scalaj.http.Http$Request.process(Http.scala:125)
at scalaj.http.Http$Request.apply(Http.scala:102)
at scalaj.http.Http$Request.asString(Http.scala:151)
at .(:11)
at .()
at .(:7)
at .()
at $print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
...

kypeli@iceberg:~$ curl -I http://www.questionablecontent.net
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Sun, 21 Jul 2013 11:40:50 GMT
Content-Type: text/html
Connection: keep-alive
X-Powered-By: PHP/5.4.13
Set-Cookie: __cfduid=dd4e453443f2d6171976787db3fdf55c81374406850; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.questionablecontent.net
CF-RAY: 9179e60760306dc

I tried to execute a POST request with 2 multiparts in Android. Something like that:

Http.multipart(url, Multipart(name, filename, mime, data string), Multipart(name, filename, mime, file input stream, chunk size, {
bytesWritten =>
publishProgress(bytesWritten.toFloat / fileLen.toFloat)
}).responseCode

I got a FixedLengthOutputStream exception. This is the log output:

Caused by: java.io.IOException: expected 30 bytes but received 4096
 15399         AndroidRuntime  E    at org.apache.harmony.luni.internal.net.www.protocol.http.FixedLengthOutputStream.write(FixedLengthOutputStream.java:38)
 15399         AndroidRuntime  E    at scalaj.http.Http$$anonfun$3$$anonfun$apply$8.readOnce$3(Http.scala:381)
 15399         AndroidRuntime  E    at scalaj.http.Http$$anonfun$3$$anonfun$apply$8.apply(Http.scala:391)
 15399         AndroidRuntime  E    at scalaj.http.Http$$anonfun$3$$anonfun$apply$8.apply(Http.scala:367)
 15399         AndroidRuntime  E    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
 15399         AndroidRuntime  E    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
 15399         AndroidRuntime  E    at scalaj.http.Http$$anonfun$3.apply(Http.scala:367)
 15399         AndroidRuntime  E    at scalaj.http.Http$$anonfun$3.apply(Http.scala:312)
 15399         AndroidRuntime  E    at scalaj.http.Http$Request.process(Http.scala:159)
 15399         AndroidRuntime  E    at scalaj.http.Http$Request.responseCode(Http.scala:178)

Perhaps this link could be the solution for this issue: http://answer.techwikihow.com/977999/upload-files-post-server-outofmemory.html

Hi

I have a project seeded from here: https://github.com/spray/spray-template

I have added this line to my librariesDependencies: "org.scalaj" %% "scalaj-http" % "1.1.4" and now I get a runtime error Exception in thread "main" java.lang.ClassNotFoundException: com.example.Boot

This is the Boot object:

object Boot extends App {
...
}
where App is scala.App

As soon as I remove the line from libraries dependencies, everything runs fine again. I don't even have to add a call to something from scalaj, it just immediately breaks when I add this as a dependency.

Other dependencies I have are:
"io.spray" %% "spray-can" % sprayV,
"io.spray" %% "spray-routing" % sprayV,
"io.spray" %% "spray-testkit" % sprayV % "test",
"io.spray" %% "spray-json" % "1.3.1",
"com.typesafe.akka" %% "akka-actor" % akkaV,
"com.typesafe.akka" %% "akka-testkit" % akkaV % "test",
"org.specs2" %% "specs2-core" % "2.3.11" % "test",
"com.typesafe.slick" %% "slick" % "2.1.0",
"com.typesafe.slick" %% "slick-extensions" % "2.1.0",
"c3p0" % "c3p0" % "0.9.1.2",
"org.slf4j" % "slf4j-api" % "1.7.10",
"org.slf4j" % "slf4j-simple" % "1.7.10"

I was trying to run the README examples and in several I've received exceptions. For instance, testing:

val request: Http.Request = Http("http://date.jsontest.com/")

val resultOne = request.asString
val resultTwo = request.asString

throws an: java.lang.NoSuchMethodError: scala.collection.mutable.StringBuilder.append([CII)Lscala/collection/mutable/StringBuilder;

Am I missing some configuration or this project is no longer updated?

Headers could contains many values for same key (see Set-Cookie - http://en.wikipedia.org/wiki/Http_cookie)

As s suggestion here is a dummy impl:

def getResponseHeaders(conn: HttpURLConnection): Map[String, List[String]] = {
  // according to javadoc, there can be a headerField value where the HeaderFieldKey is null
  // at the 0th row in some implementations.  In that case it's the http status line
  val map = Stream.from(0).map(i => i -> conn.getHeaderField(i)).takeWhile(_._2 != null).map{ case (i, value) =>
    Option(conn.getHeaderFieldKey(i)).getOrElse("Status") -> value
  }.toList.groupBy(_._1)
  for ( (k,v) <- map ) yield  k -> v.map(_._2)
}

Because the default proxy option is Proxy.NO_PROXY, there's no easy way to specify "use the proxy that http.proxyHost, http.proxyPort, and http.nonProxyHosts (and the https equivalents) point to".

A straightforward fix for this should be to default to using the Java default proxy - I'll have a PR up shortly.

[info] Resolving commons-codec#commons-codec;1.5 ...                                                
[warn]  [NOT FOUND  ] commons-codec#commons-codec;1.5!commons-codec.src (1003ms)                    
[warn] ==== public: tried                                                                           
[warn]   http://repo1.maven.org/maven2/commons-codec/commons-codec/1.5/commons-codec-1.5-sources.src
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::                                              
[warn]  ::              FAILED DOWNLOADS            ::                                              
[warn]  :: ^ see resolution messages for details  ^ ::                                              
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::                                              
[warn]  :: commons-codec#commons-codec;1.5!commons-codec.src                                        
[warn]  :::::::::::::::::::::::::::::::::::::::::::::: 

0.3.3 and 0.3.4 pom.xml included

<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.5</version>
<classifier>sources</classifier>
<type>src</type>
</dependency>

and

<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
<classifier>sources</classifier>
<type>src</type>
</dependency>

http://repo1.maven.org/maven2/org/scalaj/scalaj-http_2.9.2/0.3.4/scalaj-http_2.9.2-0.3.4.pom
http://repo1.maven.org/maven2/org/scalaj/scalaj-http_2.9.2/0.3.3/scalaj-http_2.9.2-0.3.3.pom

I think these dependencies should not be include.

Inside documentation I can find this:
Http.multipart(url, MultiPart("photo","headshot.png", "image/png", fileBytes)).responseCode

But misses the fact that some devices like android cannot handle large files.

Could it be refactored so that the fourth argument is not a byte array but a file?

And of course this file must be handled streamfully with chuncks of bytes.
A simple statement that simply reads a file into a byte array will throw an OutOfMemoryException

I get a very weird nullpointer exception which happens when I append a spaces to parameters in querystrings but only on certain URLs. I can't find a pattern or cause but I can reproduce it by doing this:

// OK: Works
println(Http("https://www.google.com/?a=b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)

// OK: 404 HttpException
println(Http("https://www.google.com/?a= b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)
println(Http("https://api.instagram.com/v1/users/12895238/?a=b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)
println(Http("https://api.instagram.com/v1/users/ 12895238/?a=b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)
println(Http("https://api.instagram.com/v1/users/12895238/?a=b ").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)

// NOT OK: NullPointerException
println(Http("https://api.instagram.com/v1/users/12895238/?a =b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)
println(Http("https://api.instagram.com/v1/users/12895238/? a=b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)

It might be related to Issue #8 but it seems to happen in another part of the code:

Exception in thread "main" java.lang.NullPointerException
    at scalaj.http.Http$.tryParse(Http.scala:155)
    at scalaj.http.Http$Request.liftedTree1$1(Http.scala:107)
    at scalaj.http.Http$Request.process(Http.scala:103)
    at scalaj.http.Http$Request.apply(Http.scala:90)
    at scalaj.http.Http$Request.asString(Http.scala:133)

Does or can Scalaj support Basic Auth header? Is there a way to do it?

The new multipart introduced in version 0.3.9 where you can upload large files using input stream would be nice to have a couple of extra features.
1)I must be able to set the buffer/chunck size
2)If there was a callback function on each buffer/chunck uploaded someone could fill a progress bar properly or do some other similar action. Thank you

Hi there! After a long time I have decided to upgrade to the latest (1.1.4) version from 0.3.x

On the old version of scalaj I used to have a line that looked like that

case e: HttpException => MyCustomErrorResultCaseClass(e.body, e.message)

Is HttpException used at all in the latest version? I only see it referenced once in the source code

It'd be very helpful to see what's changed from release to release, without getting bogged down in the git commit log. The commit log is helpful, but it doesn't really let one know about the severity of bug fixes, breaking API changes, etc. I realize it's a bit more work to maintain, but it'd be much appreciated.

The tag list stops at version 0.2.5, but 0.2.8 seems to be the latest release. I had to dig through the maven repository to work this out because installation failed when using 0.2.5 with scala 2.9.0.1.

When i make an http request(get) and the response code is not in 200-399 range, i get an exception. Wouldnt it be better if you returned the a pair with the http response code and response body or an object with response code and response body as members. This can be useful for debugging error responses

Hi;
I'm using scalaj-http Multipart to upload files with a form to a server. But I currently can't send files larger than 2Go for the field "numBytes" is an Int.
I fixed it by (forking, then) changing it into a Long in Http.scala:

case class MultiPart(val name: String, val filename: String, val mime: String, val data: InputStream, val numBytes: Long,
val writeCallBack: Long => Unit)

and precising further:

val totalBytesToSend: Long = {
...
0L + paramsLength + filesLength + finaleBoundaryLength
}

Could it be stable in scalaj-http ? should be an improvement. Thanks!

Hi

I am trying to do a POST request and some headers are added by default by Http's implementation (I assume by the underlying Java implementation). The Accept header is making my endpoint reject my POST request with a 406.

Below my code:

val result = Http(eventStoreUrl)
  .proxy("127.0.0.1", 8888)
  .header("Content-Type", "application/json")
  .header("ES-EventType", "UserSearchedHotelAvailability")
  .header("ES-EventId", UUID.randomUUID().toString)
  .postData(payload)
  .asString

and here a trace of the actual post request from Fiddler:

POST myUrl HTTP/1.1
Accept-Encoding: gzip,deflate
ES-EventId: 6bed603d-e6aa-4d9f-9e04-3b5e19f372d1
ES-EventType: UserSearchedHotelAvailability
Content-Type: application/json
User-Agent: scalaj-http/1.0
Host: 127.0.0.1:2113
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 10

{'Type':1}

I could not find a way to remove the Accept header from there. I tried setting it to null or empty string, but that meant it's just added twice, first with my value, then with this default one.

This is a pretty unusual use case, so I'll understand if this is something not considered worth fixing.

I'm using an HTTP connection to connect to a bittorrent tracker, which returns mostly ASCII text, with some raw binary data. Most characters above 0x80 get truncated to 0x3F when I use Http(foo).asString.

To get around this, I'm using java.net.URL instead of scalaj-http, and avoiding any conversions that assume some charset. If there's any interest, I can submit a pull request if I can find some spare time, but I don't know that anyone actually cares about pulling raw binary data over HTTP

readString does not seems to works as expected with 2.10.0
I suggest using bos.appendAll instead of bos.append

def readString(is: InputStream): String = {
val in = new InputStreamReader(is, charset)
val bos = new StringBuilder
val ba = new ArrayChar

def readOnce {
  val len = in.read(ba)
  if (len > 0) bos.appendAll(ba, 0, len)
  if (len >= 0) readOnce
}

readOnce

bos.toString()

}