Possible to implement asynchronous (i.e. non-blocking io) in curl? about curl

[jeroen]

There is now experimental support for async requests in the dev version of curl:

install.packages("https://github.com/jeroenooms/curl/archive/master.tar.gz", repos = NULL)
library(curl)
?multi

Let me know if you have any feedback.

Here is an example: https://github.com/jeroenooms/curl/blob/master/examples/crawler.R

[zachmayer]

Wahoo!!!

[jeroen]

curl 2.0 which includes the async stuff should be on CRAN this week.

[zachmayer]

And a follow up:
Here's an example from Rcurl:

getURIs =
function(uris, ..., multiHandle = getCurlMultiHandle(), .perform = TRUE)
{
  content = list()
  curls = list()

  for(i in uris) {
    curl = getCurlHandle()
    content[[i]] = basicTextGatherer()
    opts = curlOptions(URL = i, writefunction = content[[i]]$update, ...)    
    curlSetOpt(.opts = opts, curl = curl)
    multiHandle = push(multiHandle, curl)
  }

  if(.perform) {
     complete(multiHandle)
     lapply(content, function(x) x$value())
   } else {
     return(list(multiHandle = multiHandle, content = content))
   }
}

There is also getURIAsynchronous

Here's an example use case where this would be helpful. I want to submit 1,000 requests to the server, and each request takes 10 minutes to process (the server has to lookup some data and do some math that takes a long time). However, the sever can handle many thousands of simultaneous requests.

Currently, I'm looping through something like this:

requests <- lapply(urls, POST, ...)

Which blocks on each request and takes 1,000 x 10 minutes to complete. It'd be really nice to be able to send each request off to the server, without blocking on the request being completed. Then after they have all been submitted, we can block on collecting the results with a loop like this:

requests <- lapply(urls, POST, ..., async=TRUE)
results  <- lapply(results, httr::complete)

Where httr::complete would be similar to RCurl::complete. The second example is nice because we can submit all the requests at once and let the server start processing them before blocking on gathering the results. In theory, this loop would take ~10 minutes to complete, plus the overhead of the 2 lapply loops.

[jeroen]

I think the natural R solution might be a non-blocking connection object. Although R is limited to 128 connections which is kind of lame.

[zachmayer]

Would that be a feature request for R core? If so, how would I submit it to R core?