Comments (6)
There is now experimental support for async requests in the dev version of curl:
install.packages("https://github.com/jeroenooms/curl/archive/master.tar.gz", repos = NULL)
library(curl)
?multi
Let me know if you have any feedback.
Here is an example: https://github.com/jeroenooms/curl/blob/master/examples/crawler.R
from curl.
Wahoo!!!
from curl.
curl 2.0 which includes the async stuff should be on CRAN this week.
from curl.
And a follow up:
Here's an example from Rcurl:
getURIs =
function(uris, ..., multiHandle = getCurlMultiHandle(), .perform = TRUE)
{
content = list()
curls = list()
for(i in uris) {
curl = getCurlHandle()
content[[i]] = basicTextGatherer()
opts = curlOptions(URL = i, writefunction = content[[i]]$update, ...)
curlSetOpt(.opts = opts, curl = curl)
multiHandle = push(multiHandle, curl)
}
if(.perform) {
complete(multiHandle)
lapply(content, function(x) x$value())
} else {
return(list(multiHandle = multiHandle, content = content))
}
}
There is also getURIAsynchronous
Here's an example use case where this would be helpful. I want to submit 1,000 requests to the server, and each request takes 10 minutes to process (the server has to lookup some data and do some math that takes a long time). However, the sever can handle many thousands of simultaneous requests.
Currently, I'm looping through something like this:
requests <- lapply(urls, POST, ...)
Which blocks on each request and takes 1,000 x 10 minutes to complete. It'd be really nice to be able to send each request off to the server, without blocking on the request being completed. Then after they have all been submitted, we can block on collecting the results with a loop like this:
requests <- lapply(urls, POST, ..., async=TRUE)
results <- lapply(results, httr::complete)
Where httr::complete
would be similar to RCurl::complete
. The second example is nice because we can submit all the requests at once and let the server start processing them before blocking on gathering the results. In theory, this loop would take ~10 minutes to complete, plus the overhead of the 2 lapply
loops.
from curl.
I think the natural R solution might be a non-blocking connection object. Although R is limited to 128 connections which is kind of lame.
from curl.
Would that be a feature request for R core? If so, how would I submit it to R core?
from curl.
Related Issues (20)
- 'Remote file not found' if SMB path has Unicode characters HOT 6
- Error in curl::handle_setopt(handle, .list = req$options): Option seekfunction (20167) has unknown or unsupported type. HOT 2
- form_data documentation is lacking
- parse_headers throws an error when a site responds with an empty header HOT 2
- Out of memory errors for libcurl ≥ 7.71.0 HOT 1
- FTP directory created successfuly, but curl returns with error HOT 3
- Feature request: Use environment variable `CURL_CA_BUNDLE` if set HOT 14
- inconsistent SSL version in vignette
- Allow users to pass pool object in `multi_download()`
- Use httpuv::randomPort() in curl_echo
- Why is curl_echo() slow sometimes HOT 2
- How to resolve "SSL: no alternative certificate subject name matches target host name '<proxy.server.address>'"
- httpuv delays in curl_echo on Linux HOT 1
- Can't get all bytes received before RST HOT 1
- installation warning: skipping pax global extended headers HOT 1
- Sudden negotiate error with 5.2.1
- Xcode Run on ios with Undefined symbol error HOT 1
- Support for request rate limiting? HOT 1
- stubbed version for (non)use in webr installs requiring curl HOT 3
- curl failure with Apple M Series and macOS Sonoma 14.5 HOT 12
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from curl.