Comments (8)
You probably need to set the Accepts
header to prioritise html over json.
from curl.
That unfortunately did not work:
GET("http://www.camera.it/leg17/29?tipoAttivita=&tipoVisAtt=&tipoPersona=&shadow_deputato=300453&idLegislatura=17", accept("text/html"))
still returns the "JSON" version of the page.
from curl.
They are running a misconfiged cache server so you are getting false hits. Try this:
req <- GET("http://www.senato.it/loc/link.asp?tipodoc=CAM.DEP&leg=17&id=30350",
add_headers(Accept = "text/html", "Cache-control" = "max-age=0"))
content(req, "text")
Sometimes it helps if you just add an arbitrary parameter to the URL to bypass the cache:
url <- paste0("http://www.senato.it/loc/link.asp?tipodoc=CAM.DEP&leg=17&id=30350&_random=", runif(1))
req <- GET(url, accept("text/html"))
content(req, "text")
from curl.
Still no luck: I get a false hit, whatever the URL used.
On top of that, I've just discovered that curl -v
always returns HTML, but the content of the page is often faulty ("page temporarily inaccessible, return later").
from curl.
I think this works:
library(httr)
url <- "http://www.camera.it/leg17/29?tipoAttivita=&tipoVisAtt=&tipoPersona=&shadow_deputato=300453&idLegislatura=17"
url <- paste0(url, "&_random=", rnorm(1))
req <- GET(url, accept("text/html"))
stopifnot(req$headers[["x-cache"]] == "MISS")
stopifnot(req$headers$age == "0")
content(req, "text")
Their server is really poorly configured, not only does it give false hits but it ignores the Cache-Control: no-cache
request header. But slightly changing the URL usually forces the cache server to fetch a new copy.
from curl.
It seems wot work indeed!
Thank you very much to both of you for your help.
How did you come with the &random=
part?
from curl.
It's just something arbitrary that you add to the URL in order to trick the cache server into thinking that you are fetching a different page, so it cannot serve you a cached copy. It's a common trick to force bypassing any cache.
from curl.
Excellent. Thanks again and enjoy your days.
from curl.
Related Issues (20)
- FTP directory created successfuly, but curl returns with error HOT 3
- Feature request: Use environment variable `CURL_CA_BUNDLE` if set HOT 14
- inconsistent SSL version in vignette
- Allow users to pass pool object in `multi_download()`
- Use httpuv::randomPort() in curl_echo
- Why is curl_echo() slow sometimes HOT 2
- How to resolve "SSL: no alternative certificate subject name matches target host name '<proxy.server.address>'"
- httpuv delays in curl_echo on Linux HOT 1
- Can't get all bytes received before RST HOT 1
- installation warning: skipping pax global extended headers HOT 1
- Sudden negotiate error with 5.2.1
- Xcode Run on ios with Undefined symbol error HOT 1
- Support for request rate limiting? HOT 1
- stubbed version for (non)use in webr installs requiring curl HOT 3
- curl failure with Apple M Series and macOS Sonoma 14.5 HOT 12
- Multipart element from raw vector but with filename
- Windows ARM64 source build - fails if parallel make jobs used (winlibs download race condition) HOT 1
- Error in curl::curl_fetch_memory(url, handle = handle) : Error in curl::curl_fetch_memory(url, handle = handle) HOT 4
- Delete a file from my SFTP server using a custom request? HOT 2
- An unknown option was passed in to libcurl HOT 11
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from curl.