haskell / wreq Goto Github PK

With the following code server with ONLY receive contents encoded in gzip. But servers like SOLR require client to explicitly set content type to application/json.

data SolrAdd docType = SolrAdd
    { doc          :: docType
    , boost        :: Float
    , overwrite    :: Bool
    , commitWithin :: Integer
    }

instance ToJSON docType => ToJSON (SolrAdd docType) where
    toJSON (SolrAdd doc boost overwrite commitWithin) =
        object
            [
                "add" .= object
                [ "doc" .= toJSON doc
                , "boost" .= boost
                , "overwrite" .= overwrite
                , "commitWithin" .= commitWithin
                ]
            ]

indexThread thread = do
    let opts    = defaults & header "Content-Type" .~ ["application/json"]
        solrAdd = SolrAdd thread 1.0 True 10000
    postWith opts "http://localhost:8983/solr/collection1/update?wt=json" $ encode solrAdd

Hi:

Is there a way to get Wreq to compile on FPComplete. I have tried adding it to the extra packages, but seem to be caught in dependency hell (FPComplete seems to provide incompatible versions of http-client and lens library, and no PSQueue library).

Any way to get this to build on the FPComplete infrastructure?

Would it make sense for asJSON to not only accept application/json as content type but also allow +json suffixed types like e.g. application/sparql-results+json?

I mistakenly used the tutorial as an example to how I should use Wreq.

This was a very bad idea.

The http-client library that wreq is built on top of makes clear note that you should always use the manager, the equivalent in wreq is to use the session. Implications from not using wreq with the session will not be good for anyone doing merely a few thousand HTTP calls over the course of seconds.

I built a fairly simple tester tool for one of our projects, that happens to want a PUT request, merely calling this once every 20 seconds by 1000 threads quickly resulted in memory going from 40MB -> 1.9 GB. Sometimes the GC would kick in fast enough to bring it back down to ~ 250 MB, but usually it stayed well past 1.4 GB.

This was obviously very disappointing, since merely holding open a thousand websocket connections and sending basic pings was taking a mere 40 MB of memory. So I considered using http-client directly, when I noted the multiple warnings that http-client should always be used with a manager to ensure keep-alive re-use and connection pooling. Maybe I should use Network.Wreq.Session I thought, so I switched the code to utilize it.

Upon switching, the memory usage is no longer jumping past 1 GB. In fact its a mere 30 MB past the base 40MB my program had doing no HTTP calls, totally acceptable.

It's probably fine if the tutorial for simplicity sake doesn't show the session, but as-is, it totally fails to mention the session, what it brings with it, and why its absolutely critical that real-world usage of wreq should always use a session (the http-client stresses that this is thread-safe and should be used by all threads).

Posting to https://bucket.s3.amazonaws.com/object-name doesn't seem to work with Amazon returning an error:

The provided 'x-amz-content-sha256' header must be a valid SHA256.

When I switch to https://s3.amazonaws.com/bucket/object-name it works correctly.

I suspect this is because serviceAndRegion in Network/Wreq/Internal/AWS.hs doesn't recognize this newer style of S3 service and so is sending a x-amz-content-sha256 header, though I don't know why it would be bad to send one.

Hi @bos,

I would love to use wreq in a tool of mine which is part of stackage.
Are you interested into having wreq in stackage and being listed as maintainer?

(You basically just have to do a PR on that file: https://github.com/fpco/stackage/blob/master/build-constraints.yaml and add yourself with the package)

If not I can add it next to my packages for now.

Please let me know

Thanks!

I'm trying out the Whirlwind tour in GHCi 7.8.3 (stackage:http://www.stackage.org/stackage/533cb9cb992d7d34d12c53427b7463564d246adb)

First of all, you don't mention the need to add :set -XOverloadedStrings, without that simple things fail with complex error messages

Prelude Network.Wreq Control.Lens> let opts = defaults & param "q" .~ ["tetris"] & param "language" .~ ["haskell"]

<interactive>:4:29:
    Couldn't match expected type ‘Data.Text.Internal.Text’
                with actual type ‘[Char]’
    In the first argument of ‘param’, namely ‘"q"’
    In the first argument of ‘(.~)’, namely ‘param "q"’
    In the second argument of ‘(&)’, namely ‘param "q" .~ ["tetris"]’

<interactive>:4:37:
    Couldn't match expected type ‘Data.Text.Internal.Text’
                with actual type ‘[Char]’
    In the expression: "tetris"
    In the second argument of ‘(.~)’, namely ‘["tetris"]’
    In the second argument of ‘(&)’, namely ‘param "q" .~ ["tetris"]’

<interactive>:4:55:
    Couldn't match expected type ‘Data.Text.Internal.Text’
                with actual type ‘[Char]’
    In the first argument of ‘param’, namely ‘"language"’
    In the first argument of ‘(.~)’, namely ‘param "language"’
    In the second argument of ‘(&)’, namely
      ‘param "language" .~ ["haskell"]’

<interactive>:4:70:
    Couldn't match expected type ‘Data.Text.Internal.Text’
                with actual type ‘[Char]’
    In the expression: "haskell"
    In the second argument of ‘(.~)’, namely ‘["haskell"]’
    In the second argument of ‘(&)’, namely
      ‘param "language" .~ ["haskell"]’

Secondly, Github seems to have changed their API to require authentication or more specific search queries:

Prelude Network.Wreq Control.Lens> let opts = defaults & param "q" .~ ["tetris"] & param "language" .~ ["haskell"]
Prelude Network.Wreq Control.Lens> r <- getWith opts "https://api.github.com/search/code"
*** Exception: StatusCodeException (Status {statusCode = 422, statusMessage = "Unprocessable Entity"}) [("Server","GitHub.com"),("Date","Sat, 08 Nov 2014 22:09:56 GMT"),("Content-Type","application/json; charset=utf-8"),("Status","422 Unprocessable Entity"),("X-RateLimit-Limit","10"),("X-RateLimit-Remaining","9"),("X-RateLimit-Reset","1415484656"),("Cache-Control","no-cache"),("X-GitHub-Media-Type","github.v3; format=json"),("X-XSS-Protection","1; mode=block"),("X-Frame-Options","deny"),("Content-Security-Policy","default-src 'none'"),("Content-Length","239"),("Access-Control-Allow-Credentials","true"),("Access-Control-Expose-Headers","ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval"),("Access-Control-Allow-Origin","*"),("X-GitHub-Request-Id","8E966590:5F9A:7F35E5B:545E94B4"),("Strict-Transport-Security","max-age=31536000; includeSubdomains; preload"),("X-Content-Type-Options","nosniff"),("X-Response-Body-Start","{\"message\":\"Validation Failed\",\"documentation_url\":\"https://developer.github.com/v3/search/#search-code\",\"errors\":[{\"message\":\"Must include at least one user, organization, or repository\",\"resource\":\"Search\",\"field\":\"q\",\"code\":\"invalid\"}]}"),("X-Request-URL","GET https://api.github.com:443/search/code?language=haskell&q=tetris")] (CJ {expose = []})

Otherwise, seems like an amazing library - great that powerful combinators, lens etc can make it so easy to do things like accessing arbitrary web APIs, something I'd initially think would be very awkward in Haskell, compared to Python for example.

Installed using cabal install -j --disable-tests wreq inside a new clean cabal sandbox.

[1 of 1] Compiling Main             ( /tmp/wreq-0.2.0.0-5646/wreq-0.2.0.0/dist/dist-sandbox-ea84c63/setup/setup.hs, /tmp/wreq-0.2.0.0-5646/wreq-0.2.0.0/dist/dist-sandbox-ea84c63/setup/Main.o )
Linking /tmp/wreq-0.2.0.0-5646/wreq-0.2.0.0/dist/dist-sandbox-ea84c63/setup/setup ...
Configuring wreq-0.2.0.0...
Building wreq-0.2.0.0...
Preprocessing library wreq-0.2.0.0...
[ 1 of 11] Compiling Network.Wreq.Lens.Machinery ( Network/Wreq/Lens/Machinery.hs, dist/dist-sandbox-ea84c63/build/Network/Wreq/Lens/Machinery.o )

Network/Wreq/Lens/Machinery.hs:20:58:
    Couldn't match type ‘[Name]’ with ‘Name’
    Expected type: Name -> [Name] -> Name -> [DefName]
      Actual type: [Name] -> Name -> [DefName]
    Possible cause: ‘fieldName’ is applied to too many arguments
    In the second argument of ‘(.~)’, namely ‘fieldName id’
    In the second argument of ‘(&)’, namely ‘lensField .~ fieldName id’
cabal: Error: some packages failed to install:
wreq-0.2.0.0 failed during the building phase. The exception was:
ExitFailure 1

It'd be quite nice if OAuth1 signing was supported so API's like twitter's could be used.

Could you switch to lens 4.2? I don't think there were any major changes from 4.1.
NEVERMIND: I somehow derped up and misread the requirements... 🤦

The Postable typeclass can be used for conveniently encoding a list of form params when doing a POST. E.g.

>>> r <- post "http://httpbin.org/post" ["num" := 31337, "str" := "foo"]

However, the same doesn't seem to be possible with PUT, as the Putable typeclass doesn't include the FormParam and [FormParam] instances. Thus the only easy way to pass a bunch of parameters to a PUT request seems to be to encode the parameters as JSON and send that.

As POST supports normal URL encoded parameters, shouldn't the PUT method support that too?

any idea why Network.Wreq spends 0.53s to load https://www.google.com while curl loads it in 0.046s? the test code is at http://lpaste.net/112299

I have to do some package unregistering to get wreq to build under Windows.

After some work, it and it's dependencies all build, but it fails to link:

AppData\Roaming\cabal\i386-windows-ghc-7.8.3\network-2.6.0.2\HSnetwork-2.6.0.2.o: unknown symbol `_shutdownWinSock'

While using aura, I ran into fosskers/aura#348. It seems that the error is coming from wreq, but I don't know that for sure.

wreq is reporting a FailedConnectionException when the problem is actually insufficient entropy in the system pool, which is rather difficult to debug. Could we get a more specific error message for that case?

On OSX 10.9.2, ghc 7.6.3, cabal 1.20.0.0. Doing the following:

mkdir wreqtest
cd wreqtest
cabal sandbox init
cabal install -j --disable-tests wreq

results in the error:

Network/Wreq/Internal/Link.hs:30:8:
Not in scope: peekChar'
Perhaps you meant `peekChar' (imported from Data.Attoparsec.ByteString.Char8)
cabal: Error: some packages failed to install:
wreq-0.1.0.0 failed during the building phase. The exception was:
ExitFailure 1

When I run the following code I get a response, but I expect a timeout:

{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.Client (ManagerSettings(..))
import qualified Network.HTTP.Client as HTTP
import qualified Network.Wreq.Session as Sess
import Control.Lens
import Network.Wreq

oneSecond :: Int
oneSecond = 1000000

httpGet mgr =
  Sess.getWith (defaults & manager .~ Right mgr)

main :: IO ()
main = do
  mgr <-
    HTTP.newManager
      (HTTP.defaultManagerSettings {
        managerResponseTimeout = Just (3 * oneSecond)
      })
  Sess.withAPISession (\session -> do
    resp <- httpGet mgr session "http://httpbin.org/delay/5"
    print resp)

in ghci:

*Main> main
Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}, responseVersion = HTTP/1.1, responseHeaders = [("Server","nginx"),("Date","Mon, 24 Aug 2015 22:23:12 GMT"),("Content-Type","application/json"),("Content-Length","253"),("Connection","keep-alive"),("Access-Control-Allow-Origin","*"),("Access-Control-Allow-Credentials","true")], responseBody = "{\n  \"args\": {}, \n  \"data\": \"\", \n  \"files\": {}, \n  \"form\": {}, \n  \"headers\": {\n    \"Accept-Encoding\": \"gzip\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"haskell wreq-0.4.0.0\"\n  }, \n  \"origin\": \"8.25.197.27\", \n  \"url\": \"http://httpbin.org/delay/5\"\n}\n", responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}

There's some performance and concurrency issues using wreq under heavy concurrent use. Several problems occur:

1. Creating a HTTP Client Manager is an expensive operation, the HTTP-Client docs recommend sharing one between requests (which shares the pools of TCP connections as well for efficient control of maximum TCP connections used)
2. Wreq's Session add's a CookieJar MVar, the use of this MVar means concurrent use of a wreq session does not make concurrent requests at once as the mvar remains locked for the duration of the request/response cycle
3. Wreq has no convenient way to create or use a session with an existing Manager (withSessionWith takes ManagerSettings, not an actual Manager, which means the expensive operation of making more Managers....)

I'm currently writing some of my own code to create a new session per thread, but using a shared Manager to confirm that this alleviates the concurrency issue and doesn't incur too much RAM overhead in the process. I can submit a PR with another helper function to use an existing Manager then.

{-# LANGUAGE OverloadedStrings #-}
-- {-# LANGUAGE ExtendedDefaultRules #-}
-- {-# OPTIONS_GHC -fno-warn-type-defaults #-}

import qualified Data.ByteString as BS
import Network.Wreq
import Control.Lens
-- import           System.Environment
-- import           Shelly
-- import qualified Data.Text as T
-- default (T.Text)

opts = defaults & header "SOAPAction" .~ ["urn:schemas-upnp-org:service:AVTransport:1#SetAVTransportURI"]
                & header "Content-Type" .~ ["text/xml; charset=\"utf-8\""]

setURI = do
  xml <- BS.readFile "/home/dontdieych/SetAVTransportURI.xml"
  postWith opts  "http://192.168.219.6:7676/smp_18_" xml

I'm trying to POST xml file. This is captured packet when calling setURI function in ghci.

POST /smp_18_ HTTP/1.1
Host: 192.168.219.6:7676
Accept-Encoding: gzip
Content-Length: 764
Content-Type: application/octet-stream
SOAPAction: urn:schemas-upnp-org:service:AVTransport:1#SetAVTransportURI
User-Agent: haskell wreq-0.1.0.1

<?xml version="1.0"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
  <s:Body>
    <u:SetAVTransportURI xmlns:u="urn:schemas-upnp-org:service:AVTransport:1">
      <InstanceID>0
      </InstanceID>
      <CurrentURI>
      </CurrentURI>
      <CurrentURIMetaData/>
    </u:SetAVTransportURI>
  </s:Body>
</s:Envelope>

<!--
Orignal header from captured packets.

POST /smp_18_ HTTP/1.1
Host: 192.168.219.6:7676
SOAPAction: "urn:schemas-upnp-org:service:AVTransport:1#SetAVTransportURI"
Accept-Language: en-us;q=1, en;q=0.5
Accept-Encoding: gzip
Content-Type: text/xml; charset="utf-8"
User-Agent: gupnp-av-cp GUPnP/0.20.12 DLNADOC/1.50
Connection: Keep-Alive
Content-Length: 1405
-->

If I set content-type text/xml, it is working.

dontdieych@ych ~> cat ~/_dd
POST /smp_18_ HTTP/1.1
Host: 192.168.219.6:7676
Accept-Encoding: gzip
Content-Length: 764
SOAPAction: "urn:schemas-upnp-org:service:AVTransport:2#SetAVTransportURI"
Content-Type: text/xml; charset="utf-8"
User-Agent: haskell wreq-0.1.0.1

<?xml version="1.0"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
  <s:Body>
    <u:SetAVTransportURI xmlns:u="urn:schemas-upnp-org:service:AVTransport:1">
      <InstanceID>0
      </InstanceID>
      <CurrentURI>http://192.168.219.2:8200/MediaItems/572.mp4
      </CurrentURI>
      <CurrentURIMetaData/>
    </u:SetAVTransportURI>
  </s:Body>
</s:Envelope>

<!--
Orignal header from captured packets.

POST /smp_18_ HTTP/1.1
Host: 192.168.219.6:7676
SOAPAction: "urn:schemas-upnp-org:service:AVTransport:1#SetAVTransportURI"
Accept-Language: en-us;q=1, en;q=0.5
Accept-Encoding: gzip
Content-Type: text/xml; charset="utf-8"
User-Agent: gupnp-av-cp GUPnP/0.20.12 DLNADOC/1.50
Connection: Keep-Alive
Content-Length: 1405
-->

dontdieych@ych ~> cat ~/_dd | nc 192.168.219.6 7676
HTTP/1.1 200 OK
Content-Length: 312
Content-Type: text/xml; charset="utf-8"
EXT:
SERVER: UPnP/1.0
Connection: close

<?xml version="1.0" encoding="utf-8"?><s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:SetAVTransportURIResponse xmlns:u="urn:schemas-upnp-org:service:AVTransport:1"></u:SetAVTransportURIResponse>
</s:Body>
</s:Envelope>⏎                                                                                                      dontdieych@ych ~>

How could I override "Content-Type" header value?

In many cases PUT is relatively identical in usage to POST. As such, it'd be handy if Putable honored the same instances as Postable including the setting of the Content-Type to a form-encoded when using [FormParam].

I had to copy/edit the Postable instances, which works fine but it feels like this should be in wreq:
https://github.com/bbangert/push-tester/blob/master/spTester/SimpleTest/Types.hs#L38

For some reason, providing library functions returning a Maybe seem very "unidiomatic lens". Since there already is an operator provided specifically for targeting a Maybe a field, do you have specific objections to the use of ?~? (See: basicAuth, oauth2Bearer, httpProxy, etc.)

In the second code block under the heading "Authentication", the tutorial has

ghci> let opts = defaults & auth ?~ basicAuth "user" "pass"

I think that should actually be

ghci> let opts = defaults & auth .~ basicAuth "user" "pass"

which also obviates the need for the explanation of (?~) after the code block.

(Looks like you maybe changed the type of basicAuth from one version to another.)

withSession doesn't seem to persist anything with this implementation:

withInitialState :: (ScraperState -> IO a) -> IO a
withInitialState callback = withSession $ \s -> do
  let initialState = PS { currentOptions = Wreq.defaults
                        , currentHtml = ("" :: LBS.ByteString)
                        , currentCursor = Nothing
                        , currentSession = s
                        }
  callback initialState

runScraper :: Scraper a -> IO a
runScraper k = withInitialState (evalScraperWith k)

evalScraperWith :: Scraper a -> ScraperState -> IO a
evalScraperWith k s =  ST.evalStateT k s

I looked up the source to withSession, since I don't get why this isn't working.

withSession :: (Session -> IO a) -> IO a
withSession act = do
  mv <- newMVar $ HTTP.createCookieJar []
  HTTP.withManager defaultManagerSettings $ \mgr ->
    act Session { seshCookies = mv, seshManager = mgr }

http://hackage.haskell.org/package/wreq-0.2.0.0/docs/src/Network-Wreq-Session.html#withSession

I did however notice an addition added a few days ago, maybe this is what I need?

withSessionWith :: HTTP.ManagerSettings -> (Session -> IO a) -> IO a
withSessionWith settings act = do
  mv <- newMVar $ HTTP.createCookieJar []
  HTTP.withManager settings $ \mgr ->
    act Session { seshCookies = mv
                , seshManager = mgr
                , seshRun = runWith
                }

runWith :: Session -> Run Body -> Run Body
runWith Session{..} act (Req _ req) =
  modifyMVar seshCookies $ \cj -> do
    resp <- act (Req (Right seshManager) (req & Lens.cookieJar ?~ cj))
    return (resp ^. Wreq.responseCookieJar, resp)

https://github.com/bos/wreq/blob/master/Network/Wreq/Session.hs#L42

For example, httpProxy :: ByteString -> Int -> Maybe Proxy, although it always returns Just something. Was there a specific design decision behind this?

Configuring wreq-0.3.0.1...
Building wreq-0.3.0.1...
Preprocessing library wreq-0.3.0.1...
[ 1 of 14] Compiling Network.Wreq.Lens.Machinery ( Network/Wreq/Lens/Machinery.hs, dist/build/Network/Wreq/Lens/Machinery.o )
[ 2 of 14] Compiling Paths_wreq ( dist/build/autogen/Paths_wreq.hs, dist/build/Paths_wreq.o )
[ 3 of 14] Compiling Network.Wreq.Cache.Store ( Network/Wreq/Cache/Store.hs, dist/build/Network/Wreq/Cache/Store.o )
[ 4 of 14] Compiling Network.Wreq.Internal.Types ( Network/Wreq/Internal/Types.hs, dist/build/Network/Wreq/Internal/Types.o )

Network/Wreq/Internal/Types.hs:47:1: Warning:
The import of ‘mconcat’ from module ‘Data.Monoid’ is redundant
[ 5 of 14] Compiling Network.Wreq.Internal.Lens ( Network/Wreq/Internal/Lens.hs, dist/build/Network/Wreq/Internal/Lens.o )
[ 6 of 14] Compiling Network.Wreq.Internal.AWS ( Network/Wreq/Internal/AWS.hs, dist/build/Network/Wreq/Internal/AWS.o )

Network/Wreq/Internal/AWS.hs:130:42:
Couldn't match expected type ‘time-1.5.0.1:Data.Time.Format.Locale.TimeLocale’
with actual type ‘System.Locale.TimeLocale’
NB: ‘time-1.5.0.1:Data.Time.Format.Locale.TimeLocale’
is defined in ‘Data.Time.Format.Locale’ in package ‘time-1.5.0.1’
‘System.Locale.TimeLocale’
is defined in ‘System.Locale’ in package ‘old-locale-1.0.0.7’
In the first argument of ‘formatTime’, namely ‘defaultTimeLocale’
In the first argument of ‘(.)’, namely
‘formatTime defaultTimeLocale "%Y%m%dT%H%M%SZ"’
cabal: Error: some packages failed to install:
wreq-0.3.0.1 failed during the building phase. The exception was:
ExitFailure 1

Hello,
I am trying to reuse a cookie from a previous response to modify a request but this does not seem to work:

    l <- post "http://localhost:8081/api/login" (toJSON login)
    let Just ck = l ^? responseCookie "authenticator"
    let opts = defaults & cookie "authenticator" .~ ck
    r <- getWith opts "http://localhost:8081/api/auth"

When I try to putStrLn the opts, I get:

Options { manager = Left _, proxy = Nothing, auth = Nothing, headers = [("User-Agent","haskell wreq-0.1.0.1")], params = [("foo","bar"),("foo","quux")], redirects = 10, cookies = [] }

What am I doing wrong?

Thanks

I am trying to test broken links but, when I get and run into a 404, I get an exception (see bottom) rather than a statusCode to handle. Only 200s seem to be returned.

I tried to follow the error-handling code in the tutorial but I can't find a way to return the same type as get u. Moreover, this seems to be more complexity than I need in this instance.

How can I simply prevent the exception and just return the responseStatus as is?

verifySeatme :: Maybe URL -> IO UrlStatus
verifySeatme url = do
    case url of
        Nothing -> return None
        Just "" -> return None
        Just u -> do
            seatmeResp <- get u --`E.catch` handler
            -- r ^? responseBody . key "url"
            -- could also check for redirect to errorPage.aspx
            if seatmeResp ^. W.responseStatus . statusCode == 200
            then return (Working u)
            else return Broken
    where
        handler e@(StatusCodeException s respHeaders _) =
            do
                return respHeaders

Here is the exception thrown, and you can see it has the statusCode I need

*Main CrawlerHTTP DbHandler Lib Data.Aeson Network.Wreq Control.Lens> re <- get "https://www.seatme.nl/restaurant/1371/Londen.htm"
*** Exception: StatusCodeException (Status {statusCode = 404, statusMessage = "Not Found"}) [("Cache-Control","private"),("Content-Type","text/html; charset=utf-8"),("Server","Microsoft-IIS/7.5"),("X-Powered-By","UrlRewriter.NET 2.0.0"),("p3p","CP=\"IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT\""),("X-AspNet-Version","4.0.30319"),("X-Powered-By","ASP.NET"),("Date","Tue, 15 Dec 2015 09:20:57 GMT"),("Content-Length","25576"),("X-Cache","MISS from seatme-lb01"),("X-Cache-Lookup","MISS from seatme-lb01:443"),("Via","1.0 seatme-lb01 (squid/3.1.10)"),("Connection","keep-alive"),("X-Response-Body-Start","\r\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"

It would be very useful to have a way to define own checkStatus as an Option for example to find out url where user agent has been redirected

In http://www.serpentine.com/wreq/tutorial.html

Both partFile and partFileSource will set the filename of a part to whatever name they are given (including any leading path components such as /tmp/confidential)

File path is stripped when rendering, only file name is transmitted over the wire: https://github.com/snoyberg/http-client/blob/master/http-client/Network/HTTP/Client/MultipartFormData.hs#L203

§ Changing default behaviours
< r <- getWith opts "http://httpbin.org"

r <- getWith opts "http://httpbin.org/get"

§ Uploading file contents
< r ^.. responseBody . key "files" . members . _String
... Not in scope: `members'

§ Authentication
< r <- get "http://httpbin.org/basic-auth/user/pass

r <- get "http://httpbin.org/basic-auth/user/pass"

I'm binding to an API where delete requests take a body with an object inside. I understand that this is controversial and APIs are discouraged from being written this way. That said, I didn't write the API, and I would like to bind to it.

I don't see a way to do so from wreq at the moment? Nor can I write my own deleteWithBody function modeled on postWith, since that seems to require using something from the Internal module...

So this could totally be a problem on my end, as I am fairly new to Haskell and could have somehow borked something, but I get a failure to install because mtl-2.0.1.0 could not be installed, which a lot of things seem to depend on. The output I get was this:

cabal: Error: some packages failed to install:
aeson-0.7.0.3 depends on mtl-2.0.1.0 which failed to install.
asn1-encoding-0.8.1.3 depends on mtl-2.0.1.0 which failed to install.
asn1-parse-0.8.1 depends on mtl-2.0.1.0 which failed to install.
connection-0.2.1 depends on mtl-2.0.1.0 which failed to install.
exceptions-0.6 depends on mtl-2.0.1.0 which failed to install.
free-4.7.1 depends on mtl-2.0.1.0 which failed to install.
http-client-0.3.2.2 depends on mtl-2.0.1.0 which failed to install.
http-client-tls-0.2.1.1 depends on mtl-2.0.1.0 which failed to install.
lens-4.1.2 depends on mtl-2.0.1.0 which failed to install.
mtl-2.0.1.0 failed during the building phase. The exception was:
ExitFailure 1
network-2.5.0.0 depends on mtl-2.0.1.0 which failed to install.
parsec-3.1.5 depends on mtl-2.0.1.0 which failed to install.
pem-0.2.2 depends on mtl-2.0.1.0 which failed to install.
socks-0.5.4 depends on mtl-2.0.1.0 which failed to install.
streaming-commons-0.1.2.3 depends on mtl-2.0.1.0 which failed to install.
tls-1.2.6 depends on mtl-2.0.1.0 which failed to install.
wreq-0.1.0.1 depends on mtl-2.0.1.0 which failed to install.
x509-1.4.11 depends on mtl-2.0.1.0 which failed to install.
x509-store-1.4.4 depends on mtl-2.0.1.0 which failed to install.
x509-system-1.4.5 depends on mtl-2.0.1.0 which failed to install.
x509-validation-1.5.0 depends on mtl-2.0.1.0 which failed to install.

Any ideas?
Thanks!

Seems like time dropped its dependency on old-locale in 1.5.

Network/Wreq/Internal/AWS.hs:130:42:
    Couldn't match expected type ‘time-1.5:Data.Time.Format.Locale.TimeLocale’
                with actual type ‘System.Locale.TimeLocale’
    NB: ‘time-1.5:Data.Time.Format.Locale.TimeLocale’
          is defined in ‘Data.Time.Format.Locale’ in package ‘time-1.5’
        ‘System.Locale.TimeLocale’
          is defined in ‘System.Locale’ in package ‘old-locale-1.0.0.6’
    In the first argument of ‘formatTime’, namely ‘defaultTimeLocale’
    In the first argument of ‘(.)’, namely
      ‘formatTime defaultTimeLocale "%Y%m%dT%H%M%SZ"’

I believe this is introduced by changes in lens, though I'm not certain:

[ 1 of 11] Compiling Network.Wreq.Lens.Machinery ( Network/Wreq/Lens/Machinery.hs, dist/build/Network/Wreq/Lens/Machinery.o )

Network/Wreq/Lens/Machinery.hs:20:58:
    Couldn't match type ‘[Name]’ with ‘Name’
    Expected type: Name -> [Name] -> Name -> [DefName]
      Actual type: [Name] -> Name -> [DefName]
    Possible cause: ‘fieldName’ is applied to too many arguments
    In the second argument of ‘(.~)’, namely ‘fieldName id’
    In the second argument of ‘(&)’, namely ‘lensField .~ fieldName id’
Failed to install wreq-0.2.0.0
cabal: Error: some packages failed to install:
wreq-0.2.0.0 failed during the building phase. The exception was:
ExitFailure 1

Hi:

I am trying to use Wreq to build a library for Iron.IO message queueing service. I seem to be having some performance issues (the code runs an order of magnitude slower than python). The issue is documented here: https://stackoverflow.com/questions/27696738/why-is-my-haskell-code-so-slow

Profiler output seems to reveal that the majority of time is spent in the decodeLenient (family) of functions. I believe these calls are coming from Wreq, or one of the libraries that Wreq uses (such as http-client or tls). I suspect the issue might be coming from the tls library, or some other library that is decoding Base64 data.

I am not sure how to troubleshoot this issue further...

cookie name is not a valid Traversal since it allows you to invalidate the predicate on the target. See filtered in the lens docs for further information.

Perhaps you don't care about this, but at the very least there should be a warning in the docs. I'm happy to submit a PR with such a warning, if you like.

When I try and run any wreq command in ghci on OS X I get an error:

λ> get "http://httpbin.org/get"
*** Exception: security: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)

Here's what I get from breaking on the exception and forcing the exception value:

λ> :set -fbreak-on-exception
λ> get "http://httpbin.org/get"
Stopped at <exception thrown>
_exception :: e = _
λ> :force _exception
_exception = GHC.Exception.SomeException
               (GHC.IO.Exception.IOError
                  Nothing GHC.IO.Exception.NoSuchThing
                  "createProcess: runInteractiveProcess: exec"
                  "No such file or directory"
                  (Just (Foreign.C.Types.CInt (GHC.Int.I32# 2))) Nothing)

I'm using GHC 7.8.3 and wreq 0.2.0.0. It works fine if I compile a similar program and run it.

In the documentation, this is mentioned

let opts = defaults & proxy .~ httpProxy "localhost" 8000
Which gives error
Prelude Network.Wreq Control.Lens Data.Aeson Data.Aeson.Lens> let opts = defaults & proxy .~ httpProxy "localhost" 8000

:43:32:
Couldn't match expected type ‘Maybe Proxy’ with actual type ‘Proxy’
In the second argument of ‘(.)’, namely
‘httpProxy "localhost" 8000’
In the second argument of ‘(&)’, namely
‘proxy . httpProxy "localhost" 8000’

https://hackage.haskell.org/package/wreq-0.1.0.1/docs/Network-Wreq.html

Since proxy is "MayBe" type it should be
let opts = defaults & proxy .~ Just (httpProxy "localhost" 8000)

How would I capture the response body for a URL that doesn't return a 2xx status, which is an HttpException?
Say I want to capture the HTML of a url that returns 404. HttpClient's HttpException's StatusCodeException does not seem to have any way of capturing the response body.

Thanks!

Copy of the build log for reference:

http://hydra.cryp.to/build/179094/log

Currently in LTS Haskell 1.10 there is no wreq found.

Hi:

I am trying to implement an API for the IronMQ message queueing service. Their delete method requires that the body of the DELETE request have some sort of JSON

http://dev.iron.io/mq/reference/api/#delete_a_message_from_a_queue

It does not look like Wreq allows delete and deleteWith methods to incorporate a body into the request. Is this a feature that can be added?

Discovered during a Stackage run

Test suite tests: RUNNING...
store:
  t_fromList: [OK, passed 100 tests]
  t_lookup: [OK, passed 100 tests]
  t_lookup1: [OK, passed 100 tests]
plain:
  httpbin:
    http:
      basic:
        get: [OK]
        post: [OK]
        head: [OK]
        put: [OK]
        delete: [OK]
        404: [OK]
        headRedirect: [OK]
        redirectOverflow: [OK]
        invalidURL: [OK]
        funkyScheme: [OK]
      fancy:
        basic auth: [OK]
        redirect: [OK]
        params: [OK]
        headers: [OK]
        gzip: [OK]
        json put: [OK]
        bytestring put: [OK]
        cookiesSet: [OK]
        getWithManager: [OK]
        cookieSession: [OK]
        getCheckStatus: [OK]
    https:
      basic:
        get: [OK]
        post: [OK]
        head: [OK]
        put: [OK]
        delete: [OK]
        404: [OK]
        headRedirect: [OK]
        redirectOverflow: [OK]
        invalidURL: [OK]
        funkyScheme: [OK]
      fancy:
        basic auth: [OK]
        redirect: [OK]
        params: [OK]
        headers: [OK]
        gzip: [OK]
        json put: [OK]
        bytestring put: [OK]
        cookiesSet: [OK]
        getWithManager: [OK]
        cookieSession: [OK]
        getCheckStatus: [OK]
session:
  httpbin:
    http:
      basic:
        get: [OK]
        post: [OK]
        head: [OK]
        put: [OK]
        delete: [OK]
        404: [OK]
        headRedirect: [Failed]
HEAD of redirect throws exception
        redirectOverflow: [OK]
        invalidURL: [OK]
        funkyScheme: [OK]
      fancy:
        basic auth: [OK]
        redirect: [OK]
        params: [OK]
        headers: [OK]
        gzip: [OK]
        json put: [OK]
        bytestring put: [OK]
        cookiesSet: [OK]
        getWithManager: [OK]
        cookieSession: [OK]
        getCheckStatus: [OK]
    https:
      basic:
        get: [OK]
        post: [OK]
        head: [OK]
        put: [OK]
        delete: [OK]
        404: [OK]
        headRedirect: [Failed]
HEAD of redirect throws exception
        redirectOverflow: [OK]
        invalidURL: [OK]
        funkyScheme: [OK]
      fancy:
        basic auth: [OK]
        redirect: [OK]
        params: [OK]
        headers: [OK]
        gzip: [OK]
        json put: [OK]
        bytestring put: [OK]
        cookiesSet: [OK]
        getWithManager: [OK]
        cookieSession: [OK]
        getCheckStatus: [OK]
localhost:
  plain:
    basic:
      get: [OK]
      post: [OK]
      head: [OK]
      put: [OK]
      delete: [OK]
      404: [OK]
      headRedirect: [OK]
      redirectOverflow: [OK]
      invalidURL: [OK]
      funkyScheme: [OK]
    fancy:
      basic auth: [OK]
      redirect: [OK]
      params: [OK]
      headers: [OK]
      gzip: [OK]
      json put: [OK]
      bytestring put: [OK]
      cookiesSet: [OK]
      getWithManager: [OK]
      cookieSession: [OK]
      getCheckStatus: [OK]
    oauth2 Bearer: [OK]
    oauth2 token: [OK]
  session:
    basic:
      get: [OK]
      post: [OK]
      head: [OK]
      put: [OK]
      delete: [OK]
      404: [OK]
      headRedirect: [Failed]
HEAD of redirect throws exception
      redirectOverflow: [OK]
      invalidURL: [OK]
      funkyScheme: [OK]
    fancy:
      basic auth: [OK]
      redirect: [OK]
      params: [OK]
      headers: [OK]
      gzip: [OK]
      json put: [OK]
      bytestring put: [OK]
      cookiesSet: [OK]
      getWithManager: [OK]
      cookieSession: [OK]
      getCheckStatus: [OK]
    oauth2 Bearer: [OK]
    oauth2 token: [OK]

         Properties  Test Cases    Total        
 Passed  3           127           130          
 Failed  0           3             3            
 Total   3           130           133          
Test suite tests: FAIL
Test suite logged to: dist/test/wreq-0.3.0.1-tests.log

On Windows, the tutorial is disheartening when the first example fails with

*** Exception: FailedConnectionException2 "httpbin.org" 80 False getAddrInfo: does not exist (error 10093)

A Google search turns up this solution which works. It would be very helpful if the tutorial mentioned it. Better yet, eliminate the need for it, but I know that's involved...

When trying out r <- get "http://httpbin.org/get" from the tutorial on a Debian system with no ca-certificates package installed (meaning no /etc/ssl/certs directory) the command exits with an Exception "*** Exception: /etc/ssl/certs/: getDirectoryContents: does not exist (No such file or directory)".

Seems we should handle this more gracefully (ie allow the user not to check certificates perhaps? in Python requests that verify=False I believe).

The other question is why was this triggered when the URL that's being accessed is http only?

Awesome work btw! 👍

I could not find in the documentation a way of simply ignoring bad certificates, but I would like to do so

Currently, the only way to use a Session is using the with-style functions in Network.Wreq.Session. A standalone newSession is not provided, which can be troublesome when one wants to use wreq to implement some web service's binding library and wants to embed the wreq's Session into the new library's Session and allow users to use Session in a natural way: first initiate one, then pass it around, no need to force everything in a big ugly do-block and pass to a withSession.

And although the documentation says "The Session will no longer be valid after that function returns.", this is actually not the case! The implementation of withSession tracks down to withManager in http-client, which is deprecated and doesn't do any clean-up work. It is tested that newSession = withSession pure works, the returned Session works as long as it's still in scope and not garbage collected.

I'd wreq to follow redirects, as I'm not interested into response body but the final URL. Currently wreq doesn't expose Network.HTTP.Client.HistoriedResponse in Response type.

Could we have a Postable instance for ()? I know it seems pretty trivial, but sometimes when I want to do a post request with no data, I feel like typing Text.empty feels cumbersome (and OverloadingStrings prevents "" from working as I would want).

Actually, now that I think about it a little more, I feel like having a data type like this:

data NoPostData = NoPostData

with the appropriate instance would be best. It makes the intent of the calling code more clear.

### Failure in Network/Wreq.hs:219: expression `r ^? responseBody . key "json"'
expected: Just (Array (fromList [Number 1.0,Number 2.0,Number 3.0]))
 but got: Just (Array [Number 1.0,Number 2.0,Number 3.0])

Cause the change in vector: haskell/vector@fa96a33