A group of S3 problems with known fixes about amazonka

[rjmac]

Hm, actually I'm not sure if S3 needs to be regional or if it needs to learn to associate a bucket with a region by following the redirects that are returned when the wrong (or no) region is sent. Individual buckets are regioned, but the top-level service itself is not.

[brendanhay]

Thanks @rjmac.

Since I chose (or was forced to by the models, due to generality) to use the region specific endpoints instead of virtual host style buckets, region/endpoint logic needs to be specialised for S3. This is because it uses a different endpoint structure based on the region you're communicating with.

The API reference outlines this in detail, and the service models contain the invariants which I will parse/use to ensure the correct endpoint is chosen depending on the region.

[brendanhay]

I'm going to merge the two PR's addressing this to develop, but leaving the issue open. If you could rerun your code and let me know if your problems have been solved, or add any additional info.

Alternatively if you could supply me with a small example I could use to quickly recreate any issues, it would be greatly appreciated!

[rjmac]

Doesn't quite; it looks like the line Endpoint host' scope = endpoint s r in V4.hs should be Endpoint scope host' = ... After changing that, there are a couple of incorrect timestamp formats (RFC822s that should be ISO8601s in the definitions of Bucket and Object in S3.Types).

Anyway, here's the program I've been using to experiment:

{-# LANGUAGE OverloadedStrings #-}

module Main where

import Network.AWS (Credentials(..), Region(..), getEnv, envLogger, Logger(Debug))
import Control.Monad.Trans.AWS (runAWST, send, paginate)
import Network.AWS.S3
import Network.AWS.Prelude
import Control.Lens
import qualified Data.Text.IO as Text
import qualified Data.Text as Text
import Control.Monad.IO.Class (liftIO)
import Data.Conduit
import System.Environment (lookupEnv)

main :: IO ()
main = do
  env <- getEnv Oregon Discover <&> envLogger .~ Debug Text.putStrLn

  -- Work with the bucket named in S3_BUCKET or whichever comes back first
  defaultBucket <- (fmap.fmap) Text.pack $ lookupEnv "S3_BUCKET"
  let bucketSelector = maybe head const defaultBucket

  -- The name of an object to retrieve
  objectToRetrieve <- (fmap.fmap) Text.pack $ lookupEnv "S3_OBJECT"

  er <- runAWST env $ do
    buckets <- send listBuckets <&> (^.lbrBuckets) <&> map (^.bName) <&> catMaybes
    liftIO $ print buckets
    let bucketToList = bucketSelector buckets
    paginate (listObjects bucketToList) $$ awaitForever (liftIO . print)
    case objectToRetrieve of
      Just name -> do
        obj <- send $ getObject bucketToList name
        liftIO $ print obj
        (obj^.gorBody) `connectBody` awaitForever (liftIO . print)
      Nothing ->
        return ()
  case er of
   Left err -> print err
   Right _ -> return ()

[brendanhay]

Thanks! You beat me before I could push the fix for Endpoint to develop.

I'll check it out.

[brendanhay]

I've successfully verified that c2f3c40 in combination with previous round of fixes results in things working as expected. The S3 example mostly demonstrates this, but avoids actually printing/downloading the objects since who knows what people keep in their buckets.