Comments (9)
Please provide more detailed reproducible steps, along with sample code.
from minio-hs.
i pull a lot of file from the server for backup operation(about 300K).
the server become unstable status and response with http timeout.
i fix the issue by add simple retry operation in list conduit, and add http managerSetting timeout duration.
i just hardcode for default retry policy from hackage retry library
the getObject retry can be process in my call executable, so it's not a problem.
Network/Minio/ListOps.hs
...
-- | List objects in a bucket matching the given prefix. If recurse is
-- set to True objects matching prefix are recursively listed.
listObjectsV1 :: Bucket -> Maybe Text -> Bool
-> C.ConduitM () ObjectInfo Minio ()
listObjectsV1 bucket prefix recurse = loop Nothing
where
loop :: Maybe Text -> C.ConduitM () ObjectInfo Minio ()
loop nextMarker = do
let delimiter = bool (Just "/") Nothing recurse
let listObjectsMaybe = fmap Just (listObjectsV1' bucket prefix nextMarker delimiter Nothing)
`catch` \(e :: SomeException) -> liftIO (putStrLn (show e)) >> return Nothing
res <- lift $ fmap fromJust $ retrying def (const $ return . isNothing) (const listObjectsMaybe)
liftIO $ putStrLn $ "[debug] fetching dir ..."
CL.sourceList $ lorObjects' res
when (lorHasMore' res) $
loop (lorNextMarker res)
from minio-hs.
i upload the related code to github:
but i don't have tons of data for reproducible, as i just write simple script to use.
https://github.com/clojurians-org/haskell-example/blob/master/minio-migration/src/Main.hs
from minio-hs.
Thanks for the report, we will take a look.
from minio-hs.
@clojurians-org We understand the problem - the library does not implement any retry functionality for request timeouts. We will add this feature. Thanks again for the report!
from minio-hs.
@clojurians-org Could you try out version 1.3.0? It contains a retry mechanism.
from minio-hs.
yes, it's work now.
BTW, i use the folloing code to build MinioConn for speicifying timeout.
the ConnectInfo(..) data constructor didn't export, the connectIsSecure can't be used.
mkMC :: ConnectInfo -> Text -> Text -> Int -> IO MinioConn
mkMC ci accessKey secretKey timeout = do
let ci' = ci & setCreds (Credentials accessKey secretKey) & setRegion "us-east-1"
let timeoutSeconds = 1000 * 1000 * timeout
let settings | connectIsSecure ci' = tlsManagerSettings
| otherwise = defaultManagerSettings
mgr <- newManager settings
{ managerResponseTimeout = responseTimeoutMicro timeoutSeconds }
mkMinioConn ci' mgr
from minio-hs.
the ConnectInfo(..) data constructor didn't export, the connectIsSecure can't be used.
@clojurians-org We have a fix here #121 - we will make a minor release after it is merged.
from minio-hs.
@clojurians-org Version 1.3.1 has change #121 and should solve your problem.
from minio-hs.
Related Issues (20)
- Replace Minio with MinIO and minio.io with min.io
- minio-hs-1.2.0 build error: No instance for (MonadThrow Minio) HOT 3
- Minio constructor is hidden for runResourceT intergration. HOT 4
- minio putObject always fail! HOT 6
- putObject don't support Conduit sink way HOT 3
- bucket notify event support ? HOT 1
- Make MinioT monad transformer HOT 1
- Support for Yandex Object Storage
- presignedGetObjectUrl url is not accepted by S3 (permanentRedirect) HOT 6
- Tests start failing with unordered-containers 0.2.16
- Add support for STS credentials
- Add support for calling MinIO's STS APIs
- Add support for listing objects with versioning
- Add support for fetching and deleting specific versions of objects
- Add APIs to enable and suspend versioning on a bucket
- Add network tracing support HOT 1
- minio-hs-0.3.2 build fails with latest resourcet HOT 8
- poo* and goo* functions are lacking documentation on hackage HOT 2
- Implement setConfig/getConfig admin API
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from minio-hs.