Comments (15)
It is possible and we have min.io on the roadmap. We invite you to try Quilt with a min.io endpoint and file bugs that you encounter, as we have yet to formalize support. In theory the min.io API means that it just works, in practice it's not that simple due to assumptions in the code and/or missing features in min.io.
from quilt.
@akarve I am trying to establish Quilt as a core component of the data infrastructure at our research org. AWS is a non-starter for us so I am attempting to slowly fill in the AWS-dependent gaps with MinIO compatibility starting with the quilt3 python package - initially as a standalone that does not rely on a registry server. I quickly hacked together a solution that mainly just involves modifying the S3ClientProvider._build_client
method to create a client with endpoint_url specified. Currently I just check an environment variable for the endpoint url and if it exists, create the client with the endpoint url, otherwise the same old way.
quilt3/datatransfer.py
class S3ClientProvider:
...
def _build_client(self, get_config):
session = self.get_boto_session()
endpoint_url = getenv_s3_endpoint_url()
if endpoint_url:
return session.client(
's3',
config=Config(signature_version='s3v4'),
endpoint_url=endpoint_url,
)
return session.client('s3', config=get_config(session))
As far as credentials, I currently edit the CREDENTIALS_PATH
file with MinIO user credentials and it works fine.
Now this is just a starting implementation and far from optimal, but I'm wondering if this standalone MinIO-compatible mode is something that you're interested in supporting in the quilt3 python package and if you have any ideas as far as things to consider in the design.
Thanks!
from quilt.
Glad to hear that. I will definitely try it and let you know the results.
Thanks
from quilt.
One suggestion to encourage more people interested in testing it is to provide a small how-to.
from quilt.
from quilt.
@matheusmota I know your issue is only from six days ago but have you already given this a shot? Did you by any chance take some notes that you are willing to share if you started on this?
Either way, I will try to set up MinIO on a local node and address it with quilt in the coming days.
from quilt.
waiting for minio, grate jobs
from quilt.
@matheusmota @Midnighter Any updates on Min.IO? Maybe you can share your experience? I am currently considering giving quilt a go, but only have mio.io available
from quilt.
I briefly tried and was not successful. I haven't been able to give it a more serious attempt since then.
from quilt.
@marcodlk nice workaround and directionally correct (sorry for the slow reply). what we're planning to do here is in the next-gen client (already in the works and will be open source) to abstract the providers a little bit so that at first any object-compatible store can be interposed (GCP, Azure, MinIO) so that's the long term solution and we don't have code just yet. wanna join our Slack and we can discuss further? thank you.
from quilt.
With boto3>=1.28.0 you can use AWS_ENDPOINT_URL_S3 to customize endpoint URL.
See https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html.
from quilt.
Hi @marcodlk Can you share the diff of the change you make? It looks like quilt never access the credentials.json file.
from quilt.
@link89 I no longer have access to the codebase I was working on, but looking at the code, quilt3.session._load_credentials
still uses CREDENTIALS_PATH
so that's odd. Are you sure it is the "credentials.json" in the Quilt app directory as specified by BASE_PATH
in quilt3.util
module? Have you tried @sir-sigurd 's solution?
from quilt.
For min.io support we hopefully don't need to touch credentials.json
as that is for the special case where users authenticate to a Quilt stack. But in the more general case quilt3
just falls back onto the boto3 credential chain (and never touches credentials.json) and that is applicable in more cases, especially for pure open source users.
from quilt.
Here is a draft PR that allows users to create their own S3 clients (including min.io clients) and map them to specific buckets. #3765
We'd appreciate any feedback on the interface. This isn't necessarily the best way for Quilt to find and access min.io servers. Please let us know how you think Quilt should map min.io endpoints and bucket names.
from quilt.
Related Issues (20)
- Catalog failing to render thumbnails on package view but succeeding on file view HOT 2
- Catalog does not render large Vega JSON files (created by altair) HOT 2
- Accessing package causes ascii decoding error HOT 3
- Add package created and last updated timestamps part of the API response HOT 2
- Can't resolve 'jsoneditor/dist/jsoneditor.css' HOT 4
- Dependency Dashboard HOT 2
- Add a way to pre-fill metadata on push HOT 4
- Cannot retrieve the old version data HOT 1
- quilt3 verify enhancement
- Remove $ref restriction HOT 1
- Remove schema version restriction HOT 3
- add kms docs
- Quilt 2.9.15 within AWS Lambda execution HOT 3
- Support for running Quilt in an EKS pod with OpenID authentication to AWS disappeared/missing HOT 2
- Workflows should enforce push-level tag completeness
- quilt3 push: --dir is a required argument
- Config resolution at import
- Object level metadata from JSON schema
- Unable to delete package with empty suffix
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from quilt.