Coder Social home page Coder Social logo

open-io / oio-api-java Goto Github PK

View Code? Open in Web Editor NEW
8.0 24.0 9.0 537 KB

OpenIO SDS Java client API, giving direct access to the object storage

Home Page: https://www.openio.io/

License: GNU Lesser General Public License v3.0

Java 100.00%
java object-storage openio-sds openio-sds-api openio

oio-api-java's Introduction

OpenIO SDS API

OpenIO SDS API is a java remote API for OpenIO Software Defined Storage. It is designed to be as simple and fast as possible.

Build Status Codecov


Getting started


OioUrl object

Each methods of the api will ask you for an OioUrl instance. This class simplify methods signatures and make code cleaner. You will build it simply by calling OioUrl#url(String, String) for container url, i.e. to perform operations on container, and by calling OioUrl#url(String, String, String) for object url.

The first parameter is called "ACCOUNT". It define a storage space inside your SDS namespace. It could be used for a dedicated application space for example. If you don't care about that, just choose an account name and keep the same all the time.

The second parameter is the container name. It is the space in which your future objects will be created directly. Its name is unique by account.

And the last parameter is the object name, which is the identifier of your object inside a container, so, like container in account, an object name is unique inside a container.

Container url example:

OioUrl url = OioUrl.url("MY_ACCOUNT", "MY_CONTAINER_NAME");

Object url example:

OioUrl url = OioUrl.url("MY_ACCOUNT", "MY_CONTAINER_NAME", "MY_OBJECT_NAME");

Storage client usage


OpenIO SDS API entry point is the Client implementations classes. Instances of these implementations are built with ClientBuilder class. You could create a basic client by calling ClientBuilder#newClient(String, String)} method, specifying the OpenIO namespace and proxyd service url as argument. If you don't know what is proxyd service, please refer to OpenIO SDS documentation here. Clients built from ClientBuilder are ready to be used. Let's see some basics examples.

Basic client instantiation
Client client = ClientBuilder.newClient("OPENIO", "http://127.0.0.1:6002");
Advanced client configuration
Settings settings = new Settings();
    settings.proxy()
            .ns("OPENIO")
            .url("http://127.0.0.1:6002")
            .ecd("http://127.0.0.1:5000"); //setup an ecd url for Erasure cogin management
    client = ClientBuilder.newClient(settings);
Container creation example

As simple as it could be:

OioUrl url = OioUrl.url("MY_ACCOUNT", "MY_CONTAINER_NAME");
client.createContainer(url);
Upload an object from an InputStream
OioUrl url = OioUrl.url("MY_ACCOUNT", "MY_CONTAINER_NAME", "MY_OBJECT_NAME");
File file = new File("MY_SAMPLE_FILE.txt");
FileInputStream fis = new FileInputStream(file);
client.putObject(url, 1024L, fis);
Retrieve and download an object
OioUrl url = OioUrl.url("MY_ACCOUNT", "MY_CONTAINER_NAME", "MY_OBJECT_NAME");
ObjectInfo oinf = client.getObjectInfo(url);
InputStream data = client.downloadObject(oinf);
List objects inside a container
OioUrl url = OioUrl.url("MY_ACCOUNT", "MY_CONTAINER_NAME");
ObjectList list = client.listContainer(url, new ListOptions());
Delete a content
OioUrl url = OioUrl.url("MY_ACCOUNT", "MY_CONTAINER_NAME", "MY_OBJECT_NAME");
client.deleteObject(url);
Delete a container

An empty container (it should be explicitly empty) could be deleted from your SDS namespace, as follow.

OioUrl url = OioUrl.url("MY_ACCOUNT", "MY_CONTAINER_NAME");
client.deleteContainer(url);

Exception management


OpenIO choose to make all possible exceptions as RuntimeException extensions. By this way, we allow cleaner code when you decide to not check exceptions because you can't deal with unexpected behaviour (IOException etc...). So our client throws only OioException : OioSystemException which indicates server error (connection issues, hardware issues,...), and others OioException extension for all use case errors. See io.openio.sds.exceptions package for exceptions detail.

Usage exceptions are detailed just below

Name Description
AccountNotFoundException The account you try to deal with does not exist
BadRequestException Some parameter is wrong in your request
ContainerExistException The container you try to create is already created
ContainerNotFoundException The container you want to reach does not exist
ObjectExistException The object you try to created is alreay created
ObjectNotFoundException The object you try to reach does not exist
OioException Main OpenIO exception class
OioSystemException A system error occurred during your request
ReferenceAlreadyExistException The reference you try to create is alreayd created
ReferenceNotFoundException the reference you try to reach does not exist

Contributing

This project was originally indented with tabulations but should from now on be indented with 4 spaces.

oio-api-java's People

Contributors

aymericdu avatar cdedeurwaerder avatar fvennetier avatar murlock avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

oio-api-java's Issues

Rawx client upload endless recursion after error

At some load testing, I came across the following issue:

io.openio.sds.exceptions.OioException: Failed to schedule rawx upload
        at io.openio.sds.storage.rawx.RawxClient.uploadPosition(RawxClient.java:286)
        at io.openio.sds.storage.rawx.RawxClient.uploadChunks(RawxClient.java:132)
        at io.openio.sds.DefaultClient.putObject(DefaultClient.java:150)
        at io.openio.sds.DefaultClient.putObject(DefaultClient.java:129)
        ... 56 more
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1e06266b rejected from java.util.concurrent.ThreadPoolExecutor@1900ae28[Running, pool size = 100, active threads = 100, queued tasks = 0, completed tasks = 675]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)[:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)[:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)[:1.8.0_111]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)[:1.8.0_111]
        at io.openio.sds.storage.rawx.RawxClient.uploadPosition(RawxClient.java:272)
        ... 52 more

For the active rawx threads I see stack traces like the following:

"RawxClient-Worker" Id=1134 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1ff6eed
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:64)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.http.OioHttp$RequestBuilder.stream(OioHttp.java:322)
    at io.openio.sds.http.OioHttp$RequestBuilder.sendRequest(OioHttp.java:272)
    at io.openio.sds.http.OioHttp$RequestBuilder.execute(OioHttp.java:213)
    at io.openio.sds.http.OioHttp$RequestBuilder.execute(OioHttp.java:183)
    at io.openio.sds.storage.rawx.RawxClient$2.call(RawxClient.java:261)
    at io.openio.sds.storage.rawx.RawxClient$2.call(RawxClient.java:236)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

After some minutes, stack traces have enlarged massively and will eventually (every 10 seconds an additional stack frame) die with a StackOverflowError:

"RawxClient-Worker" Id=1134 in TIMED_WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1ff6eed
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:64)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    ... ~200 times more ...
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.common.FeedableInputStream.read(FeedableInputStream.java:69)
    at io.openio.sds.http.OioHttp$RequestBuilder.stream(OioHttp.java:322)
    at io.openio.sds.http.OioHttp$RequestBuilder.sendRequest(OioHttp.java:272)
    at io.openio.sds.http.OioHttp$RequestBuilder.execute(OioHttp.java:213)
    at io.openio.sds.http.OioHttp$RequestBuilder.execute(OioHttp.java:183)
    at io.openio.sds.storage.rawx.RawxClient$2.call(RawxClient.java:261)
    at io.openio.sds.storage.rawx.RawxClient$2.call(RawxClient.java:236)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

This results in a complete denial of service for any data operations.

How to authenticate?

I didnt see any thing about login / password, how can I authenticate?

(swift client options: --os-tenant-name --os-username --os-password)

Thanks you.

getNamespaceInfo(): Expected BEGIN_OBJECT but was STRING

While using this code:

package it.customer.prova.openio.api;
import io.openio.sds.Client;
import io.openio.sds.ClientBuilder;
import io.openio.sds.models.NamespaceInfo;

public class GetNamespaceInfo {

    public static void main(String[] args) {
        Client client = ClientBuilder.newClient("OPENIO", "http://192.168.99.100:6007");

        NamespaceInfo namespace = client.getNamespaceInfo();
    }

}

We got the following error:

Exception in thread "main" com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 path $
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:221)
    at com.google.gson.Gson.fromJson(Gson.java:861)
    at io.openio.sds.http.Verifiers.extractError(Verifiers.java:144)
    at io.openio.sds.http.Verifiers.access$000(Verifiers.java:26)
    at io.openio.sds.http.Verifiers$4.verify(Verifiers.java:105)
    at io.openio.sds.http.OioHttp$RequestBuilder.execute(OioHttp.java:217)
    at io.openio.sds.http.OioHttp$RequestBuilder.execute(OioHttp.java:191)
    at io.openio.sds.http.OioHttp$RequestBuilder.execute(OioHttp.java:240)
    at io.openio.sds.proxy.ProxyClient.getNamespaceInfo(ProxyClient.java:135)
    at io.openio.sds.DefaultClient.getNamespaceInfo(DefaultClient.java:55)
    at it.customer.prova.openio.api.GetNamespaceInfo.main(GetNamespaceInfo.java:11)
Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 path $

It seems that the JSON return by getNamespaceInfo() is not matching anymore.

Support container listing

As a user I want to list containers within an account so that I can let clients of my application discover the object space without having to know where they have put objects months ago.

`IOException: Unexpected end of stream` on proxy request after period of inactivity

After some period of inactivity, it seems that a connection can be reused after it has been closed server-side, leading to IOException: Unexpected end of stream. Following is a shortened stack trace.

2016-12-07 10:54:12 Failed to perform request
io.openio.sds.exceptions.OioSystemException: Http request execution error
    at io.openio.sds.http.OioHttp$RequestBuilder.execute(OioHttp.java:235)
    at io.openio.sds.http.OioHttp$RequestBuilder.execute(OioHttp.java:191)
    at io.openio.sds.proxy.ProxyClient.getObjectInfo(ProxyClient.java:711)
    at io.openio.sds.DefaultClient.getObjectInfo(DefaultClient.java:169)
    at io.openio.sds.DefaultClient.getObjectInfo(DefaultClient.java:162)
    
...

Caused by: java.io.IOException: Unexpected end of stream
    at io.openio.sds.http.OioHttpResponse.read(OioHttpResponse.java:125)
    at io.openio.sds.http.OioHttpResponse.next(OioHttpResponse.java:108)
    at io.openio.sds.http.OioHttpResponse.readHeaders(OioHttpResponse.java:101)
    at io.openio.sds.http.OioHttpResponse.responseHead(OioHttpResponse.java:91)
    at io.openio.sds.http.OioHttpResponse.build(OioHttpResponse.java:46)
    at io.openio.sds.http.OioHttp$RequestBuilder.readResponse(OioHttp.java:255)
    at io.openio.sds.http.OioHttp$RequestBuilder.execute(OioHttp.java:214)
    ... 11 more

Deprecation warning during doc build

When building the doc (17.04 branch) Iget this warning:

With gradle-4.3.1:

+ gradle javadocJar
The ConfigurableReport.setDestination(Object) method has been deprecated and is scheduled to be removed in Gradle 5.0. Please use the method ConfigurableReport.setDestination(File) instead.
	at build_exkyn9y6uida8hg3qylcooyn0$_run_closure9$_closure19.doCall(/home/test/repo/oio-docs/repositories.work/oio-api-java/build.gradle:70)
	(Run with --stacktrace to get the full stack trace of this deprecation warning.)
:compileJavawarning: [options] bootstrap class path not set in conjunction with -source 1.6
1 warning

:processResources NO-SOURCE
:classes
:javadoc
:javadocJar

BUILD SUCCESSFUL in 3s
3 actionable tasks: 3 executed

With gradle-4.6:

+ gradle javadocJar
Starting a Gradle Daemon (subsequent builds will be faster)
:compileJavawarning: [options] bootstrap class path not set in conjunction with -source 1.6
1 warning

:processResources NO-SOURCE
:classes
:javadoc
:javadocJar

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
See https://docs.gradle.org/4.6/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 16s
3 actionable tasks: 3 executed

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.