Stocator Based Implmentation about streamsx.objectstorage

[apyasic]

@markheger right, the idea is to copy BigData.jar from $STREAMS_INSTALL/toolkits/com.ibm.streamsx.hdfs/impl/lib during build.

[apyasic]

@ddebrunner and @markheger
Working on stocator based object store implementation, i've run into the following dilemma.

ObjectScan/Source/Sink operators code mimic a lot of logic of HDFSDirectoryScan, HDFSSource and HDFSSink.

In the initial ObjectScan/Source/Sink operators'
implementation I even used BigData.jar from HDFS toolkit
(containing HDFS operator implementations) as a library.
Yet, it required some tweaking, like hiding unrelevant parameters,
like hdfsUri, hdfsUser, etc. In addition, BigData.jar contains
hadoop client implementation (based on Apache Hadoop client)
which was replaced with stocator-based client.

What do you think is the best approach to take?
Is it better to duplicate relevant HDFS classes in the object storage toolkit
or still to reuse HDFS operators implementation as a library?

[markheger]

There are several branches of the HDFS toolkit. A good code base would be the "bluemix" branch. The difference between the branches is mainly the loading of the jars. We should not depend on any HADOOP_HOME environment variable.
I would go with the approach to have a copy of the classes, because it would be very confusing if this toolkit would depend on the hdfs toolkit.

[apyasic]

@markheger what if would take kind of intermediate approach: independent operators hierarchy from one side as you proposed and utilization of "infrastructure" classes like HDFSFile and AsyncBufferWriter from HDFS Toolkit BigData.jar?

[markheger]

@apyasic I agree to your suggestion to take few classes from the BigData.jar.
You mean, that during toolkit build the BigData.jar is copied from $STREAMS_INSTALL/toolkits/com.ibm.streamsx.hdfs/impl/lib into the com.ibm.streamsx.objectstorage toolkit "opt" directory?
Or depend on hdfs toolkit and let the jar located in hdfs toolkit?

[alexpy11]

@markheger looking for collective name for "container" and "bucket".
Do you have ideas?

[markheger]

Shall we keep the word container for both?
"A bucket is a container for objects stored in Amazon S3. Every object is contained in a bucket" http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html

[alexpy11]

@markheger agreed. In the "wrapped" version the "bucket" parameter will be used instead to make it compiant with S3 vocabulary.