osglworks / java-storage Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 2.0 193 KB

a java storage layer

CSS 14.82% Java 85.18%

java-storage's People

Contributors

Stargazers

Watchers

Forkers

laurentiudelcea fullee

java-storage's Issues

The `ss.storage.s3.defStorageClass` configuration is not effective

The default storage class is always stick to the hardcoded value: reduced redundance

Missing attributes file since 1.4.0

When persist an SObject it shall persist the object binary content plus an attribute file. It found that there is no attribute file since 1.4.0

FileSystem service always init root_ to null when started up

as title

Add factory method to load `SObject` from class path

S3 allow only US-ASCII characters in the meta data

See https://stackoverflow.com/questions/45734959/aws-s3-error-when-there-are-chinese-characters-in-the-usermetadata-of-putobject/45746614#45746614

This request is to revamp the current S3 storage serive by using object tag to replace user meta for saving the storage object attributes

File system service: filename not read on sobject load

The attribute file is not correctly read on loading the sobject

It cannot determine `.png` file's mime type

SObject.of(File) logic is using java activation's MimetypesFileTypeMap to check mimetype based on filename. However java activation's MIME type detection is very limited that it cannot determine png, pdf file.

This request is to enhance the MIME type detection when constrcuting an sobject from a file

Optimize FileObject implementation

rewrite FileObject.asFile() and many other parts so it performs better

Improve storage logic on handling resource not found and access denied exceptions

Refer to osglworks/java-tool#231

Make logic changes in specific storage service class accordingly.

Refactor azure/aws service/sobj implementation

as title.

Extract common logic between AWS and Azure impl to improve the code reuse
Rename azure impl to improve the readability

It reports `NullPointerException` when trying to load an SObject from persistent storage without attribute file

Error stack:

14:53:10.170 [XNIO-1 task-1] ERROR a.h.b.c.RequestHandlerProxy - Error handling request: [POST] /single/by_file
java.lang.NullPointerException: null
	at java.util.Properties$LineReader.readLine(Properties.java:434)
	at java.util.Properties.load0(Properties.java:353)
	at java.util.Properties.load(Properties.java:341)
	at org.osgl.storage.impl.FileSystemService.doGetMeta(FileSystemService.java:77)
	at org.osgl.storage.impl.StorageServiceBase.getMeta(StorageServiceBase.java:337)
	at org.osgl.storage.impl.StorageServiceBase.getMeta(StorageServiceBase.java:332)
	at org.osgl.storage.impl.StorageObject.<init>(StorageObject.java:21)
	at org.osgl.storage.impl.FileObject.<init>(FileObject.java:8)
	at org.osgl.storage.impl.FileSystemService.newSObject(FileSystemService.java:139)
	at org.osgl.storage.impl.StorageServiceBase.getFull(StorageServiceBase.java:256)
	at act.util.UploadFileStorageService._store(UploadFileStorageService.java:89)
	at act.util.UploadFileStorageService.store(UploadFileStorageService.java:73)
	at act.data.ApacheMultipartParser.parse(ApacheMultipartParser.java:57)
	at act.app.ActionContext.bodyParams(ActionContext.java:378)
	at act.app.ActionContext.getBody(ActionContext.java:366)
	at act.app.ActionContext.paramVal(ActionContext.java:345)
	at act.inject.param.StringValueResolverValueLoader.load(StringValueResolverValueLoader.java:42)
	at act.inject.param.JsonParamValueLoader.load(JsonParamValueLoader.java:70)
	at act.inject.param.ScopedParamValueLoader.load(ScopedParamValueLoader.java:49)
	at act.inject.param.ParamValueLoaderService.loadMethodParams(ParamValueLoaderService.java:182)
	at act.handler.builtin.controller.impl.ReflectedHandlerInvoker.params(ReflectedHandlerInvoker.java:695)
	at act.handler.builtin.controller.impl.ReflectedHandlerInvoker.handle(ReflectedHandlerInvoker.java:282)
	at act.handler.builtin.controller.ControllerAction.handle(ControllerAction.java:53)
	at act.handler.builtin.controller.RequestHandlerProxy._handle(RequestHandlerProxy.java:494)
	at act.handler.builtin.controller.RequestHandlerProxy.handle(RequestHandlerProxy.java:203)
	at act.handler.DelegateRequestHandler.handle(DelegateRequestHandler.java:47)
	at act.route.Router$ContextualHandler.handle(Router.java:1351)
	at act.xio.NetworkHandler$3.run(NetworkHandler.java:150)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Add getter and setter for filename, contentType on ISObject

The getter and setter for filename, contentType is used as a easier way to call

sobject.getAttribute(ATTR_FILENAME);
sobject.setAttribute(ATTR_FILENAME, "myfile.ext");
...

DumbObject shall return valid url in getUrl call

When a storage service is configured with GET_NO_GET setting to true it will never grab the object instead it will just returning a dumb object.

This is good for the case when app use cloud based storage and saves server time because those item shall be accessed directly through the cloud instead of through the app.

This request is to get dumb object save the correct URL, normally by concatenation of the configured staticWebEndpoint and the key of the object

Exception uploading file into S3 bucket

Put a file storage object with filename contains & into S3Service, it raised an exception:

Error stack:

2020-02-22_16:02:16.173 [XNIO-1 task-7] ERROR a.h.b.controller.RequestHandlerProxy - Server error encountered on handling request: [POST]/admin/resources/upload
com.amazonaws.services.s3.model.AmazonS3Exception: The TagValue you have provided is invalid (Service: Amazon S3; Status Code: 400; Error Code: InvalidTag; Request ID: B287CF020325CC88; S3 Extended Request ID: ib4BClwjqql+On8NusSCsO7sIYoX6Y7WsgVgMtXaeEeIrIC8Ce58ivwgaSVZ9HwHLoMoyYtnXxI=)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1640)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4365)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4312)
	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1755)
	at org.osgl.storage.impl.S3Service.doPut(S3Service.java:212)
	at org.osgl.storage.impl.StorageServiceBase.put(StorageServiceBase.java:292)

File system service: attribute file inputstream is not closed on read

When reading attribute file on loading sobject, the input stream is not closed after reading

Enable user defined `KeyGenerator`

At the moment the KeyGenerator has only limited options. This request is to allow osgl-storage support plugin user defined KeyGenerator logic

Put azure sobject might cause 0 length object be saved on the cloud

At the moment Azure and AWS SObject implementation returns 0 as length. This might lead to empty length object be saved on the cloud

NPE triggered when loading storage object without attr file

Error stack:

org.osgl.exception.UnexpectedIOException: File does not exists: /home/luog/p/act/samples/upload/store1/2017/04/18/844dc10d-fac9-4a43-b3c4-70319c61f0a5.attr
	at org.osgl.util.E.ioException(E.java:119)
	at org.osgl.util.IO.is(IO.java:146)
	at org.osgl.storage.impl.FileSystemService$1.transform(FileSystemService.java:142)
	at org.osgl.storage.impl.FileSystemService$1.transform(FileSystemService.java:139)
	at org.osgl.Osgl$Transformer.apply(Osgl.java:2541)
	at org.osgl.storage.impl.FileSystemService.doOperate(FileSystemService.java:119)
	at org.osgl.storage.impl.FileSystemService.doOperate(FileSystemService.java:93)
	at org.osgl.storage.impl.FileSystemService.doGetMeta(FileSystemService.java:75)
	at org.osgl.storage.impl.StorageServiceBase.getMeta(StorageServiceBase.java:337)
	at org.osgl.storage.impl.StorageServiceBase.getMeta(StorageServiceBase.java:332)
	at org.osgl.storage.impl.StorageObject.<init>(StorageObject.java:21)
	at org.osgl.storage.impl.FileObject.<init>(FileObject.java:5)
	at org.osgl.storage.impl.FileSystemService.newSObject(FileSystemService.java:126)
	at org.osgl.storage.impl.StorageServiceBase.getFull(StorageServiceBase.java:255)
	at org.osgl.storage.impl.StorageServiceBase.put(StorageServiceBase.java:294)
	at demo.upload.UploadDemo.testRemoteFetch(UploadDemo.java:47)
	at demo.upload.UploadDemoMethodAccess.invoke(Unknown Source)
	at act.handler.builtin.controller.impl.ReflectedHandlerInvoker.invoke(ReflectedHandlerInvoker.java:413)
	at act.handler.builtin.controller.impl.ReflectedHandlerInvoker.handle(ReflectedHandlerInvoker.java:220)
	at act.handler.builtin.controller.ControllerAction.handle(ControllerAction.java:42)
	at act.handler.builtin.controller.RequestHandlerProxy._handle(RequestHandlerProxy.java:445)
	at act.handler.builtin.controller.RequestHandlerProxy.handle(RequestHandlerProxy.java:187)
	at act.handler.DelegateRequestHandler.handle(DelegateRequestHandler.java:47)
	at act.route.Router$ContextualHandler.handle(Router.java:1245)
	at act.xio.NetworkHandler$1.run(NetworkHandler.java:122)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Allow plugin key name provider and key generator

At the moment there are only UUID based key name provider and three predefined key generator. This request allow user supplied key name provider and key generator to be plugged into the aaa library

Allow store sobject with suffix attached to the key

Previously when the sobject is stored there is no suffix:

This request is to allow AAA to store sobject with suffix when possible (e.g. when storing an uploaded file, use the filename's suffix as the sobject's key suffix)

"Cannot create dir" on 1.4.0 with FileSystemService

org.osgl.exception.UnexpectedIOException: Cannot create dir: /home/luog/p/act/samples/upload/store2/2017/08/31
	at org.osgl.util.E.ioException(E.java:119)
	at org.osgl.storage.impl.FileSystemService.doOperate(FileSystemService.java:120)
	at org.osgl.storage.impl.FileSystemService.doPut(FileSystemService.java:141)
	at org.osgl.storage.impl.StorageServiceBase.put(StorageServiceBase.java:293)

Add getUrl() method to ISObject

Calling to ISObject.getUrl() function should have the same effect as calling ISObject.getAttribute(ISObject.ATTR_URL)

Add getFile(String fullPath) protected method into FileSystemService

This allows extended FileSystemService implementation to get File from a full path key. ActFramework uses it to implement a UploadFileService, which will keep upload file content into memory unless the size exceeds in memory threshold setting

osglworks / java-storage Goto Github PK

java-storage's People

Contributors

Stargazers

Watchers

Forkers

java-storage's Issues

Recommend Projects

Recommend Topics

Recommend Org