libreio / libre

Simple AWS gateway for storing and sharing files

#libre

Libre is a cloud storage platform that enables users to deploy a simple AWS gateway for storing and sharing files.

##Starting the application:

$ mvn clean package
$ java -cp target/libre.jar:target/deps/* com.libre.Launch --port=8080

##If you're on Windows:

java -cp target\libre.jar;target\deps\* com.libre.Launch --port=8080

##How to contribute

Fork repository, make changes, send us a pull request. We will review your changes and apply them to the master branch shortly, provided they don't violate our quality standards. Before sending us your pull request please run full Maven build:

$ mvn clean install -Pqulice,cobertura

Don't forget to set your default encoding to UTF-8. On Windows:

SET JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF-8

To avoid build errors use maven 3.2+. Minimum Java version is 1.7.

To start a web server locally run:

$ mvn clean integration-test -Phit-refresh -Dport=8080

If you have questions or general suggestions, don't hesitate to submit a new GitHub issue.

libre's Issues

MkStorage fails to add a file if directory doesn not exist

This test fails

    @Test
    public void writesFileToAbsentDir() throws Exception {
        final String path = "foo/bar/write.txt";
        final String content = "A content";
        final String absolute = this.temp.getRoot().getAbsolutePath();
        new MkStorage(absolute).put(
            path, IOUtils.toInputStream(content)
        );
        MatcherAssert.assertThat(
            IOUtils.toString(
                Files.newInputStream(
                    FileSystems.getDefault().getPath(absolute, path)
                )
            ),
            Matchers.is(content)
        );
    }

Because path foo/bar/ does not exist and is not created before try.
MkStorage.put() should create missing folders before trying to put a file.

Storage design is too simple

At the moment we have this Storage interface, which, I believe, doesn't express the correct design. I would suggest these interfaces to the object model of the app:

interface User {
  Docs docs();
}
interface Docs {
  List<Doc> find(String name); // could return empty list if not found
  List<Doc> list(String prefix); // finds all docs by prefix, e.g. "/my/pictures"
}
interface Doc {
  String name();
  void load(OutputStream output);
  void save(InputStream input);
  void delete();
}

I would suggest to place them into com.nerodesk.om package. Then, create a package com.nerodesk.om.mock for their mock versions. Then, create com.nerodesk.aws for classes that implement this object model and communicate with AWS.

rultor should work with java 8

For the moment we cannot merge changes to use java 8 since rultor is working with JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64. To continue we need to update java. Full build log is here.

UPD 2
We need to update .rultor.yml appending the following snippet

docker:
  image: "mostalive/ubuntu-14.04-oracle-jdk8"

Launch.java:52-54: Bind getFile operation to the...

Puzzle 14-462b19d5 in src/main/java/com/nerodesk/Launch.java:52-54 has to be resolved: Bind getFile operation to the GET method only. This might require updates in Takes framework. Don't forget about unit tests.... The puzzle was created by longtimeago on 7-Mar-2015.

If you have any technical questions, don't ask me, submit new tickets instead

Launch.java:50-52: This is a utility class...

Puzzle 12-18bd4eea in src/main/java/com/nerodesk/Launch.java:50-52 has to be resolved: This is a utility class and should only launch the app. The web server code should be moved to its own class. Remove the checkstyle suppression above and also the PMD suppression below.... The puzzle was created by Felipe Pina on 4-Mar-2015.

If you have any technical questions, don't ask me, submit new tickets instead

Incorrect permissions.

Something is wrong with permission level, please see #23.

Let's introduce web framework

If we want to create webapp we have to serve CSS, html files; url mappings etc. We should not reinvent the wheel and use some existing framework that will take care of this for us.

Incomplete documentation: how to start the application

Let's add information about executing the application into README.md. For the moment the application is executed by

java -jar target\nerodesk-jar-with-dependencies.jar port

The system should be able to upload/download and delete files in S3

Currently we have no notion of S3. Let's fix that. Implement another version of Storage interface that will use Amazon S3.

We might use http://s3.jcabi.com/ for that purpose.

removed

Option UseSplitVerifier; support was removed in 8.0

Seems like this option is configured in pom.xml for maven-surefire

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier; support was removed in 8.0

this happens when building with mvn clean install -Pqulice,cobertura. Using java8 oracle:

java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)

Launcher.main: ArrayIndexOutOfBoundsException

$ java -jar target/nerodesk-jar-with-dependencies.jar

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
at com.nerodesk.Launch.main(Launch.java:65)

Program should either start with a fixed default port or, if you want to enforce the specification of a port, the program should exit cleanly with an error message instead of throwing an Exception.

removed

README.md contains broken link

For the moment README.md contains broken link to the image and looks like

0.2

Please, publish a new release 0.2 (or higher)

Launch.java:68-70: Add new section to the...

Puzzle 14-78da13fc in src/main/java/com/nerodesk/Launch.java:68-70 has to be resolved: Add new section to the README described REST API for the file operations available. GET, PUT and DELETE operations should be described.... The puzzle was created by longtimeago on 7-Mar-2015.

If you have any technical questions, don't ask me, submit new tickets instead

Launch.java:65-67: Update FkRegex patterns for all...

Puzzle 14-186752b9 in src/main/java/com/nerodesk/Launch.java:65-67 has to be resolved: Update FkRegex patterns for all file operations (get, put, delete) to manage filename with path. Now it works with pure file name only. Don't forget about unit tests.... The puzzle was created by longtimeago on 7-Mar-2015.

If you have any technical questions, don't ask me, submit new tickets instead

Re-estimate project size

The latest estimate is rather old (created on 20-Feb-2015),we have to create a new one as explained here

removed

User should be able to login via Facebook OAuth.

User should be able to login via Facebook OAuth but there is no notion about this in code yet.

We should use provided access codes and PsFacebook class is implemented in Takes 0.5

Access codes:

• facebook acc ID:provided via file`
• secret code:provided via file`

Please use suggested by Yegor way of providing those values (see #11 (comment))

removed

LaunchTest.java:59-60: Application should be able to...

Puzzle 14-8cd54fa2 in src/test/java/com/nerodesk/LaunchTest.java:59-60 has to be resolved: Application should be able to get binary file properly. Add a test to check it works and fix if doesn't.... The puzzle was created by longtimeago on 7-Mar-2015.

If you have any technical questions, don't ask me, submit new tickets instead

Let's create support for Facebook OAuth

User should be able to login/logout using Facebook OAuth. We have to implement it.

Facebook OAuth will provide us the user ID, which is enough to identify the user. We just encrypt it and send back to the user in some custom X-Nerodesk-User cookie.

FtCLI and upgrade to takes 0.8.4

Let's use FtCLI in Launch, since it allows command line arguments parsing. Also, let's upgrade takes to version 0.8.4 (the latest one), which will give us that class.

Re-estimate project size

The latest estimate is rather old (created on 3-Mar-2015),we have to create a new one as explained here

removed

Launch.java:55-57: Implement PUT operation to upload...

Puzzle 14-9e19a32c in src/main/java/com/nerodesk/Launch.java:55-57 has to be resolved: Implement PUT operation to upload file to the store under specific path. This might require updates in Takes framework. Don't forget about unit tests.... The puzzle was created by longtimeago on 7-Mar-2015.

If you have any technical questions, don't ask me, submit new tickets instead

Storage.java:42-44: Implement put operation to add...

Puzzle 13-97594e76 in src/main/java/com/nerodesk/Storage.java:42-44 has to be resolved: Implement put operation to add file to the storage. In MkStorage.put() implementation file should be stored on local file system. Don't forget about unit tests.... The puzzle was created by longtimeago on 23-Feb-2015.

If you have any technical questions, don't ask me, submit new tickets instead

Launch.java:58-61: Implement DELETE operation to remove...

Puzzle 14-da645c40 in src/main/java/com/nerodesk/Launch.java:58-61 has to be resolved: Implement DELETE operation to remove file from the store by file specific path. This might require updates in Takes framework. Don't forget about unit tests.... The puzzle was created by longtimeago on 7-Mar-2015.

If you have any technical questions, don't ask me, submit new tickets instead

RqRegex.Fake from takes 0.8.5

Let's use RqRegex.Fake from takes 0.8.5 in TkGetFileTest. The use of this class will make the test way shorter and easier to read.

Also, it's a bad idea to have that public static property PATH in TkGetFile. There is not connection between a regular expression used in Launch. We can use two absolutely different regular expressions - one in Launch and another one in tests.

In general, public static is a terrible violation of OOP principles, try to avoid that.

user interface isn't available in Russian

Current version of README doesn't say anything about translating Nerodesk for our Russian users. This is a bug, since this feature is important and will improve usability of the system. Let's modify the README and document this feature. Then, let's implement it and test.

Also see discussion #191

removed

removed

Rultor and Travis inconsistence

For the moment .rultor.yml makes mvn clean install -Pqulice,cobertura while .travis.yml executes only mvn clean install -Pqulice. Let's include cobertura checks into travis builds as well.

removed

Storage.java:45-47: Implement delete operation to remove...

Puzzle 13-f1fbb1e7 in src/main/java/com/nerodesk/Storage.java:45-47 has to be resolved: Implement delete operation to remove file from the storage. In MkStorage.delete() implementation file should be removed from local file system. Don't forget about unit tests.... The puzzle was created by longtimeago on 23-Feb-2015.

If you have any technical questions, don't ask me, submit new tickets instead

Missing JSR-303 validator

mvn clean install -Pqulice WARNs about JSR-303 validator:

[ERROR] JSR-303 validator failed to initialize: Unable to create a Configuration, because no Bean Validation provider could be found. Add a provider like Hibernate Validator (RI) to your classpath. (see http://aspects.jcabi.com/jsr-303.html)

No PDD restrictions

For the moment there are no .pdd restrictions defined. Do we need this file to define min- and max-estimate, min-words, etc?

Estimate project size

There is no estimate in the project yet. We have to create one as explained here

Launch.java:45-48: This is a utility class...

Puzzle 48-cc1173e1 in src/main/java/com/nerodesk/Launch.java:45-48 has to be resolved: This is a utility class and should only launch the app. The web server code should be moved to its own class. Remove the checkstyle suppression above and also the PMD suppression below. This is continuation of task #48. I moved TkIndex to separate class but it's not enough.... The puzzle was created by Grzegorz Gajos on 6-Mar-2015.

If you have any technical questions, don't ask me, submit new tickets instead

User should be able to login/logout

We should implement flexible authorisation and session storage mechanism. So it would be possible to extend authorisation and support multiple "bindings". Let's focus on flexible design here. For version one we could use the easiest form of authentication. Username only :), no password.

We should be able to easily extend our auth mechanism and support login/logout using Facebook OAuth mechanism.

Let's create file REST resource

User should be able to upload/download and delete file. Because frontend is not yet specified we can introduce REST interface that could be reused later on.

When #12 will be introduced and #13 available let's create file REST resource:

• PUT /api/file/examplefilename.jpg - uploads file and store it under specific name
• GET /api/file/examplefilename.jpg - gets specific file or return 404 if res. not available
• DELETE /api/file/examplefilename.jpg - deletes specific file or return 404 if res. not available

We should document this in readme file also.

Let's create storage interface

We have no file management for now.

We have to implement some simple file management. Let's create Storage interface and simple impl. that will be able to put/get/delete any file. For now we can create simple implementation that will store all files in ~/.nerodesk/storage.

LaunchTest.returnsFileContent randomly fails

LaunchTest.returnsFileContent randomly fails. Log below:

Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 9.997 sec <<< FAILURE! - in com.nerodesk.LaunchTest
returnsFileContent(com.nerodesk.LaunchTest)  Time elapsed: 3.882 sec  <<< ERROR!
com.jayway.awaitility.core.ConditionTimeoutException: com.nerodesk.LaunchTest.returnsFileContent Callable expected "some text content" but was <null> within 2 seconds.
    at com.jayway.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:96)
    at com.jayway.awaitility.core.AbstractHamcrestCondition.await(AbstractHamcrestCondition.java:87)
    at com.jayway.awaitility.core.ConditionFactory.until(ConditionFactory.java:590)
    at com.jayway.awaitility.core.ConditionFactory.until(ConditionFactory.java:408)
    at com.nerodesk.LaunchTest.returnsFileContent(LaunchTest.java:134)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:318)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Results :
Tests in error: 
  LaunchTest.returnsFileContent:134 » ConditionTimeout com.nerodesk.LaunchTest.r...

Please see https://travis-ci.org/teamed/nerodesk/builds/54266652 for full log.

Code coverage check is missing

Code coverage check is missing.
I propose to configure cobertura-maven-plugin as a separate profile in pom.xml to check code coverage by tests

Launch.java:62-64: Add exception handling to return...

Puzzle 14-7a1bdbb3 in src/main/java/com/nerodesk/Launch.java:62-64 has to be resolved: Add exception handling to return 404 when resource not available. Now we show not nice stacktrace. Don't forget about unit tests.... The puzzle was created by longtimeago on 7-Mar-2015.

If you have any technical questions, don't ask me, submit new tickets instead

Documentation does not specify minimum Java version

Is Java 7 allowed or is Java 8 the minimum version? Keep in mind that Spark 2 requires Java 8.

skeleton, version 0.1

I will create a skeleton of the project and will configure deployment pipeline

Section "How to contribute" is out of date

The section "How to contribute" of README.md should tell us to invoke mvn clean install -Pqulice,cobertura since cobertura checks are added to travis build.

Launch.java:47-51: This is a utility class...

Puzzle 48-8978ee07 in src/main/java/com/nerodesk/Launch.java:47-51 has to be resolved: This is a utility class and should only launch the app. The web server code should be moved to its own class. Remove the checkstyle suppression above and also the PMD suppression below. This is continuation of task #48. I moved TkIndex to separate class but it's not enough. suppression above and also the PMD suppression below.... The puzzle was created by Grzegorz Gajos on 6-Mar-2015.

If you have any technical questions, don't ask me, submit new tickets instead