yegor256 / s3auth

Amazon S3 HTTP Basic Auth Gateway: put your files into S3 bucket and make them accessible with a login/password through a browser

s3auth.com is a Basic HTTP Auth gateway in front of your private Amazon S3 bucket. Read this blog post for a more detailed explanation: Basic HTTP Auth for S3 Buckets.

Point your test.example.com CNAME to relay.s3auth.com, and register the domain in s3auth.com web panel. You will be able to access bucket's content in a browser with an HTTP basic auth. Your bucket will be accessible using your Amazon IAM credentials and with custom user/password pairs in your .htpasswd file (similar to Apache HTTP Server).

For example, try http://maven.s3auth.com/ (with username s3auth and password s3auth). You will access content of Amazon S3 bucket maven.s3auth.com, which is not readable anonymously otherwise.

How to contribute

Fork repository, make changes, send us a pull request. We will review your changes and apply them to the master branch shortly, provided they don't violate our quality standards. To avoid frustration, before sending us your pull request please run full Maven build:

$ mvn clean install -Pqulice

To avoid build errors use JDK 11+ and Maven 3.3+.

To run it locally:

$ mvn clean install -Phit-refresh -Dport=8080

You will be able to open it at http://localhost:8080

s3auth's Issues

let's use @Immutable annotation

Let's annotate immutable classes with com.jcabi.aspects.Immutable

restart after broken download

Would be nice to implement a "restart" option for large files download.

stack overflow in Htpasswd#authorize()

Mar 17 20:05:09 relay.s3auth.com user:  [ERROR] bc550d9a com.s3auth.hosts.Htpasswd: #authorized('s3auth', 's3auth'): thrown java.lang.StackOverflowError (NULL) in 4ms
Mar 17 20:05:09 relay.s3auth.com user:  [ERROR] bc550d9a com.s3auth.hosts.DefaultHost: #authorized('s3auth', 's3auth'): thrown java.lang.StackOverflowError (NULL) in 6ms
Mar 17 20:05:09 relay.s3auth.com user:  [ERROR] bc550d9a com.s3auth.hosts.SmartHost: #authorized('s3auth', 's3auth'): thrown java.lang.StackOverflowError (NULL) in 7ms
Mar 17 20:05:09 relay.s3auth.com user:  [ERROR] bc550d9a com.s3auth.relay.SecuredHost: #authorized('s3auth', 's3auth'): thrown java.lang.StackOverflowError (NULL) in 8ms
Mar 17 20:05:09 relay.s3auth.com user:  [ERROR] bc550d9a com.s3auth.relay.SecuredHost: #fetch('/'): thrown java.lang.StackOverflowError (NULL) in 9ms
Mar 17 20:05:09 relay.s3auth.com user:  [ERROR] bc550d9a com.s3auth.relay.HttpThread: #dispatch(): thrown java.lang.StackOverflowError (NULL) in 8s

pass all HTTP headers from Amazon to the client

Discussion started in #43. Let's pass all HTTP headers received from Amazon to the client. I remember that there was some reason in the past why we weren't able to do this, but let's re-consider that decision. Looks like it should be a harmful mechanism.

implement MD5 algorithm for htpasswd

Let's implement MD5 algo there

interpert If-Modified-Since header

Let's interpret If-Modified-Since HTTP header. This will significantly increase speed of page delivery for end-users

show bucket name in WWW-Authenticate header at first try

Let's show bucket name in WWW-Authenticate header at first try, instead of "s3auth"

incorrect exception when bucket is not registered

Misleading exception message when bucket is not found: failed to fetch /.htpasswd from.... see #40 for more details

support query string in request for asset

First off this is an awesome tool, thanks for writing.

Requests to files in s3 that contain a qs fail to fetch.

It is handy to pass along the query string for things like making sure browser is not caching css/js assets.

example:

http://foo.mydomain.com/api/resources/bootstrap.min.css?blah results in:

failed to fetch '/api/resources/bootstrap.min.css?blah' from 'foo.mydomain.com' (key=abcd)

http://foo.mydomain.com/api/resources/bootstrap.min.css however works...

don't fetch .htpasswd

We should protect .htpasswd file and never deliver it

write up full scale Terms of Use and Privacy Policy

We should create these documents..

URN instead of plain number for facebook user IDs

required before #21

Let's use com.jcabi.urn.URN instead of plain number for identities of Facebook users.

syslog trails per domain

As a new feature, would be nice to have an ability to forward all domain-specific events to a custom syslog host/port (configurable from the web front).

new jcabi-maven-skin

Let's apply it correctly

syslog logging doesn't work

syslog logging doesn't work for some reason

jcabi-heroku-maven-plugin instead of antrun script

Let's use jcabi-heroku-maven-plugin instead of antrun script

heroku deployment needs retries

heroku deployment needs retries

add config to docs

Very sweet tool guys.

Could you add something like the following to your readme/wiki, may help the next person save some time.

Create a new user via IAM (use these creds in s3auth.com), give this IAM user read only access to a bucket via:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": ["arn:aws:s3:::bucket.name"]
    }
  ]
}

Generate .htpasswd file with sha1:

htpasswd -nbs user password

upload it to root of your bucket, give it 'any authenticated user' access (or the new user that u just added).

Now add the files and give anyone with auth ability to read:

s3cmd -v -r --add-header=x-amz-acl:authenticated-read -M put ../api s3://api.mydomain.com/

Wait for DNS to propagate and you are done

enforce only HTTPS for content delivery

Let's enforce SSL connections only for content delivery

---

• 8-5eb4ae33/#191
• 191-7e266bd1/#290 (by )

large files: broken pipe

[ERROR] back-8 com.s3auth.relay.HttpResponse: #send('Socket[addr=/85.200.xx.xx,port=40605,localport=80]'): thrown java.net.SocketException ('Broken pipe') in 120ms
[WARN] back-8 com.s3auth.relay.HttpThread: #run(): IO problem: Broken pipe
[ERROR] back-8 com.s3auth.relay.HttpResponse: #send('Socket[addr=/85.200.xx.xx,port=40605,localport=80]'): thrown java.net.SocketException ('Broken pipe') in 190mcs
[ERROR] back-8 com.s3auth.relay.HttpThread: #run(): thrown java.lang.IllegalStateException ('java.net.SocketException: Broken pipe') in 8s
[WARN] back-8 com.jcabi.log.VerboseRunnable: escalated exception: java.lang.IllegalStateException: java.net.SocketException: Broken pipe
    at com.s3auth.relay.HttpThread.failure(HttpThread.java:224)
    at com.s3auth.relay.HttpThread.run_aroundBody0(HttpThread.java:139)
    at com.s3auth.relay.HttpThread$AjcClosure1.run(HttpThread.java:1)
    at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
    at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:70)
    at com.s3auth.relay.HttpThread.run(HttpThread.java:101)
    at com.jcabi.log.VerboseRunnable.run(VerboseRunnable.java:143)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Wor:603)
    at com.jcabi.log.VerboseThreads$2.run(VerboseThreads.java:181)
    at java.lang.Thread.run(Thread.java:679)
Caused by: java.net.SocketException: Broken pipe
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:220)
    at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:290)
    at sun.nio.cs.StreamEncoderimplFlush(StreamEncoder.java:294)
    at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:140)
    at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
    at com.s3auth.relay.HttpResponse.send_aroundBody0(HttpResponse.java:169)
    at com.s3auth.relay.HttpResponse$AjcClosure1.run(HttpResponse.java:1)
    at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
    at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:70)
    at com.s3auth.relay.HttpResponse.send(HttpResponse.java:142)
    at com.s3auth.relay.HttpThread.failure(HttpThread.java:222)... 15 more

---

• 33-9ff485f7/#95 (by )

flash messages are still sticky (even after #14)

changes were made in #14, but messages are still sticky and don't disappear after page refresh

flash messages are sticky

flash messages are sticky. they need better testing

directory issue

Am moving an octopress blog over to s3, and I want to hide it for the moment, hence s3auth.

Setup ok and working fine at the top level page, but not any directories.

The octopress pages map like this
/domain/page -> /domain/page/index.html

when I access /domain/page/index.html it works fine, but /domain/page/ or /domain/page just gives me an access error.

my domain is dev.damon.io

Thanks

papertrail for logs

Let's use papertrailapp.com for logs

flash messages

Let's introduce flash messages between REST pages

rexsl 0.4.8 with insets

Let's migrated to rexsl 0.4.8 and start using "insets"

can't login with FB company page

The login from Facebook seems to require a personal FB account. I tried to log inn with the company FB page, but that did not work.

detailed error report in case of rejected authorization

Would be convenient to show a detailed error report in case of rejected authorization. Now we just return 401 status code and a short message. This is not enough to detect possible errors in configuration.

Empty endpoint doesn't work during domain registration

Hi,

First off, thanks a lot for putting up this free service. I tried to register a new domain following the instruction.

Hostname: sa-sanofi.tsibiocomputing.com

I left "Optional S3 endpoint" blank since I use AWS US standard.

However, I got the following error message:

invalid AWS S3 region 's3.amazonaws.com'; invalid domain

Do you know what is wrong?

504 errors

I'm getting HTTP/1.1 504 Gateway Timeout's when I try to request my page(s). Used to work - I don't think I changed anything. .htpasswd in root of bucket has not changed.

any ideas?

Ryan Pendergast is my FB login. Its http://[mydomain].com/api/index.html . If you cant tell my domain from your end let me know and I'll email you it.

build is broken because of cobertura and aspectj conflict

build is broken because of cobertura and aspectj conflict

implement Crypt algorithm for Htpasswd

Let's implement it

reload from dynamo every few minutes

DynamoHosts should reload from Dynamo every few moments

---

• 1-f2c33b38/#90
• 90-43f9c549/#292 (by )

enable configuration of bucket names

At the moment we require bucket names to have the same names as domain names. Would be convenient to let users have any bucket name.

can't start RELAY because of broken maven dependencies

[ERROR] The build could not read 1 project -> [Help 1]
org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs:
[ERROR] Unresolveable build extension: Plugin org.cyclopsgroup:awss3-maven-wagon:0.1 or one of its dependencies could not be resolved: Failed to collect dependencies for org.cyclopsgroup:awss3-maven-wagon:jar:0.1 () @ 

    at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:363)
    at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:636)
    at org.apache.maven.DefaultMaven.getProjectsForMavenReactor(DefaultMaven.java:585)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:234)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
[ERROR]   
[ERROR]   The project com.s3auth:s3auth-relay-production:1.0-SNAPSHOT (/home/ubuntu/pom.xml) has 1 error
[ERROR]     Unresolveable build extension: Plugin org.cyclopsgroup:awss3-maven-wagon:0.1 or one of its dependencies could not be resolved: Failed to collect dependencies for org.cyclopsgroup:awss3-maven-wagon:jar:0.1 (): No versions available for org.codehaus.jackson:jackson-core-asl:jar:[1.4,) within specified range -> [Help 2]
org.apache.maven.plugin.PluginResolutionException: Plugin org.cyclopsgroup:awss3-maven-wagon:0.1 or one of its dependencies could not be resolved: Failed to collect dependencies for org.cyclopsgroup:awss3-maven-wagon:jar:0.1 ()
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:211)
    at org.apache.maven.project.DefaultProjectBuildingHelper.resolveExtensionArtifacts(DefaultProjectBuildingHelper.java:377)
    at org.apache.maven.project.DefaultProjectBuildingHelper.createProjectRealm(DefaultProjectBuildingHelper.java:237)
    at org.apache.maven.project.DefaultModelBuildingListener.buildExtensionsAssembled(DefaultModelBuildingListener.java:106)
    at org.apache.maven.model.building.ModelBuildingEventCatapult$1.fire(ModelBuildingEventCatapult.java:43)
    at org.apache.maven.model.building.DefaultModelBuilder.fireEvent(DefaultModelBuilder.java:1053)
    at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:388)
    at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:371)
    at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:560)
    at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:352)
    at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:636)
    at org.apache.maven.DefaultMaven.getProjectsForMavenReactor(DefaultMaven.java:585)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:234)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.sonatype.aether.collection.DependencyCollectionException: Failed to collect dependencies for org.cyclopsgroup:awss3-maven-wagon:jar:0.1 ()
    at org.sonatype.aether.impl.internal.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:258)
    at org.sonatype.aether.impl.internal.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:308)
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:199)
    ... 24 more
Caused by: org.sonatype.aether.resolution.VersionRangeResolutionException: No versions available for org.codehaus.jackson:jackson-core-asl:jar:[1.4,) within specified range
    at org.sonatype.aether.impl.internal.DefaultDependencyCollector.process(DefaultDependencyCollector.java:376)
    at org.sonatype.aether.impl.internal.DefaultDependencyCollector.process(DefaultDependencyCollector.java:544)
    at org.sonatype.aether.impl.internal.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:240)
    ... 26 more
[ERROR] 
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

IndexOutOfBoundsException when user photo is absent

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
  at java.util.LinkedList.entry(LinkedList.java:382)
  at java.util.LinkedList.get(LinkedList.java:332)
  at com.s3auth.rest.GoogleRs$1.photo(GoogleRs.java:147)
  at com.s3auth.rest.CryptedUser.photo(CryptedUser.java:123)
  at com.s3auth.rest.CryptedUser.toString(CryptedUser.java:136)

non-US -Region buckets don't work

When S3 bucket is not in US region this is the exception I'm getting:

The bucket you are attempting to access must be addressed using 
the specified endpoint. Please send all future requests to this endpoint

Deleted configuration appears to be cached

Thanks for what promises to be a very useful service for us once I get it working!

I've removed our existing configurations but it when GET'ing http://photobox-core-gems.core.photobox.com/.htpasswd s3auth is returning https://gist.github.com/d1a7f8e88c3cdbebef88, when I would expect "huh?".

Am I doing something dumb?

Thanks.

NPE in DefaultHost when bucket configuration is NULL

NPE in DefaultHost when bucket configuration is NULL

integration test for relay JAR

Let's add an integration test for the JAR in s3auth-relay

implement HTTP Digest Auth method for better security

Let's implement HTTP Digest Auth method for better security

log traffic consumption per domain

Would be nice to log traffic consumption per each domain and show that statistics to end-users

login with LinkedIn account

Let's implement one more login provider - LinkedIn

configure papertrailapp explicitly in log4j.properties

Let's configure papertrailapp explicitly in log4j.properties

my s3auth domain stopped working

Worked last I checked (~48 hrs ago). Now I get the following:

failed to fetch / from 'wsapi.mydomain.com' (key=AKIAJ....: ["'+suffix': The specified key does not exist."]

Any ideas? If you need my domain/key specifics let me know and I'll direct message/email you.

transmit HTTP headers from S3 to the client

besides getObject() let's use S3 meta data in order to generate Content-Type and Content-Length in responses.

login with Twitter account

Would be nice to implement login with Twitter account

ArrayIndexOutOfBoundsException in Htpasswd

[WARN] com.jcabi.log.VerboseRunnable: escalated exception: java.lang.ArrayIndexOutOfBoundsException: 1
        at com.s3auth.hosts.Htpasswd.fetch(Htpasswd.java:130)
        at com.s3auth.hosts.Htpasswd.authorized(Htpasswd.java:107)
        at com.s3auth.hosts.DefaultHost.authorized(DefaultHost.java:132)
        at com.s3auth.hosts.GuardedHost.authorized(GuardedHost.java:95)
        at com.s3auth.relay.SecuredHost.fetch(SecuredHost.java:134)
        at com.s3auth.relay.HttpThread.run(HttpThread.java:105)
        at com.jcabi.log.VerboseRunnable.run(VerboseRunnable.java:129)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at com.jcabi.log.VerboseThreads$2.run(VerboseThreads.java:182)
        at java.lang.Thread.run(Thread.java:679)

high-level UML documentation

Let's create some simple high-level UML documentation

login mechanism for corporate users

Let's introduce some other login mechanism for corporate users. Facebook looks like a good solution for individuals, but not for companies.