hibernating-rhinos / ravenfs Goto Github PK

At present, trying to upload a file larger than about 80 - 90Mb from the UI fails, with an out of memory exception being reported.

Presumably we need to rework UploadAsync to use explicit chunked uploading when called from Silverlight?

StorageSignatureRepository shouldn't be singleton because it have to manage with temporal files during its life. Having singleton we may accidentally remove files from other processes.
ISignatureRepository should be created for one file only.

throws a stack overflow exeption.

When I select a lot of files (around 200 in this case) to upload in one go through the U I quite often see some of the files fail to upload, with this exception having been thrown.

System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: System.Net.WebException: The remote server returned an error: NotFound.
at System.Net.Browser.ClientHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Func2.Invoke(T arg) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise) {"ExceptionType":"Microsoft.Isam.Esent.Interop.EsentWriteConflictException","Message":"Write lock failed due to outstanding write lock","StackTrace":" at Microsoft.Isam.Esent.Interop.Api.Check(Int32 err) in C:\\Work\\ravendb\\SharedLibs\\Sources\\managedesent-61618\\EsentInterop\\Api.cs:line 2739\r\n at Microsoft.Isam.Esent.Interop.Api.JetDelete(JET_SESID sesid, JET_TABLEID tableid) in C:\\Work\\ravendb\\SharedLibs\\Sources\\managedesent-61618\\EsentInterop\\Api.cs:line 2341\r\n at RavenFS.Storage.StorageActionsAccessor.Delete(String filename) in C:\\Projects\\HibernatingRhinos\\RavenFS\\RavenFS\\RavenFS\\Storage\\StorageActionsAccessor.cs:line 371\r\n at RavenFS.Controllers.FilesController.<>c__DisplayClass21.<Put>b__1c(StorageActionsAccessor accessor) in C:\\Projects\\HibernatingRhinos\\RavenFS\\RavenFS\\RavenFS\\Controllers\\FilesController.cs:line 130\r\n at RavenFS.Storage.TransactionalStorage.ExecuteBatch(Action1 action) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Storage\TransactionalStorage.cs:line 244\r\n at RavenFS.Storage.TransactionalStorage.Batch(Action1 action) in C:\\Projects\\HibernatingRhinos\\RavenFS\\RavenFS\\RavenFS\\Storage\\TransactionalStorage.cs:line 222\r\n at RavenFS.Controllers.FilesController.Put(String name) in C:\\Projects\\HibernatingRhinos\\RavenFS\\RavenFS\\RavenFS\\Controllers\\FilesController.cs:line 128\r\n at lambda_method(Closure , Object , Object[] )\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.Execute(HttpControllerContext controllerContext, IDictionary2 arguments)\r\n at System.Web.Http.Controllers.ApiControllerActionInvoker.<>c__DisplayClass2.b__0()\r\n at System.Threading.Tasks.TaskHelpers.RunSynchronously[TResult](Func1 func, CancellationToken cancellationToken)","InnerException":null} at RavenFS.Client.ExceptionExtensions.<TryThrowBetteError>b__0(Task task) at System.Threading.Tasks.ContinuationResultTaskFromTask1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.InvalidOperationException: System.Net.WebException: The remote server returned an error: NotFound.
at System.Net.Browser.ClientHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Func2.Invoke(T arg) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise) {"ExceptionType":"Microsoft.Isam.Esent.Interop.EsentWriteConflictException","Message":"Write lock failed due to outstanding write lock","StackTrace":" at Microsoft.Isam.Esent.Interop.Api.Check(Int32 err) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Api.cs:line 2739\r\n at Microsoft.Isam.Esent.Interop.Api.JetDelete(JET_SESID sesid, JET_TABLEID tableid) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Api.cs:line 2341\r\n at RavenFS.Storage.StorageActionsAccessor.Delete(String filename) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Storage\StorageActionsAccessor.cs:line 371\r\n at RavenFS.Controllers.FilesController.<>c__DisplayClass21.<Put>b__1c(StorageActionsAccessor accessor) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Controllers\FilesController.cs:line 130\r\n at RavenFS.Storage.TransactionalStorage.ExecuteBatch(Action1 action) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Storage\TransactionalStorage.cs:line 244\r\n at RavenFS.Storage.TransactionalStorage.Batch(Action1 action) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Storage\TransactionalStorage.cs:line 222\r\n at RavenFS.Controllers.FilesController.Put(String name) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Controllers\FilesController.cs:line 128\r\n at lambda_method(Closure , Object , Object[] )\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.Execute(HttpControllerContext controllerContext, IDictionary2 arguments)\r\n at System.Web.Http.Controllers.ApiControllerActionInvoker.<>c__DisplayClass2.b__0()\r\n at System.Threading.Tasks.TaskHelpers.RunSynchronously[TResult](Func1 func, CancellationToken cancellationToken)","InnerException":null} at RavenFS.Client.ExceptionExtensions.<TryThrowBetteError>b__0(Task task) at System.Threading.Tasks.ContinuationResultTaskFromTask1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()<---

While doing some concurrent operations with the same file Esent throws some exceptions i.e. EsentWriteConflictException. What kind of experience should we then provide user with? In RavenDB there is thrown ConcurrencyException in such cases - should we copy that way to handle that?

I spotted this issue when the server threw an exception (I think it was something like a SearcherClosed exception) during a search whilst it was also processing add file requests.

From the code, it looks like ReplaceSearcher is always called inside a lock, but use of the searcher isn't protected by a lock.

I can't consistently reproduce this, but I've seen an issue where, after I've deleted a file, the /search api will return a null value in the place where the file would have been in the list.

When I break in the SearchController, I can see that Search.Query is still returning the deleted file, but the call to accessor.ReadFile for that key returns null.

The following test fails:

  public class UploadingManySmallFiles : WebApiTest
  {
    [Fact]
    public void ShouldWork()
    {
        var client = NewClient();

        var tasks = Enumerable.Range(0, 100).Select(i =>
                                                       {
                                                           var ms = new MemoryStream(new byte[12]);
                                                           return client.UploadAsync(Path.GetRandomFileName(), ms);
                                                       }).ToList();

        var allTasks = TaskEx.WhenAll(tasks);

        allTasks.Wait();

        Assert.False(allTasks.IsFaulted);
    }
}

The exception is EsentKeyDuplicateException ("Illegal duplicate key") and the stack trace is

at Microsoft.Isam.Esent.Interop.Update.Save(Byte[] bookmark, Int32 bookmarkSize, ref Int32 actualBookmarkSize) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Update.cs: line 82
at RavenFS.Storage.StorageActionsAccessor.InsertPage(Byte[] buffer, Int32 size) in StorageActionsAccessor.cs: line 132
at RavenFS.Controllers.FilesController.ReadFileToDatabase.<>c__DisplayClass29.b__27(StorageActionsAccessor accessor) in FilesController.cs: line 202
at RavenFS.Storage.TransactionalStorage.ExecuteBatch(Action1 action) in TransactionalStorage.cs: line 244 at RavenFS.Storage.TransactionalStorage.Batch(Action1 action) in TransactionalStorage.cs: line 222
at RavenFS.Controllers.FilesController.ReadFileToDatabase.b__25(Task1 task) in FilesController.cs: line 200 at System.Threading.Tasks.Task1.InvokeFuture(Object futureAsObj)
at System.Threading.Tasks.Task.Execute()

I discovered this issue in the process of trying to reproduce another issue that happens when uploading lots of files through the UI, logged as Case #35.

If you try downloading a small file (< 100k or so), the browser seems to keep downloading and downloading. In Chrome the progress wheel just keeps going in a circle, rather than filling up, suggesting Chrome doesn't know how big the file is. I saw it reach 10s of MBs before I cancelled it.

Downloading larger files (a couple of MBs or so) works fine.

It doesnt work with >= 1MB file (1Kb file passed). Server throws following exception:

[ArgumentException: Value was invalid.]
   System.Security.Cryptography.HashAlgorithm.ComputeHash(Byte[] buffer, Int32 offset, Int32 count) +2822400
   RavenFS.Util.HashKey..ctor(Byte[] buffer, Int32 size) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Util\HashKey.cs:12
   RavenFS.Storage.StorageActionsAccessor.InsertPage(Byte[] buffer, Int32 size) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Storage\StorageActionsAccessor.cs:89
   RavenFS.Handlers.<>c__DisplayClass9.<Execute>b__7(StorageActionsAccessor accessor) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Handlers\SimplePutHandler.cs:81
   RavenFS.Storage.TransactionalStorage.ExecuteBatch(Action`1 action) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Storage\TransactionalStorage.cs:245
   RavenFS.Storage.TransactionalStorage.Batch(Action`1 action) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Storage\TransactionalStorage.cs:223
   RavenFS.Handlers.ReadFileToDatabase.<Execute>b__5(Task`1 task) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Handlers\SimplePutHandler.cs:79
   System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() +77
   System.Threading.Tasks.Task.Execute() +51

[AggregateException: One or more errors occurred.]
   System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) +3587209
   System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) +118
   System.Threading.Tasks.Task.Wait() +10
   RavenFS.Infrastructure.AbstractAsyncHandler.EndProcessRequest(IAsyncResult result) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Infrastructure\AbstractAsyncHandler.cs:121
   System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar) +89

Reading multipart message uses MultipartMemoryStreamProvider by default what makes the server throws OOM for the large requests. Instead of keeping parts in a memory we could use MultipartFileStreamProvider and then all parts are saved as files under indicated location - it should help to resolve OOM problem. Something what I think is worth to try is to write own MultipartStreamProvider which will transfer all parts directly into Esent and next if synchronization finish it will just reorder pages (actually reassign them) of the final synchronized file (will order transfered parts that are already in Esent and existing chunks of synchronized file that were not changed, and will remove old parts that has changed). So we can save the price of copying from file to Esent if synchronization finish. Ayende, what do you think about that?