hibernating-rhinos / ravenfs Goto Github PK
View Code? Open in Web Editor NEWHome Page: http://hibernatingrhinos.com
Home Page: http://hibernatingrhinos.com
At present, trying to upload a file larger than about 80 - 90Mb from the UI fails, with an out of memory exception being reported.
Presumably we need to rework UploadAsync to use explicit chunked uploading when called from Silverlight?
StorageSignatureRepository shouldn't be singleton because it have to manage with temporal files during its life. Having singleton we may accidentally remove files from other processes.
ISignatureRepository should be created for one file only.
throws a stack overflow exeption.
When I select a lot of files (around 200 in this case) to upload in one go through the U I quite often see some of the files fail to upload, with this exception having been thrown.
System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: System.Net.WebException: The remote server returned an error: NotFound.
at System.Net.Browser.ClientHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Func2.Invoke(T arg) at System.Threading.Tasks.TaskFactory
1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action
1 endAction, Task1 promise) {"ExceptionType":"Microsoft.Isam.Esent.Interop.EsentWriteConflictException","Message":"Write lock failed due to outstanding write lock","StackTrace":" at Microsoft.Isam.Esent.Interop.Api.Check(Int32 err) in C:\\Work\\ravendb\\SharedLibs\\Sources\\managedesent-61618\\EsentInterop\\Api.cs:line 2739\r\n at Microsoft.Isam.Esent.Interop.Api.JetDelete(JET_SESID sesid, JET_TABLEID tableid) in C:\\Work\\ravendb\\SharedLibs\\Sources\\managedesent-61618\\EsentInterop\\Api.cs:line 2341\r\n at RavenFS.Storage.StorageActionsAccessor.Delete(String filename) in C:\\Projects\\HibernatingRhinos\\RavenFS\\RavenFS\\RavenFS\\Storage\\StorageActionsAccessor.cs:line 371\r\n at RavenFS.Controllers.FilesController.<>c__DisplayClass21.<Put>b__1c(StorageActionsAccessor accessor) in C:\\Projects\\HibernatingRhinos\\RavenFS\\RavenFS\\RavenFS\\Controllers\\FilesController.cs:line 130\r\n at RavenFS.Storage.TransactionalStorage.ExecuteBatch(Action
1 action) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Storage\TransactionalStorage.cs:line 244\r\n at RavenFS.Storage.TransactionalStorage.Batch(Action1 action) in C:\\Projects\\HibernatingRhinos\\RavenFS\\RavenFS\\RavenFS\\Storage\\TransactionalStorage.cs:line 222\r\n at RavenFS.Controllers.FilesController.Put(String name) in C:\\Projects\\HibernatingRhinos\\RavenFS\\RavenFS\\RavenFS\\Controllers\\FilesController.cs:line 128\r\n at lambda_method(Closure , Object , Object[] )\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.Execute(HttpControllerContext controllerContext, IDictionary
2 arguments)\r\n at System.Web.Http.Controllers.ApiControllerActionInvoker.<>c__DisplayClass2.b__0()\r\n at System.Threading.Tasks.TaskHelpers.RunSynchronously[TResult](Func1 func, CancellationToken cancellationToken)","InnerException":null} at RavenFS.Client.ExceptionExtensions.<TryThrowBetteError>b__0(Task task) at System.Threading.Tasks.ContinuationResultTaskFromTask
1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.InvalidOperationException: System.Net.WebException: The remote server returned an error: NotFound.
at System.Net.Browser.ClientHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Func2.Invoke(T arg) at System.Threading.Tasks.TaskFactory
1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action
1 endAction, Task1 promise) {"ExceptionType":"Microsoft.Isam.Esent.Interop.EsentWriteConflictException","Message":"Write lock failed due to outstanding write lock","StackTrace":" at Microsoft.Isam.Esent.Interop.Api.Check(Int32 err) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Api.cs:line 2739\r\n at Microsoft.Isam.Esent.Interop.Api.JetDelete(JET_SESID sesid, JET_TABLEID tableid) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Api.cs:line 2341\r\n at RavenFS.Storage.StorageActionsAccessor.Delete(String filename) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Storage\StorageActionsAccessor.cs:line 371\r\n at RavenFS.Controllers.FilesController.<>c__DisplayClass21.<Put>b__1c(StorageActionsAccessor accessor) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Controllers\FilesController.cs:line 130\r\n at RavenFS.Storage.TransactionalStorage.ExecuteBatch(Action
1 action) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Storage\TransactionalStorage.cs:line 244\r\n at RavenFS.Storage.TransactionalStorage.Batch(Action1 action) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Storage\TransactionalStorage.cs:line 222\r\n at RavenFS.Controllers.FilesController.Put(String name) in C:\Projects\HibernatingRhinos\RavenFS\RavenFS\RavenFS\Controllers\FilesController.cs:line 128\r\n at lambda_method(Closure , Object , Object[] )\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.Execute(HttpControllerContext controllerContext, IDictionary
2 arguments)\r\n at System.Web.Http.Controllers.ApiControllerActionInvoker.<>c__DisplayClass2.b__0()\r\n at System.Threading.Tasks.TaskHelpers.RunSynchronously[TResult](Func1 func, CancellationToken cancellationToken)","InnerException":null} at RavenFS.Client.ExceptionExtensions.<TryThrowBetteError>b__0(Task task) at System.Threading.Tasks.ContinuationResultTaskFromTask
1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()<---
While doing some concurrent operations with the same file Esent throws some exceptions i.e. EsentWriteConflictException. What kind of experience should we then provide user with? In RavenDB there is thrown ConcurrencyException in such cases - should we copy that way to handle that?
I spotted this issue when the server threw an exception (I think it was something like a SearcherClosed exception) during a search whilst it was also processing add file requests.
From the code, it looks like ReplaceSearcher is always called inside a lock, but use of the searcher isn't protected by a lock.
I can't consistently reproduce this, but I've seen an issue where, after I've deleted a file, the /search api will return a null value in the place where the file would have been in the list.
When I break in the SearchController, I can see that Search.Query is still returning the deleted file, but the call to accessor.ReadFile for that key returns null.
The following test fails:
public class UploadingManySmallFiles : WebApiTest
{
[Fact]
public void ShouldWork()
{
var client = NewClient();
var tasks = Enumerable.Range(0, 100).Select(i =>
{
var ms = new MemoryStream(new byte[12]);
return client.UploadAsync(Path.GetRandomFileName(), ms);
}).ToList();
var allTasks = TaskEx.WhenAll(tasks);
allTasks.Wait();
Assert.False(allTasks.IsFaulted);
}
}
The exception is EsentKeyDuplicateException ("Illegal duplicate key") and the stack trace is
at Microsoft.Isam.Esent.Interop.Update.Save(Byte[] bookmark, Int32 bookmarkSize, ref Int32 actualBookmarkSize) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Update.cs: line 82
at RavenFS.Storage.StorageActionsAccessor.InsertPage(Byte[] buffer, Int32 size) in StorageActionsAccessor.cs: line 132
at RavenFS.Controllers.FilesController.ReadFileToDatabase.<>c__DisplayClass29.b__27(StorageActionsAccessor accessor) in FilesController.cs: line 202
at RavenFS.Storage.TransactionalStorage.ExecuteBatch(Action1 action) in TransactionalStorage.cs: line 244 at RavenFS.Storage.TransactionalStorage.Batch(Action
1 action) in TransactionalStorage.cs: line 222
at RavenFS.Controllers.FilesController.ReadFileToDatabase.b__25(Task1 task) in FilesController.cs: line 200 at System.Threading.Tasks.Task
1.InvokeFuture(Object futureAsObj)
at System.Threading.Tasks.Task.Execute()
I discovered this issue in the process of trying to reproduce another issue that happens when uploading lots of files through the UI, logged as Case #35.
If you try downloading a small file (< 100k or so), the browser seems to keep downloading and downloading. In Chrome the progress wheel just keeps going in a circle, rather than filling up, suggesting Chrome doesn't know how big the file is. I saw it reach 10s of MBs before I cancelled it.
Downloading larger files (a couple of MBs or so) works fine.
It doesnt work with >= 1MB file (1Kb file passed). Server throws following exception:
[ArgumentException: Value was invalid.] System.Security.Cryptography.HashAlgorithm.ComputeHash(Byte[] buffer, Int32 offset, Int32 count) +2822400 RavenFS.Util.HashKey..ctor(Byte[] buffer, Int32 size) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Util\HashKey.cs:12 RavenFS.Storage.StorageActionsAccessor.InsertPage(Byte[] buffer, Int32 size) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Storage\StorageActionsAccessor.cs:89 RavenFS.Handlers.<>c__DisplayClass9.<Execute>b__7(StorageActionsAccessor accessor) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Handlers\SimplePutHandler.cs:81 RavenFS.Storage.TransactionalStorage.ExecuteBatch(Action`1 action) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Storage\TransactionalStorage.cs:245 RavenFS.Storage.TransactionalStorage.Batch(Action`1 action) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Storage\TransactionalStorage.cs:223 RavenFS.Handlers.ReadFileToDatabase.<Execute>b__5(Task`1 task) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Handlers\SimplePutHandler.cs:79 System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() +77 System.Threading.Tasks.Task.Execute() +51 [AggregateException: One or more errors occurred.] System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) +3587209 System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) +118 System.Threading.Tasks.Task.Wait() +10 RavenFS.Infrastructure.AbstractAsyncHandler.EndProcessRequest(IAsyncResult result) in C:\workspaces\NetProjects\rhinos\RavenFS\RavenFS\Infrastructure\AbstractAsyncHandler.cs:121 System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar) +89
Reading multipart message uses MultipartMemoryStreamProvider by default what makes the server throws OOM for the large requests. Instead of keeping parts in a memory we could use MultipartFileStreamProvider and then all parts are saved as files under indicated location - it should help to resolve OOM problem. Something what I think is worth to try is to write own MultipartStreamProvider which will transfer all parts directly into Esent and next if synchronization finish it will just reorder pages (actually reassign them) of the final synchronized file (will order transfered parts that are already in Esent and existing chunks of synchronized file that were not changed, and will remove old parts that has changed). So we can save the price of copying from file to Esent if synchronization finish. Ayende, what do you think about that?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.