Trying to test for large data sets and I get failures about litedb

[mbdavid]

Did you have error message exception? There is no 4GB limit on database size.

[wcmiller3]

I've repeated it a few times. I've set the sample customer program to add 3 million records. And I run it again. By the 3rd time, I receive a NotImplementedException during an insert.

On Aug 14, 2015, at 6:48 PM, Mauricio David [email protected] wrote:

Did you have error message exception? There is no 4GB limit on database size.

—
Reply to this email directly or view it on GitHub.

[wcmiller3]

I'm seeing great potential but I really have a need for larger data sets. It failed much earlier with the app set to 32 bit.

On Aug 14, 2015, at 6:48 PM, Mauricio David [email protected] wrote:

Did you have error message exception? There is no 4GB limit on database size.

—
Reply to this email directly or view it on GitHub.

[sherry-ummen]

I can confirm it happens for me too.

[mbdavid]

Hi guys! Yes, I do this test and fails to me too. I don't know why yet, but I will investigate this weekend. I guess can be something in FileStream or like that.

[mbdavid]

Hi @wcmiller3 and @sherry-ummen, bug fund! I simple uint<->long convert error. I will create a new release for this

[sherry-ummen]

ok I tested this. One thing is strange which I am not sure whether its a bug or some OS feature.

so my program is simple

 foreach (var index in Enumerable.Range(1, 10000)) {
                    var emp = new Employee() {
                        EmployeeID = "owsddyu" + index,
                        FirstName = "one",
                        LastName = "two",
                        Bytearray = new byte[1 * 1000 * 1000]
                    };
                         col.Insert(emp);
                  }

So when its running you can check in the folder that the file size is increasing but after it reached the 4Gb then the updating of the file stops and when you stop the execution then you see the final size which is more that 4Gb depending on for how much time you keep it running.

My doubt is that why after 4Gb it stops showing the file size changing? And when the program is stopped then only it updates the size. Whats going on?

Thanks

[mbdavid]

Hi @sherry-ummen, do not use Insert method to bulk many data. Take a look on new unit test: https://github.com/mbdavid/LiteDB/blob/master/UnitTest/BigFileTest.cs

I'm using Windows 10 x64 and updates on file explorer are normal after 4gb. In this example, grows in ~500mb each update.

[sherry-ummen]

Hi, yes I checked the unit test and actually ran that also and there it worked. About using of Insert the thing which I am trying to check here is that it may happen that there may be random read and write but there may be huge number of writes so I really cannot determine when to use Insertbulk.

Could you give a better solution then? Thanks