New Features / Ideas about litedb

[mbdavid]

• Shrink database (from @lppanariello)
• Export database as JSON files: one JSON file per collection + 1 file per file storage file - all files in a zip stream
• Full text search index

[lppanariello]

👍 Shrink Database :)

PS: Another improvement could be the implementation of many Linq extended methods such as "OrderBy" or "Distinct/DistinctBy"

[darkl]

What about in memory database (without any backing file), like SQLite's option?

[izackp]

Ah lol. Fantastic, if you make it compatible you will probably get a much larger user base considering everything on unity either costs money or sucks (missing important features).

[podkolzzzin]

+1 Full text search index

Please support a lightweight key-value mode/construct

[Dessix]

Upsert would be pretty awesome.

[shycohen]

@mbdavid - sorry for the delay in my response. I think that more people know how to write thread-safe code than manual transactions, so I'd vote for transactions 😄

I would take ease of coding over a small performance hit - it's a really easy choice 😃 After all, we're talking a local database here, so expectations are set correctly.

Full text search is cool indeed, but might be a bit complex for most people. It also seems a bit of an overkill for LiteDB given the typical use case. I would prioritize it after "unlimited indexes".

[izackp]

Maybe instead of saving to a file after every update or insert you an add a SaveToPersistentStore() method. This way the developer can decide when to save in order to speed things up?

There's also a neat feature with CoreData for iOS i'd like to mention. CoreData has the concept of contexts (or isolated space). The developer can make a bunch of changes to it (insert, update, delete) then decide to commit it to a parent context (or a persistent store/file) or scrap all the changes. This also helps with concurrency, as you would create a child context for every thread to ensure separation as well access from only 1 thread.

Edit: no idea how complicated this would be. Just a suggestion. Another example of contexts (or isolated space): http://www.iboxdb.com/

Edit: I just tested a hacked up version of SaveToPersistentStore() with 1000 iterations of insert,update,insert,update,update and it went from 23 seconds to around 700ms

[jordansjones]

It would be great if support was added for the System.Runtime.Serialization data attributes.

At least

• DataContractAttribute
• DataMemberAttribute
• IgnoreDataMemberAttribute

[quietjoy]

Hello!

I want to use this project with asp.net vNext applications. Is anyone else working on this as well? Is anyone interested?

[Scramblejams]

How about a .Net 3.5 backport so that the library can be used with Unity3D?

[qart2003]

@mbdavid I read in #332 and ReadWriteLockSlim using in V3.
And thinking, probably will useful https://github.com/Wintellect/PowerThreading/blob/master/PowerThreading-Desktop/Wintellect.Threading/ResourceLocks/OneManyResourceLock.cs for LiteDB

[sheryever]

@mbdavid I am trying to port TransformedDirectory on dotnet.
Here is one written for Sql Server https://github.com/MahyTim/LuceneNetSqlDirectory. It basically stores the Lucene index files in the database. First I tried to write this for SQLite database but in SQLite blob are headache. So then I moved to write the TransformedDirectory in dotnet. It is better than the idea of SQLiteDirectory, because in SQLiteDirectory i was storing the index files in the database but not the index.

TransformedDirectory is written in Java and the author has used some complex synchronized thread safe code because of IO with performance. Some collections which has no any equivalent in C#.

Today, I will try your database for my application, if that worked, first i will write the initial LiteDBDirecotry cause I my SQLiteDirectory is almost done but i stuck with in writing chunks in Blob type. I think that will not be any issue with your database FileStorage.

[mbdavid]

@nwestfall, I think about this in early v1, when I convert direct POCO class into Binary data (v.0x used another FastBinaryJson classes). But then I made BsonDocument class based on BsonValue data types to restrict all .net types into few basic types. This BsonDocument are converted into byte[] using BSON format. I don't see benefits of support any another format (because this format must convert from BsonDocument). Also, in v5, I'm working now on partial deserialization from disk to support data transformation before load document. For this, BSON format also works nice, supporting skip fields and saving few (or lots) of bytes.

[adeel41]

+1 for Full text search index

[caverna]

+1 Export database as JSON

[deivisonrpg]

+1 Export database as JSON

[adeel41]

+1 for supporting linq extension methods

[mbdavid]

Hi @darkl, I though in this option too, but the main use, to me, is for testing. In real world, I don't know where to use. SQLite has more sense, because you can create in memory database to use SQL language to join data and fetch results. Do you have any example of use?

[darkl]

@mbdavid I thought about running lots of LINQ queries on a given dataset and using LiteDB in order to index the dataset to get faster queries.

[mbdavid]

@adeel41 @lppanariello: Linq is an important part because gets LiteDB more usable in .NET world. But to run any query command I need an index to make sense not use Linq of IEnumerable<T>. For example:

col.EnsureIndex(x => x.Age);

var result = col.Find(x => x.Age > 30) // Use Age index
    .Where(x => x.Name.Length > 10) // Do not use index, use Linq of IEnumerable<T>
                                    // on result of Find() command

// Do not use any index, use only Linq of IEnumerable
var result = col.FindAll() // Returns all documents
    .Where(x => x.Age > 30 && x.Name.Length > 10) 

Operations like OrderBy or Distinct used in IEnumerable will be as-fast-as internal implementation in LiteDB, because LiteDB is a local database. I always need fetch all documents (deserialize) and make the tests (to order, to distinct) like IEnumerable do.

Its different from mongodb when this operation will be executed in a server and returns a ready-to-use result.

[mbdavid]

If someone are interested in LiteDB next versions ideas, this is project trello board: https://trello.com/b/SxGATTlM

[mykemaas]

Would like the see querying against arrays. For example, in MongoDB, I can have a collection of "Contact" objects with an Addresses array. A sample Contact object would look like this:

   {
       "_id" : "1233123",
       "FirstName":"Bonnie",
       "LastName":"Palmer",
       "Addresses": [
            {
                "Street":"123 Scott Ave",
                "City":"Indianapolis",
                "State":"Indiana",
                "Zip":"46239"
            },
            {
                "Street":"323 Paget",
                "City":"Richmond",
                "State":"Virginia",
                "Zip":"23208" 
           }
        ]
    }

I could then query to get all contacts that have an address in Oregon like this:

collection.Find(Query.EQ("Addresses.State", "Oregon"));

It would be really cool if LiteDB supported this type of query as well!

[shycohen]

More indexes per collection. 16 will be too restrictive since we want to enable searching over multiple document properties (first name, last name, company, account number, phone number, invoice number, paid status, item code, etc.).

[mbdavid]

Hi @shycohen. What is your ideia in numbers? I fixed in 16 because I need reserve, for each document, all 16 spaces (96 bytes). In a 4096 byes for each page, 96 bytes is a quite big number. I'm testing LiteDB with 8192 block page and I can increse this number in y tests.

[shycohen]

@mbdavid, do you keep the 96 bytes per document? This could lead to significant waste if the documents are small. Ideally, the indexes would be kept outside of the document so that there's no waste or limit (yes, I understand that this may come on the expense of performance if you need to touch 2 or more pages for each operation).

[VoronFX]

Hi!
What about transactions in future?
It's very important feature, sadly you've removed it from v2.

[mbdavid]

Hi guys!

@VoronFX, transaction was removed to when I figure out there is no databases with transactions (no-sql databases). All databases are only "document transaction". In v2 I improve cache control, so you can have massive data inserts/update/deletes without memory overflow (using journal file disk). But now, only one operation at time.

@Dessix, It´s on my list, maybe in v2.1... I need study more string expression parsers for that.

@Scramblejams, initial LiteDB was for 3.5, because LiteDB doesnt need any v4 framework. Was updated to use with async Task<> operation, but I never did.. I have a branch to works with PCL .NET (under development yet).

[shycohen]

Transactions were a cool differentiating feature, and I believe that the introduction of JSON support to "traditional" (RDBMS) databases will eventually push the NoSQL databases to offer similar features.

That said, @VoronFX you can implement transactions manually using a transaction document. I have attached an example of how to do that.

Manual Transactions.pdf

@mbdavid, any new thoughts about "unlimited" indexes?

[mbdavid]

Hi @shycohen, maybe transaction can back in v2.1. Using current implementation I need to choice: transactions or thread safe?

But about unlimted indexes its a problem much more complex. I think in some solution, but no one was fast as current solution. I will need more Page Types to link a index key to document position. This will change all data structure.

What do you thing about Lucene/FullTextSearch? I have plans, in future, to create full text seach (hard job, but cool) or/and a Lucene directory implementation (write Lucene files into LiteDB).

[negue]

@izackp do you have an example or a pullrequest for this SaveToPersistentStore ?

[shakram02]

@mbdavid Hey, I want to add something like this, it came across while I was playing with liteDb

var custs = db.GetCollection(nameof"Customerbson");
                var customer = new BsonDocument
                {
                    ["Name"] = "John Doe",
                    ["Phones"] = "00",
                    ["Att"] = 1
                };

                var bsonVal = custs.Insert(customer);
                var val = custs.FindOne(x => x["Name"] == "Jhon Doe");

What do you think about it ?
I sort of have a conceptual idea of what I should do and I'm willing to contribute to this awesome project, how can I contact you -in case some of the code gets me stuck-? my email is [email protected]
Thanks,

[izackp]

@negue
look at my fork of litedb. Specifically:
https://github.com/izackp/LiteDB/blob/master/LiteDB/DbEngine/DbEngine.cs
https://github.com/izackp/LiteDB/blob/master/LiteDB/DbEngine/Services/TransactionService.cs
https://github.com/izackp/LiteDB/blob/master/LiteDB/Core/LiteDatabase.cs

Please note that I originally made these changes before TransactionService started keeping a stack of transactions. My changes will probably negate whatever feature that intended to add.

[mbdavid]

Hi @shakram02, nice code. My plans is suppor some like that, using BsonDocument inherit IDictionary to use:

var customer = new Customer
{
    { "Name", "John" },
    { "Age", 35 }
}

This is the some that MongoDB api uses. But support Linq using BsonDocument is a little bit more difficult.

[shakram02]

Yes, I dug in the code, and realized that I have to make a new query class that will query the keys first and then find the desired property from the keys

[tolstenko]

Unity3D Support! Please! Your project is a life saver!
In order to achieve that, emit mustnt be used. Unity build target is almost 3.5(almost all features present on 3.5 is supported), but it ensures that it is 2.0 compilant.

[izackp]

@tolstenko My fork currently works in unity (https://github.com/izackp/LiteDB) I have yet to bring it up to speed with main repo. Use at your own risk.

[mbdavid]

Hi @izackp, what changes must be done to be compatible with Unity? Must be a different target framework? Is not only refer net35 dll?

[izackp]

@mbdavid I just directly embed the code into my unity project. I pretty much just removed any async, tasks, UWP, or Windows Store stuff. I changed everything to use bit converter, but I don't know for sure if that was necessary. Here's the commit: izackp@58f81a8

There are a few things here and there that I had to change. For example I had to use .ToString() instead nameof().

[mbdavid]

Yes, it's easy... I will update all source to compile native with Unity. By the way, this ToString are wrong because the test is if(propertyInfo == null) ... to you cann't use propertyInfo.ToString(). In this case, just use string "propertyInfo"

[mindryu]

Data Change Notification Callbacks!!

+1 Export database as JSON

[DeanVanGreunen]

custom plugin for search? as I'll be adding Plugin class/instance to litedb v3+

…

On Jan 12, 2017 4:00 PM, "Podkolzin" ***@***.***> wrote: +1 Full text search index — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#20 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVQ2C4el228M_yhTiuLtZPeRNegjjQ3Vks5rRjH1gaJpZM4D3vSJ> .

[qart2003]

Hi @mbdavid Wil you use for full text search index Lucene inside?

[mbdavid]

@qart2003, that was my idea. I already used Lucene in a project and it´s fantastic tool. So, I tried to implement custom directory to store lucene files inside LiteDB. But I didnt found any example to do.

[qart2003]

@mbdavid I thinking need using RamDirectory-Lucene and try to access it then/before store/restore in LiteDB
Some methods for store/restore index to/from LiteDB. But I dont know how to implement mapping of any properties of classes, probably by attributes

[mbdavid]

Hi @sheryever, nice one. I never saw this SQL implementation of Lucene directory. It's really awesome. I always try found some Lucene directory implementation but never found. You can use FileStorage to read/write large streams (in v3 supports OpenRead and OpenWrite streams). If you starts you LiteDBDirectory here in Github, let me know to follow your code.

[sheryever]

@mbdavid I am working on it at this time, once got a result I will upload on github

[sheryever]

@mbdavid I have uploaded LiteDbDirectory.
It is an initial commit, At this time, It indexing 10,000 documents and after it hangs on IndexWriter.Dispose(), I waited till 30 minutes no response, while in SqlServerDirectory it is giving the same behavior after 101,000 documents.

I tried my best to find the issue but couldn't found any glitch in my code. I have also setup lucenenet source code in my machine, to debug and find the solution.

IndexSearcher is also taking about 15sec in initialization. but if we first index and then Initialize IndexSearcher then it takes almost 31sec.

If it's is possible then please provide the inner MemoryStream of LiteFileStream, at this time first i need to open the file in read / write mode and then copy the complete LiteFileStream bytes into the MemoryStream before any read / write.

Take a look, The main read / write is happening in LiteDbStreamingReader and LiteDbStreamingWriter.

[sheryever]

@mbdavid I asked @MahyTim for update in SQLDirectory, He updated his repo and improving that more. I also following him.

Now LiteDBDirectory is also indexing 101,000 document :) Hope we will improved more performance.

On the other side i am also reading the lucenenet code and thinking about saving only the document in the LiteDb collection as BsonObject instead of saving the index files. It will be better to read and write the document only in the db instead of writing and reading (single byte at a time) files in the database, which is also confusing me now.

[nictaylr]

@mbdavid possibility for server version?
then we share database over socket on local system, make administration easier, very useful
thank very much

[yar3333]

@mbdavid Memory-only version of LiteDB can be used as a very good cache system with a linq support. I spend several hours, but not found a good alternative. SQLite is not comfortable at all (overkill for such tasks, many restrictions to code).

[podkolzzzin]

@yar3333 and what would be advantages of in-memory litedb in comparison with System.Collections? And Linq2Objects?

[yar3333]

@podkolzzzin Indexes. Linq does not use indexes even for sorted collections like SortedList.

[podkolzzzin]

@yar3333 understand what you are talking about.
But as for me, for in-memory in-process caches .net collections, if chosen correctly, works really good.
If we imagine that in-process litedb exists, it will be less effective than collections because of serialization and deserialization to bson.
It would be interesting to research what projects exists in this sphere.

[Jacknq]

Ef core or linq2db driver would be really awesome, they dont support nosqls yet.
So it would bring more possibilities to lots of people. Dynamic queries in linq covered.

[nwestfall]

@mbdavid has it ever come up to support other data parsing libraries instead of just BSON? Whether it be JSON or XML or even better into Bond or Protobuf? Is this something you'd consider if someone did the work of interfacing it out so other implementations could be supported?

[andrewvk]

Async support?

[qart2003]

Async support?

Idea: https://forums.xamarin.com/discussion/150462/repository-pattern-with-litedb-pattern-improvement

[andrewvk]

It’s just imitation, not real support.

[qart2003]

It’s just imitation, not real support.

Yes. I think the real support is impossible cos LiteDB is a file. It means FileStream used inside which doesnt support async. Jeffrey Richter described it in books C# via CLR... (https://docs.microsoft.com/en-us/dotnet/api/system.io.filestream.readasync?view=netframework-4.8
FileStream.ReadAsync is imitation by FileStream - Jeffrey Richter described it)

[andrewvk]

Look here - https://referencesource.microsoft.com/mscorlib/system/io/filestream.cs.html#157c7e73144c9238
Read/WriteAsync use overlapped IO inside, not worker threads from pool.

[qart2003]

Look here - https://referencesource.microsoft.com/mscorlib/system/io/filestream.cs.html#157c7e73144c9238
Read/WriteAsync use overlapped IO inside, not worker threads from pool.

The main call is _handle = Win32Native.SafeCreateFile(tempPath, fAccess, share, secAttrs, mode, flagsAndAttributes, IntPtr.Zero); where flagsAndAttributes is Asynchronous = unchecked((int)0x40000000), // FILE_FLAG_OVERLAPPED FROM https://referencesource.microsoft.com/#mscorlib/system/io/fileoptions.cs,4d439b7469a1b468
I'm thinking Asynchronous is better but some ntfs driver funcs works in sync mode only therefore probably async support for LiteDB is not very important except case when WOW LiteDB support async too! (We are not the first but we are the champions!) And using two FileStream for sync and async modes per file is not good idea as for me cos a little bit win for a very difficult code which will depend from .net inside.

[lbnascimento]

Hi! With the objective of organizing our issues, we are closing old unsolved issues. Please check the latest version of LiteDB and open a new issue if your problem/question/suggestion still applies. Thanks!