Improve performance for large note collections (preventing lags and loss of access for seconds at scale) about tomboy-ng

[davidbannon]

Hi kernel, v0.14 is faster than previous but the improvements are mainly around indexing and searching rather than opening. I am not happy with the -ng's speed when editing a large note in particular and rather fear there may not be anything I can do about that. So, be interesting to hear your feedback.
If you don't already have gtk libaries installed, you will need them, I have not had much to do with Gentoo for a long time so, again, your feed back ....

[davidbannon]

Yep, no doubt you have that right Benjamin !
I do have some experimental code that resets a timer every time you type something. If the timer does, in fact, time out, its because you are having a think and I can then dive in and do the housekeeping. I was a little disappointed in the level of improvement it delivered however. As a matter of interest, try turning off "display links" from the settings form. That might give you some idea of the improvement possible. The rest of the delay, if any, is right there in KMemo, and I'm afraid thats a black box.
I do most of my development (and other stuff) on a i3 celeron, spinning disk and 4gRam so a lot less of a machine than you are using. But I don't use or even like the links popping up so run with them off all the time and like that, its fine.
I'll look further into that experimental code. Git will find a way back for me if needed.
Indexes - I keep a data structure in RAM with the names and IDs of all notes (not the content). That should allow me to check words for links quite fast, any saving delivered by a database would be at startup only. So I don't think that would help with your immediate problem.
Mind you, I have only tested with my 200 odd notes, not your 16K. So, maybe its time I looked at how I search that internal list ? Right now its just a for loop asking each time does something match something. A BTree or whatever could help there too.

[mlncn]

I think keeping the data structure of all notes in RAM is not going to scale for my needs (although i am looking for a computer which supports at least 32GB RAM to replace my seven year old machine...)

So i should disable "Show Internal Links" ?

Yes, if i turn that off there's an immediate improvement, showing text as i type it initially, but pretty quickly it reverted to high CPU usage and delay. Typing the below at about 50 words per minute it showed about the first 50 characters i typed, but then took 15-20 seconds to show the rest.

So there's definitely an improvement if it's not trying to show internal links (and i wouldn't want these to not be shown at all, but i'd be fine if it only did it when opening a note or when manually requested, though personally i like the serendipity of immediately learning when i'm referencing a person, place, or topic i already have a whole note about!)

But it seems there's something else slowing trying to consult or just keep that memory alive that significantly slows Tomboy NG in regular usage (typing in an existing note, pasting in text into a new note).

[davidbannon]

Hmm, with Links off the other housekeeping things are pretty quick. And you are running a pretty gutsy machine. I suspect any delays we are seeing now are the underlying KMemo component, thats really a black box. At some stage it may be appropriate to go back there and do some profiling and see if the author is interested in removing some aspects, making a stripped down version but I am not overly confident that will work.
I do find performance on my slowish laptop acceptable. On an old Atom powered netbook, I must turn off links and even then its painful to use. But so is Tomboy ....
One problem is I cannot, in my wildest dreams, type at 50wpm ...
D

[davidbannon]

OK, I have checked in code that sets or resets a timer every time the user makes a change. Thats set to 4 seconds right now and I thinks thats a bit disconcerting so I think its going to need some experimenting to determine a good value. Similarly, another one controls the autosave at 10 seconds. Its now setup to also be restarted when a user types something. But it does deliver an improved responsiveness, not a huge change but a worthwhile one. 3ce5f08
I can still saturate a core on my laptop typing quickly.

[kernelOfTruth]

@davidbannon thanks for your work !

I also got quite a note collection, carried it over to gnote (cherrytree had issues with import so would have to be imported manually)

and it usually takes 30 seconds or several minutes to open a new note, edit & search files

tomboy was way faster with recent mono releases, got almost no lags.

Unfortunately support/maintainership got ended on Gentoo

Looking forward to trying out tomboy-ng in the near future

[kernelOfTruth]

@davidbannon performance appears to be speeding fast,

searching works really quickly in a few moments,

haven't tried editing, etc. so far yet.

I've only copied a rather big note from gnote to tomboy-ng and it was instantly there (so far formatting doesn't work)

on gnote this takes ages (30 seconds to several minutes usually)

[erik-brangs]

For my note collection (~ 1800 notes), search is very fast on Linux with 0.18b (feels like the results appear instantly). However, on Windows, searches always lead to a lockup for a few seconds when searching for a term for the first time. Subsequent searches on Windows are fast enough.

[davidbannon]

Hmm, I wonder why ? Honestly, I have not done a lot performance testing on windows, just assumed what worked well on Linu would also work well on Win/Mac. A bad assumption.

I'll see if I can mock up a windows test rig when I get home again. Don't have one right now, traveling in Canada....
Davo

[davidbannon]

I have not been able to reproduce this to be honest. One problem I have is that the only windows platform I have access to is a fairly gutsy desktop with a everything on an SSD so differences in first and subsequent searches is imperceptible. And only about 200 notes.

Just to clarify, by 'subsequent searches' you mean a second search for a different term ?
If so, I have to assume what we are seeing is just windows initial read of the relevant files, after the first search, they are all in the disk cache and so much faster. Maybe a fast computer with lots of memory but a relatively slow hard disk, or a hard disk sharing a channel with some other slow hardware (such as a CDRom drive) ?

Davo

[erik-brangs]

Yes, by 'subsequent searches' I mean a second search for a different term.

For me, the first search on Windows on a SSD is slower than on Linux on a normal HDD. I'm using version 0.19. I'm seeing similar lockups (tomboy-ng doesn't react for a few seconds) when doing a sync on Windows. For searches, lockups also seem to occur for the first search after I haven't used the computer for a while.

[davidbannon]

Yes, I'm afraid that sounds like disk access time to me. To search the notes, they need to be read from disk, once read, they will remain in the disk cache for a while so second and subsequent searches will be faster. I'd expect an SSD to be quicker than you describe but I guess we have to assume thats how it is. The sync operation is similar so not surprising to see same problem. Do you see the same thing as tomboy-ng first opens up ? We read the same notes then too.

I could, perhaps, make up a special version that records some numbers of the various aspects of that search if you are interested in that level of debugging ? It might reveal something we can fine tune ....

How big is your notes dir ? (on linux, if in default dir du -sh ~/.local/share/tomboy-ng )
Davo

[erik-brangs]

Yes, there's also a delay when opening up tomboy-ng for the first time. My notes dir is 8,5 MB.

I suppose I could investigate where there are any disk optimziations that could be done on the SSD.

[davidbannon]

Yep Erik, I suspect you have a disk i/o issue on the windows box. I just made a tool to generate a whole lot of notes using my existing notes as templates. I made 1800 notes, they turned out to take up about 12Meg so, my (real) notes must be a bit longer on average than yours.
I then timed a couple of searches of that notes dir on my windows box, every run, including the first was about 360mS.
Thats on an i5, 8Gram, 256G SSD, Windows 10 box.
Davo

[davidbannon]

Erik, probably time I closed this issue.
I have tweaked the way it works about as far as I can to speed things up. I believe tomboy-ng is quite a lot faster than Tomboy in a number of aspects.
Thanks for your input.