Opening new My Tracks very slow with larger number of tracks about osmand

[sonora]

Small success story here: After quite some work on streamlining the code, removing recursions, and removing surplus track and folder analysis calculations, @Chumva and I were able to reduce the time it takes to load the 'MyPlaces>Tracks' screen by an order of magnitude. On my setup from once ~40 sec to now <3 sec.

And also the logcats now reflect significant improvement:

05-16 01:34:39.260 31493 32007 I net.osmand: GpxDbHelper Time to loadGpxItems 4173 ms, 3445 items {Initializing app}
05-16 01:34:39.288 31493 32007 I net.osmand: GpxDbHelper Time to loadGpxDirItems 28 ms items count 3445 {Initializing app}
05-16 01:34:39.288 31493 32007 I System.out: Initialized GPX_DB_INITIALIZED in 4201 ms
05-16 01:34:39.367 31493 32007 I System.out: Initialized POI_FILTERS_INITIALIZED in 79 ms

Still, 3sec feels a bit 'laggyy', of course, that's what we still have #19683 for, I guess.

[pebogufi]

17257 -> vshcherb closed this as completed on May 29.
:-(

[Corwin-Kh]

sorry for link - turn put it's closed. 1994f23
this one should work. The case i'm talking about is fixed in this pull request. It is already in last builds.
So, if it is still present then problem somewhere else. we'll try to find out.

[vshcherb]

Refactoring is needed with GPX Database.

1. We need to generate all long SQL queries from Enum of GPX characterestics and GPX setting.
2. We shouldn't read file system list and only get track items GPXDatabase. GPXDatabase should be updated in background first time My Places open or Configure Map > Tracks. On Start we should just read GPXDatabase.
3. Filters code should be simplified as we shouldn't have separate classes for filters - they don't require any specific functions. Instead we should have Conversion function (1 per unit), GPXDatabase should provide generic API to retrieve min/max field from database.
4. GPXDatabase should have automatic regeneration code for characteristic when new characterestic is added or when schema is changed. For that purpose we can store timestamp of generation. Scheme will be defined simply as date (in Android there is API to check existing column).

[sonora]

I wonder how we did it with the old, expandable list view. That one was comparatively lightning- fast...

[sonora]

PS: When I start the app specifically to check out a track, i.e. when after a fresh app start and immediately after the map becomes visible (but presumably the app is still busy with a lot of other initialization tasks), I pull up the My Tracks screen, that takes up to 40 seconds of watching the spinner.

[Chumva]

Work is still in progress, probably this week there will be UI fixes that should improve speed with my places - tracks screen

[sonora]

Yes much better. Roughly halves the times to fully display the My Places > Tracks and Configure map > Tracks screens.

But I also managed to now produce the enclosed crash:
exception.log.txt.

PS: It's a bit annoying that there is a "pull-down refresh": Every time I scroll the track folder list back to the top I accidentally r'efresh' the screen (which serves no purpose) and again have to wait for 6 seconds watching the spinner until the list screen becomes responsive again.

[sonora]

Is there any activity and hope to still fix this regression?

If not I will have to start compiling and publishing some instructions to offload tracks, so that OsmAnd still remains usable for those users of my scientific community who conventionally have several thousand track files in their folder structure and the app keeps stalling all the time.

[sonora]

@vshcherb Here are some benchmark figures for the time it takes to display the Tracks screen (or again for the Visible tracks screen):

• 3500 tracks: 12 - 14 sec
• 2500 tracks: 7 - 8 sec
• 1500 tracls: 3 sec

So the time to display the screen seems to grow overproportionately with the number of total tracks present in OsmAnd's track folder. This is e.g. typical of sorting mechanisms. But in any case the code strategy seems flawed in the following aspects:

(1) The time to open the Tracks screen should not depend on the total number of tracks in all folders, but merely on the number of items to be shown on the screen to be opened. (In my case simply 70 subfolders and zero tracks on the top level screen)

(2) Similarly, there should not have to be another wait time (and every time) when going from the MyPlaces/Tracks screen to the "Visible Tracks" screen. It seems that latest at this time there should already be a table of all visible tracks present in memory, rather than going through the long wait time again (it seems like the complete flat list of all tracks is searched on every occasion?)

(3) I suggest to disable the manual pull-down list-refresh functionality on these screens. The list of tracks should not change while users are on these screens, so a pull-down refresh makes no sense (but causes 12 sec delays for me time and time again when I scroll to the top of the list).

Initially I thought we may simply have to introduce some caching here of whatever the code produces during these long wait times, so they are not needed upon every (even repeated) opening of the screens. But now I am fairly convinced the code strategy is flawed: Whatever the app keeps doing on these occasions should not be needed via such costly operations at all, for the simple purpose of producing a mostly rather short inventory list on a dialog.

-> I would appreciate if whoever implemented the "New Tracks screen" code could take a look at these findings, and at least explain what the code does during these wait periods, and why that would be required upon every opening of the screen or changing between them?

[sonora]

@Chumva I believe that this is the culprit:

OsmAnd/OsmAnd/src/net/osmand/plus/myplaces/tracks/TrackFoldersHelper.java

Line 124 in a763ecc

public void reloadTracks() {

and it leads to this

OsmAnd/OsmAnd/src/net/osmand/plus/configmap/tracks/TrackFolderLoaderTask.java

Line 82 in a763ecc

File[] files = GPXFolderUtils.listFilesSorted(sortByMode, folderFile);

where the problem may be: We always sort the entire list of all files recursively (regardless of which folder they are in), which is a very costly operation, and probably totally unnecessary? Because almost never do we display the entire flat list, so the sorting should not be applied in the beginning to everything when cataloging all file/folder items, but in the end and only to the much fewer files / folders which are to be shown on the screen's current view?

[sonora]

PS: A considerable portion of the wait time (about half) is spent for this line, even if I have no Smart Folders created;

OsmAnd/OsmAnd/src/net/osmand/plus/configmap/tracks/TrackFolderLoaderTask.java

Line 95 in 5ca0048

smartFolderHelper.addTrackItemToSmartFolder(item);

[pebogufi]

With S10:
After removing all but 50 files in tracks subfolders and no tracks visible the function My tracks / "Visible on map" needs 3-5 sec to show only one line "currently recording track" while recording is off.

[sonora]

It's already been noted a number of times, e.g. here #17257, and I believe often before the statistics display was added. Also, I see the same effect under Android 11, even when I use a build which targets Android 10, and/or using the "All files access" Android permission, which circumvents the new file access mechanisms.

So chances are the root cause for this may be in how we do things different in our code now vs. the previous (collapsable list) MyPlaces>Tracks screen (which was comparably instantaneous), and not necessarily in the stat calculation or in the new google storage access limitations...

[Corwin-Kh]

This issue should be checked on latest build as it could be already fixed. Aside from this issue, there was another one that is related to SmartFolders and affected opening My places -> tracks. It is fixed in #2159

[pebogufi]

tracks. It is fixed in #2159

Link reports error 404, not found

After using again my folders with 767 gpx files
opening track folder needs 10 sec, that is no improvement to the initial report.
Tested with S10 (as used above), OsmAnd~ 4.6.0#930m, veröffentlicht: 2023-10-27

[sonora]

Unfortunately, I have the same experience, there is no improvement with latest builds. It is still very slow to open either the My Tracks or the Visible Tracks screen. Something we do there seems to be extraordinarily resource intensive. (Interestingly, while it is always slow, times can vary from 3 to 15 sec, Android 11, ~3500 tracks in the folder).

What's worse is #18077, which is a showstopper for me using our 4.6.x releases at all, I am forced to stay with 4.5 for all my work.

[sonora]

Fun fact: When you create a smart folder with filter condition "Visible on map", that one opens essentially instantaneously. But opening the built-in "Visible tracks" screen creates the long wait, every time.

[pebogufi]

@sonora VERY fun fact!
Thanks a lot for that hint, it is great.
I can confirm, that it opens instantaneously and
build in quite slow every time.

[sonora]

I tested a new build with the GPX db mprovement merge of 2 weeks ago. The effect was that upon the first app start I received a black, empty map screen with no visible activity, I believe for 2, perhaps 3 minutes. Knowing that the GPX database was probably being updated, I waited things out, and eventually the map appeared and the app functioned normally.

Then testing MyPlaces/Tracks and Visible Tracks, there is no decisive speed-up. It still takes about 12 seconds watching the spinner every time I open one of the 2 screens, or go from one to the other.

So also this supports my observations that we did not introduce this bug by adding the statistics functionality (bug was here definitely before that), and perhaps anything else which can be fixed in the db, but we added it when we changed from the prior collapsible view to the new paged view.

So it looks like we still need an analysis which process or lookup we call every time when opening one of the 2 screens and which scales so badly with increasing number of tracks on the device?

(Android 11, OsmAnd build targeting Android 10 and having 'All files access' permission, so no file access limitation in play for this test at all.)

[Chumva]

@sonora can you try new nightly? There was an added progress update for loading tracks similar to the one that was before the changes with these screens

[sonora]

@Chumva , @vshcherb Let me re-open this, the crash is quite frequent, so we should not leave it like that...

[Chumva]

@sonora Do you have this crash in latest nightly?

[sonora]

@Chumva I just noticed it's a different one: exception.log.txt (And it is less frequent, only when visiting the Tracks screen repeatedly.)

Unfortunately, I am also back to about 12+ seconds every time I open the screen ... ;)

[vshcherb]

@sonora did you test it on nighhlty or release build? On my phone startup on nightly takes 16 seconds but release build just 3 seconds (installed via gplay)

[sonora]

I have buit from our master branch, on 2023-12-21 (evening).

PS: When I say "12+ seconds", that's the time (every time)iit takes to open MyPlaces>Tracks (from the map screen, via the menu). (Not the time to start the app or anything.)

[sonora]

Here's another crash:
exception.log2.txt

[sonora]

Also: Possibly connected to the commits here, I am again losing the selected track sorting on occasion. I have my track screen sorted A-Z, and had lately several times found it showing to be sorted by "Last modified", am 100% sure I have not set that myself. May happen when updating the app, or on sone other occasion, not sure.

Can open another issue, if you think ithat's better than noting that here.

[vshcherb]

This issue will be closed cause the fixes are complete in 4.6, so next issue could continue in 4.7

[sonora]

New crash... exception.log.txt

[vshcherb]

Unfortunately we had a bad code practice... 5086fb4 Still fixing it.
Checked the code different places, please let me know if it happens again

[pebogufi]

New crash... exception.log.txt

see also
#18828

[sonora]

Still crashing, even with 5086fb4: New exception.log.txt

[vshcherb]

Fixed 7a362ff

[sonora]

At least no more crashes so far, thanks!

But it is still soo slow... 11to 14 seconds to open the My tracks screen. And the same again when from there you tap on "Visible on map".

Somehow what we do there does not scale well at all. It seems to somehow scale with the number of tracks in the device, why? It is hardly acceptable that a screen to display a list of 68 folders takes so long to come up. And again so long, to display a screen of visible tracks, even if zero tracks are displayed on the map??

[vshcherb]

Do you test only nightly builds or do you have release builds?

[vshcherb]

I mentioned in another issue that I see for last 6 months tremendous difference between builds and I can't improve debug build anyhow with adb package compile...

[sonora]

Hm, I use the master branch to build, turn all debugging off, use no debug libs, and turn all optimizations on, so it should be like a "release". I can explicitly download our 4.6.11 release build, if you want.

[vshcherb]

I didn't test release builds itself with https://source.android.com/docs/core/runtime/jit-compiler adb shell cmd package compile -m speed -f my-package

[vshcherb]

I can see that build is slow by just enabling /disabling pev plugin in plugins section

[sonora]

I find no systematic correlation between plugins being on or off. But it is weired that the time it takes to open the screen varies so much, I get everything between 8 and 30 seconds, most of the time between 10 and 14, though. Perhaps a memory issue?

Things definitely scale with the amount of data you have. If I go to only 3 folders with about 100 tracks in each, I am more in the 3 second each region to open the screens.

[vshcherb]

Well I don't think you've got my point. Obviously every program should scale linearly to the number of files, the issue I think is how slow 1 unique operation perform.

You need to try https://download.osmand.net/releases/net.osmand-4.6.11-4611.apk (and probably compile it extra with adb shell cmd package compile -m speed -f net.osmand)

[sonora]

I have just now tried that build as is, and it is equally slow. Upon the first starts (after placing the tracks in the ttrack folder) Android even complained several times that the app has stalled, and offered to kill it (the app was frozen at the time, ptobably creating the track db).

Afterwards, this build performed no better than the nightly.

Regarding

Obviously every program should scale linearly to the number of files,

Absolutrly not! My Windows PC has 300,000 files. If Microsoft followed that scaling law, I could never even start the File Explorer to display the hard drive's root... 😉

And my Android phone surely also has thousands of files - but the Samsung My Files app displays every folder level instantaneously.

Reading the entire tracks db (if that's what we do?) every time the user opens the My Places>Tracks screen and then again the "Visible on map" screen seems just the wrong strategy.

EDIT: Do we perhaps perform a sync from the file sysrem's 'tracks' folder, every time we open the screen? Syncs could be limited to one time, at App startup

[vshcherb]

With nightly build that starts up for about 10-15 seconds, My places Tracks with 531 Track, distance 42 000 km and size 122.8 MB - still opens for 1 second. So in that case I can't confirm this issue.
RAM Memory footprint - 90 MB.

[sonora]

Yes, the app startup time is about 12 sec for me. I think it has increased some lately (from perhaps half that), but that's not my primary issue.

Can you please try this: Use a file manager app to duplicate all your subfolders in your tracks folder, do that a few times to get to perhaps 4000 files. When the file manager asks to overwrite the existing entries, chose "Rename" or similar so you really create duplicate folders.

I have just tried this:

• For originally 10 folders with about 500 gpx tracks total, the "Tracks screen display time" is like 2 seconds or so.
• But with the duplication to 80 folders / 4000 tracks, I am up to about 30 seconds loading time...

[vshcherb]

There is always a hope to fix though issue not reproduced. No video / no logging, so there is no progress here, only hope that 1 day we will see something.

I think we should focus to get clear logs + video on synthetic tests, in this conversation there is no specific at all. S10 of course is not a target device any more. But if we reproduce then descision could be made.

[sonora]

@vshcherb Ah, a misunderstanding, Victor, my apologies! When you had re-opened the issue after my post #17805 (comment) above, I was under the impression you had successfully re-produced it with the steps I had noted there.

Well, I am a little at loss what could help here: a video showing a 12 sec wait before the tracks screen shows content does not seem very helpful? And regarding log files I am not sure what to produce using what mechanism, maybe someone who can reproduce can help out or explain?

[sonora]

PS: The workaround seems actually quite clear and we could document it somewhere: You have to limit the amount of data in the sub-tree of your tracks data folder. I would still have to research the limits, that's why I am asking - it is likely a certain number of tracks (like <1000 or something), or it could be by number of accumulated trackpoints...

[sonora]

PS2: One side effect we could probably fix fast is to disable the pull-down refresh.

If you view a longer list of line items (folders or tracks), and scroll it down, then up again, reaching the top of the list initiates a refresh, it seems, and the screen is again stuck for 12 sec for me every such time.

I suggest initiating a refresh when pulling down while you have reached the top of a list on this screen is a feature not really needed here, so we could disable that?

[sonora]

This obvious fix at least brings some noticeable improvement whenever no smart folders are present: #19228.

In my setup this PR speeds things up from 12-14 sec to 5-9 sec. There is more potential, but merging this is a good start.

[sonora]

More low hanging fruits:

• Reloading all tracks when on the MyPlaces/Tracks screen tapping"Visible on map" seems obsolete

• And also: Isn't this sorting code https://github.com/osmandapp/OsmAnd/blob/5ca0048428c1d16821eae8b089fde852265fdecb/OsmAnd/src/net/osmand/plus/track/helpers/GPXFolderUtils.java#L30C1-L45C2 superseded by TracksAdapter.java? (At least for the purpose of populating the new Tracks screen, but we seem to execute it there.)

[vshcherb]

(1) The time to open the Tracks screen should not depend on the total number of tracks in all folders, but merely on the number of items to be shown on the screen to be opened. (In my case simply 70 subfolders and zero tracks on the top level screen)

It's not correct ... Time depends on the number of tracks that's totally fine. Loading & sorting each time your open a screen is fine as well to create a structure especially on new load.

What's not fine that 3500 is a very small number! Loading 3500 records from db and sorting 3500 numbers is about 0.1 seconds should be.

270 t/s - 3500 tracks: 12 - 14 sec
300 t/s - 2500 tracks: 7 - 8 sec
~500 t/s - 1500 tracks: 3 sec

It looks totally fine that it's linear dependency but 0.2 millisec per track is too much. Did you try release versions ? https://download.osmand.net/releases/net.osmand-4.7.2.apk

[sonora]

@vshcherb

• Can we please merge this PR #19228? It makes sense in any case, speeds things up by ~50% (if you don#t have a smart folder), saves CPU cycles and battery power.
• Sorting every time: Well, the sorting is important for what you see on the screen, not what's in the background list... 😉 I almost never look at the "All" list, the only one which would require sorting of all 3500 line items.
• Time stamps: see e.g this log file (this already includes the improvements by #19228?): logcat.log.txt. It implies like 4 seconds, e.g.

03-03 18:13:52.584   810  2715 I net.osmand: TrackFolderLoaderTask Start loading tracks in tracks {pool-20-thread-1}
03-03 18:13:56.668   810  2715 I net.osmand: TrackFolderLoaderTask Finished loading tracks. took 4085ms {pool--20-thread-1}

But the actual time I am waiting for the list to appear on screens seems even longer (often like twice that), not sure where the extra time is spent. (PS: I find that using a pull-down refresh on the Tracks screen often is quicker than when I re-enter the screen from the menu, also that seems weird.) an, of course, as stated above, I believe there is no need to again re-load the very same the tracks list when changing to the "Visible on maps" screen.

And yes, you are right, reading a file list table of 3500 files should be peanuts, that's what I am saying all along ... ;)

[vshcherb]

1. That's up to @Chumva I think

Tonight (or later this week) we decided finally switch nightly builds to release builds. We just tested that startup time is 5 times faster on release builds

[sonora]

@Chumva Some more code questions:

• OsmAnd/OsmAnd/src/net/osmand/plus/myplaces/tracks/filters/SmartFolderHelper.kt

  Line 248 in 39185d4

  return HashSet(allAvailableTrackItems)

  • Should simply return allAvailableTrackItems, it already is class HashSet?

• Remove

  OsmAnd/OsmAnd/src/net/osmand/plus/myplaces/tracks/filters/SmartFolderHelper.kt

  Line 37 in 39185d4

  private val LOG = PlatformUtil.getLog(SmartFolderHelper::class.java)

  and the 3 occurrences of LOG.Debug, this speeds things up considerably (can even back out my interim fix then)

[sonora]

@Chumva I believe we can now merge #19239 , unless you see anything I overlook. This beats the previous fix, the time needed now dropped to 4 - 6 sec for my setup, and I see no drawback. For up to ~1500 tracks I am now at 1 sec.

It looks like most of the now remaining time is used on reading the GPX database and creating the stats.

[vshcherb]

Today was new build released that's not debug and signed as release apk... So everything needs to be reinstalled actually. I tested it still doesn't work as fast as release but this one became 20-50% faster.
So I think to let you test the new builds and close the issue. Actually the only extra test we could do again is to load 3000 tracks as @sonora mentionned

[vshcherb]

Actually I was wrong ... Now is build officially released, so it's ready to test.

[vshcherb]

yes I can confirm current nightly build 3x-5x faster and a lot faster actually

[sonora]

Ok, thanks, I am out on a trip and will not be able to re-install my environment for a test, hope someone else can help out here.

Looks like you have not merged #19239 yet, it's definitely worth it in any case, though.

[vshcherb]

Yes assigned to review & merge

[sonora]

@vshcherb Just out of curiosity: Using e.g. assembleAndroidFullOpenglFatRelease should produce a "fast" build, no matter if we sign it with our debug or our release key, or am I missing anything here?

EDIT: see #18168 (comment)