Virtual Scroll - add/remove non-visible elements depending on scroll position. about angulargrid

[s-yadav]

@acondiff @alereisan @essn @keithics

So finally long awaited feature is out in 0.5.1 . It include removing non visible dom element, their watchers and infinteScroll . Updated docs http://ignitersworld.com/lab/angulargrid/ . Also will be improving docs to explain different use case.
In master build I have also uploaded an example with infiniteScroll and performantScroll. Thanks to @acondiff for it.

Let me know if you guys find any issue.

[s-yadav]

Will be working on this soon. Marked as enhancement.

[austincondiff]

Thanks. Can I get a rough timeline for this feature? I need to know whether or not I need to change modules or develop this myself so that we can launch in a month or two.

[alereisan]

+1 this would be awesome

[essn]

+1 I'm using this on a current project as well and would love the feature before it goes live

[s-yadav]

@acondiff
This feature is complicated one, to calculate the top position of each item, it need to know heights of previous elements. And if we remove the previous elements from DOM on the scroll, we can't calculate the proper position.
I will see if I can find an elegant solution, which does not require removing of elements from DOM, and can improve the performance.
Being busy with other projects I can't commit a timeline, but at least I will let you know by next weekend that I found some solution for it or not.
I will also be very happy if someone can take a fork, and work on this feature.

To start this, can you let me know what are the use cases in which it's giving performance issue. ?

[austincondiff]

@s-yadav That should be a fairly easy problem to solve. Simply store four arrays, gridTopValues, gridLeftValues, gridWidths and gridHeights. They would look something like this at any given time:

gridLeftValues = [0,354,753,1121,1452,0,354,753,1121,1452,354,0,354,753, ...];
gridTopValues = [0,0,0,0,0,423,634,753,2311,756,856,2231,3545,4232, ...];
gridWidths = [455,455,455,455,455,455,455,455,455,455,455,455,455, ...]; //could be varying widths
gridHeightsValues = [476,865,342,967,945,354,753,1121,1452,354,1354,753, ...];

Then just bind each of the array values to it's corresponding grid item's top, left, and width properties like so, (without knowing too much regarding how it works):

for(var i in gridItems) {
  var itemWidth = gridWidths[i];
  var itemTop = gridTopValues[i];
  var itemLeft = gridLeftValues[i];

  // apply variables to grid item...
}

When the element is no longer on the DOM it would simply not use those values in the array yet you still would have them to calculate future values. Where it get's tricky is when the grid is resized once scrolled down a ways, but I think that is definitely solvable as well.

[s-yadav]

Here breaking the feature complexity on sub problem.

1. Calculating position
   As you mentioned we can keep the top values for element in a array, but the grid width is expected to be changed a lot (which will affect the height). So if we are removing element from DOM, we need to add it back to dom to calculate its height. This can be more jerky.
2. Adding / removing dom on scroll
   • Need to find how optimally we can remove and add dom, without user noticing about.
   • Need to see how angular behave when we add/remove ng repeated element, as they are binded to some object. We may need to write a seperate repeat module.
   • Need to see how ng-animate will be effected.

As there are chances of breaking exisiting functionalities by adding requested feature, we need to go through all cases. One simple way is just to detach watcher from elements which are not on view, but not how much performance gain it will give.

[austincondiff]

It might be helpful to refer to this https://github.com/sroze/ngInfiniteScroll

You might be able to scroll infinitely with some ng-repeat's, especially if you have implemented some sort of a lazy-loading mechanism where it loads more items when you get to the bottom. Though this may be the case, memory is limited, and as we approach that limit the browser can get very sluggish. The end result is the user experiences a fair amount of jank which provides for a very poor user experience.

Pinterest was able to solve this problem by having their cards be a fixed width. That might be a quick route to solve this problem. But it might be possible to do it while maintaining that dynamic width as well. Seeing as it really depends on the layout flow of the browser, it might be difficult.

If you would like we could both jump on a Hangout to go through my particular use case so you can then evaluate how to move forward with this feature.

[alereisan]

Maybe this helps too: https://github.com/angular-ui/ui-scroll

This demo shows how non visible items are removed and loaded again if scrolling up, and when scrolling down loading in more: http://angular-ui.github.io/ui-scroll/demo/examples/windowviewport.html

[s-yadav]

@acondiff ngInfiniteScroll does lazy loading when we scroll to end of page. It does not remove element from top which are not on view.
I think, what we are trying to solve is on removing and adding elements dynamically depending on what element will be on view at specific scroll position. Please correct me if this is not what you are expecting.

@alereisan In demo you can see there is blank screen for some time on scroll and than it populates. This can be a bad user experience.

I have done this see https://github.com/s-yadav/LongListScroller , (Its is completly on jQuery).
But here its not just removing or adding element, there are some other cases as well. (Like calcluating height of element, support with ng repeat and ng animate)

For sure we can come on hangout and discuss the problem as well as solution.

[s-yadav]

@acondiff Can you create a example where you getting a performance lag, so we test with particular thing.

[austincondiff]

@s-yadav Oh sorry, you are right in regards to ngInfiniteScroll. It must have been a late night.
Here is an example. http://dev.rss.com/the-hill-3

Scroll down so that it loads more items about 15x. They are loaded in chunks of 50 items.

[keithics]

+1 for this feature

[alereisan]

Do you know when this will be available?

[austincondiff]

@s-yadav @keithics @alereisan I've modified angulargrid to have infinite scroll functionality. I want to try my hand at virtual scroll too but that will take a little more time to solve. I will submit a pull request for now if thats cool. Regardless I would like to contribute as I am heavily invested in this layout in my app and I really need this functionality. Check it out here http://codepen.io/acondiff/pen/yepggM

[austincondiff]

Pull request submitted #30

[austincondiff]

@s-yadav Some notes I jotted down on expected behavior and how it might respond to various problem scenarios (it is up for change if need be):

The user scrolls down. We load in new a new batch of items from the server and render them in cards that are appended to the DOM. We set the left and top position based on the shortest column. After 50 times the number of columns cards are on the DOM, we begin to remove some items at the top. A new batch of 25 loads, as the user scrolls down we start taking away cards one by one until there are no more than 50 times the number of columns. When the user scrolls up, we do the opposite of what was done had the user scrolled down. We take items from the bottom to maintain that 50 times the number of columns and add items to the top one by one. We calculate its position by placing it in the column with the highest top value (bottom-most column).

If the user resizes their browser while we have items taken from the top, we simply set the top values of the first available card in each column too the what it is currently and any new columns will have a top value of the first available card in the first column. Then from there we do a reflow of subsequent available cards. In this condition, if the user get’s to the top and we have more items not added to the DOM, we add the remaining items to the top and do a reflow. If the user is scrolling up reaches the beginning of the master array and still has room left to scroll up, we will just do a reflow.

As far as data is concerned, we will keep a master array of objects of all of the items retrieved from the server (the model), then based on each item’s index, if it matches the set of available items based on the users scroll position, we will append those items and compile their html to render out any angular expressions or directives used.

Let me know if that makes sense and your thoughts/ideas/concerns on this.

[austincondiff]

Look at Pinterest and check out the DOM in the dev tools as you scroll down. It adds to in chunks as it makes requests to the server. Scroll down a ways and then start scrolling back up. It removes them one by one from the bottom and starts adding to the top. As you scroll back down it does the opposite if the below items have already been retrieved and are in the source array.

[s-yadav]

Humm I see.

So here is the process I can think of is required to do it on angulargrid.

1. Whenever model change, store/restore reference of all element in the array.
2. Whenever window is resized, put the clones of all element in dom, with opacity zero, top zero, z-index -1, and calculate each clones height which will be used to calculate the top position of each element. This process is happening but needs to tweak a little bit.
3. (i). Keep a maximum number of item in scroll view (let's say 50, this can be taken from user wit 50/60 as a default value), at a particular scroll position, find what 50 sets of the element need to be on view and display those. For doing this in a performant way you can see how I have done on (https://github.com/s-yadav/LongListScroller ). I am calculating trigger points (scroll position during which this calculation should be done, and not every time.)

(ii). Or can use the three-page technique, divide the total scroll hight of the container with container height, so we will be having scrollHeight/ contHeight pages. At any scroll position, find out which page is covering most of the scroll view. Now at that scroll position, the page which is covering most of the scroll view, next page and previous page need to be present in dom, you can easily find out which all elements need to be present in those three pages, other elements you can detach from dom.

Whatever technique you are following, need to make sure following things.

1. Keep the constants (like 50 items at a given time, trigger points, the number of pages) such that user does not see the white screen during scroll.
2. All the constants should be optimized to different devices and screen resolutions.
3. Some mobile browser does not execute js during scroll, in which condition people may use some custom scroll plugin, so you need to provide an API through which they can pass scrollTop value(which they get from their custom scroll plugin.). You can use the scrollTop value to do the same thing which you do on native scroll listener.

[austincondiff]

@s-yadav Brilliant. It sounds like you might have a better idea than I do on how to tackle it. I don't want to get in your way so take a shot at it and I would be more than willing to work with you if you need me to. I'd love to jump on a Hangout call to discuss it. :)

[austincondiff]

By the way, that project you did that you linked to is pretty impressive. It sounds like we just need to merge the two to get this working.

[austincondiff]

Also, use this code in your development to keep track of how many watchers are on the page at once. I've found it extremely helpful.

(function() {var a = document.createElement("script");a.src = "https://rawgit.com/kentcdodds/ng-stats/master/dist/ng-stats.js";a.onload=function(){window.showAngularStats({position: 'topright'})};document.head.appendChild(a)})();

(https://github.com/kentcdodds/ng-stats)

This I think this would also help in the fact that we are not only limiting the number of elements on the DOM, but also watchers on the scope.

[s-yadav]

Hi @acondiff ,
Tried something for this check http://codepen.io/s-yadav/pen/EPoOZo, this removes and adds element on dom as required, but yet not got any performance boost. Will be trying to limit watchers for visible element .
And also added a infinite scroll code, as ngInfiniteScroll does not allow custom scroll plugins, while a lot of people use custom scroll plugin in mobile.

[austincondiff]

@s-yadav I looked at the code and at the DOM as I scrolled via the inspector and ran the watch script I shared. That looks great. The objective was not to limit watchers necessarily (thought it helps), it was mainly to prevent the DOM from being cluttered with elements that you are not looking at. The only minor performance hit I see is as you scroll down, the digest cycle takes longer with each load of new items. I am not sure if you are going to be able to get around that one though.

Anyways, hey I know you are contributing to your own project, but this has been something I have been trying to do for months. I wanted to send some good karma your way. Do you like Starbucks? Shoot me a message on my personal website (austincondiff.com), and I can get you a gift card. :)

[s-yadav]

@acondiff Thanks a lot acondiff, that will not be required. We may come to help to each other in different way.