Request only visible range of rows How

[davidglezz]

Is it possible to load chunks of rows (visible range)? as done at https://material.angular.io/cdk/scrolling/overview where Datasource connect has argument CollectionViewer to get range.start and range.end.
I wanted to load a list of, for example 1 million elements, and not download all the elements, only the visible part as you scroll. I have not found any example or code that allows me to load only a part of the array, and define how much is the total number of rows.
Of course I wanted to avoid pagination (next page, prev page, ...) but I find interesting that it allows automatic downloading of pages as the user scrolls.

[dbas-dn]

I'm interested in this question too

[shlomiassaf]

Hi

This is doable right now with the current implementation, this will be a very simple example:

---

EDIT: See link to live demo in next comment

import { ChangeDetectionStrategy, Component, ViewEncapsulation } from '@angular/core';
import { createDS, columnFactory } from '@pebula/ngrid';

import { Person, DemoDataSource } from '@pebula/apps/shared-data';
import { Example } from '@pebula/apps/shared';

@Component({
  selector: 'pbl-infinite-scroll-example',
  template: `
<pbl-ngrid [dataSource]="ds" [columns]="columns">
  <h1 *ngIf="loading">infi</h1>
</pbl-ngrid>
`,
  encapsulation: ViewEncapsulation.None,
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class InfiniteScrollExample {
  loading = false;

  columns = columnFactory()
    .table(
      { prop: 'id', width: '100px' },
      { prop: 'name', width: '100px' },
      { prop: 'gender', width: '50px' },
      { prop: 'birthdate', type: 'date', width: '25%' },
    )
    .build();

  ds = createDS<Person>()
    .onTrigger(event => {
      if (!this.allPeople) {
        return this.datasource.getPeople(100, 1000)
          .then( people => {
            this.allPeople = people;
            return this.allPeople.slice(0, Math.min(this.allPeople.length, (this.ds.source || []).length + 100));
          });
      } else {
        this.loading = false;
        return Promise.resolve(this.allPeople.slice(0, Math.min(this.allPeople.length, this.ds.source.length + 100)));
      }
    })
    .create();

  private allPeople: Person[];

  constructor(private datasource: DemoDataSource) {
  }

  ngAfterViewInit() {
    this.ds.onRenderedDataChanged.subscribe(() => {
      if (this.ds.length - this.ds.renderStart < 20) {  // in real example you let user provide threshold, ratio, etc..
        if (!this.loading) {
          this.loading = true;
          setTimeout(() => this.ds.refresh(), 100);
        }
      }
    });
  }
}

This will not work for large scale implementation, so you must wrap it with some directive (plugin) or class, to control the flow, various state and ofcourse visual helpers to show it's fetching data.

I will build a directive plugin for that soon.

Note that there are still open questions here, like what to do when filtering, sorting, etc... because the data capacity is unknown and the control is client based.

I think I'll let sort and filter run locally on the existing data, if someone wants more control he should opt-in to handle custom triggers (setCustomTriggers('pagination')

[shlomiassaf]

I'v uploaded a demo of infinite scroll to the documentation site, SEE HERE

[davidglezz]

@shlomiassaf Thank you very much for everything. Infinite scrolling is very useful in certain scenarios. But I wanted to explain another type: lazy load of rows. so you don't have to load all the rows at once, just load the visible part.

In the example "Specifying data" of https://material.angular.io/cdk/scrolling/overview

The scroller streams view position and the datasource fills that section of array

[shlomiassaf]

@davidglezz you're talking about Virtual Scroll where you only render the rows visible on screen + some buffers.

This exists for a long time now, you can read more in the documentation

There is also a demo here showing more advanced options of the virtual scroll.

[davidglezz]

Sorry, you still haven't understood me, I'll try to explain it better.

I'm not talking about render, but about asynchronous loading of the dataset, allowing me to make http requests to the server.

I show you how cdk / scrolling does it but only for lists

• DataSource has connect method that returns the array or better: BehaviorSubject of array
• We can make subject.next(Array.from({length: 1000000})) with array of size but undefined items.
• The method connect has the argument CollectionViewer that streams start item and end item of viewport.
• At this point I use debounce() over collectionViewer.viewChange to only request the visible range/visible chunks where the user stops scroll.
• When I receive the new data from the server, I fill the corresponding part of the array and notify the change with subject.next(modified array)

This is an official example, the same as what I am telling you but without debounce. Load the parts of the array into chunks of 100 rows.

export class MyDataSource extends DataSource<string | undefined> {
  private _length = 100000;  // Set size, without having to download all rows
  private _pageSize = 100; 
  private _cachedData = Array.from<string>({length: this._length}); // Array with undefined elements where we load chunks
  private _fetchedPages = new Set<number>();
  private _dataStream = new BehaviorSubject<(string | undefined)[]>(this._cachedData);  // To pass the array with updates
  private _subscription = new Subscription(); // Subscription that gives me the visible range

  connect(collectionViewer: CollectionViewer): Observable<(string | undefined)[]> {
    this._subscription.add(collectionViewer.viewChange.subscribe(range => {
      const startPage = this._getPageForIndex(range.start);
      const endPage = this._getPageForIndex(range.end - 1);
      for (let i = startPage; i <= endPage; i++) {
        this._fetchPage(i);
      }
    }));
    return this._dataStream;
  }

  disconnect(): void {
    this._subscription.unsubscribe();
  }

  private _getPageForIndex(index: number): number {
    return Math.floor(index / this._pageSize);
  }

  private _fetchPage(page: number) {
    if (this._fetchedPages.has(page)) {
      return;
    }
    this._fetchedPages.add(page);

    // Use `setTimeout` to simulate fetching data from server.
    setTimeout(() => {
      this._cachedData.splice(page * this._pageSize, this._pageSize,
          ...Array.from({length: this._pageSize})
              .map((_, i) => `Item #${page * this._pageSize + i}`));
      this._dataStream.next(this._cachedData);
    }, Math.random() * 1000 + 200);
  }
}

Please reopen the issue

[shlomiassaf]

@davidglezz Sorry missed your comment.

Infinite Scroll is exectly what you need, please see here

In this link you can see a demo with 3 million rows, loaded on demand.

If, for some reason, it's not enough for your, you can extend the datasource that the grid accepts and build whatever logic you desire.

Note that the Infinite Scroll feature does exactly that, it extends the datasource components (datasource, adapter, factory) and implements internal logic to allow simple and easy infinite scrolling with messages to load new items when those required when scrolled into view.

You can read more about how the datasource works in nGrid here

[davidglezz]

How good!! Thank you very much for your effort @shlomiassaf