Working with large datasets.

[OmarRafique]

Hi.

I have a datasets of one 1 million samples and each sample has 50000 features. My machine is Intel i9 with 64GB DDR4 RAM.
Will the giotto-tda Mapper scale up to this dataset on the specified machine?

In general, what are the scalability issues that I should expect while using giotto-tda Mapper?

[ulupo]

This depends a lot on your choice of filter function and clustering algorithm, which has nothing to do with the implementation of Mapper per se.

One known factor which does have to do with the Mapper implementation, in giotto-tda specifically, is the handling of the covering step. Here we internally create a dense boolean matrix of size

n_samples x n_interval

where n_samples in your case is 1 million. As long as you don't choose more than 60000 intervals or so, you should be fine.