Experiment with polars

[ecomodeller]

Rationale

• Polars optimizes expressions, e.g. calculating residuals is done in almost every metric, this will be optimized.
• Polars has in many cases a more readable api, where mutation is not possible.
• Polars is strict and data type conversion must be made explicit, avoiding surprises.

This is an experiment.

Some conclusions:

• Polars is strict about order of columns and data types in order to be fast!
• Pandas and it's weird MultiIndex is deeply intertwined in modelskill.
• It is not easier to apply a list of python functions to a polars dataframe, than it is to a pandas one.

But if you buy into polars, it is actually quite readable:

obs = pl.col("obs_val")
mod = pl.col("mod_val")
diff = obs - mod

named_metrics: dict[str, pl.Expr] = {
      "bias": diff.mean().alias("bias"),
      "rmse": diff.pow(2).mean().sqrt().alias("rmse"),
# TODO add more ...
}

sel_metrics = [named_metrics[metric] for metric in metrics]

res = df.group_by(by).agg(*sel_metrics)

And the aggregated dataframe looks like this in the simplest case:

shape: (1, 3)
┌─────────────┬──────────┬──────────┐
│ observation ┆ rmse     ┆ bias     │
│ ---         ┆ ---      ┆ ---      │
│ cat         ┆ f64      ┆ f64      │
╞═════════════╪══════════╪══════════╡
│ alti        ┆ 0.111143 ┆ 0.064629 │
└─────────────┴──────────┴──────────┘

[ecomodeller]

Remaining:

• Temporal aggregation by frequency (cc.skill(by="freq:D"))
• Temporal aggregation by date (cc.skill(by="dt:month")
• Directional metrics
• Custom metrics
• Quantities
  

[ecomodeller]

The skill table output with polars include the datatypes, but lacks the nice display of the multiindex.

One option is to convert back to pandas for display and style and the other option is to use great_tables