StatOperator

class nvtabular.ops.stat_operator.StatOperator

Bases: nvtabular.ops.operator.Operator

Base class for statistical operator classes. This adds a ‘fit’ and ‘finalize’ method on top of the Operator class.

fit(col_selector: nvtabular.columns.selector.ColumnSelector, ddf: dask.dataframe.core.DataFrame) → Any

Calculate statistics for this operator, and return a dask future to these statistics, which will be computed by the workflow.

fit_finalize(dask_stats)

Finalize statistics calculation - the workflow calls this function with the computed statistics from the ‘fit’ object’

clear()

zero and reinitialize all relevant statistical properties

set_storage_path(new_path, copy=False)

Certain stat operators need external storage - for instance Categorify writes out parquet files containing the categorical mapping. When we save the operator, we also want to save these files as part of the bundle. Implementing this method lets statoperators bundle their dependent files into the new path that we’re writing out (note that this could happen after the operator is created)