mlair.run_modules.post_processing

Post-processing module.

Module Contents

Classes

PostProcessing

Perform post-processing for performance evaluation.

Attributes

__author__

__date__

mlair.run_modules.post_processing.__author__ = Lukas Leufen, Felix Kleinert
mlair.run_modules.post_processing.__date__ = 2019-12-11
class mlair.run_modules.post_processing.PostProcessing

Bases: mlair.run_modules.run_environment.RunEnvironment

Perform post-processing for performance evaluation.

Schedule of post-processing:

  1. train an ordinary least squared model (ols) for reference

  2. create forecasts for nn, ols, and persistence

  3. evaluate feature importance with bootstrapped predictions

  4. calculate skill scores

  5. create plots

Required objects [scope] from data store:

  • model [.] or locally saved model plus model_name [model] and model [model]

  • generator [train, val, test, train_val]

  • forecast_path [.]

  • plot_path [postprocessing]

  • model_path [.]

  • target_var [.]

  • sampling [.]

  • output_shape [model]

  • evaluate_feature_importance [postprocessing] and if enabled:

    • create_new_bootstraps [postprocessing]

    • bootstrap_path [postprocessing]

    • number_of_bootstraps [postprocessing]

Optional objects

  • batch_size [model]

Creates

  • forecasts in forecast_path if enabled

  • bootstraps in bootstrap_path if enabled

  • plots in plot_path

_run(self)
estimate_sample_uncertainty(self, separate_ahead=False)

Estimate sample uncertainty by using a bootstrap approach. Forecasts are split into individual blocks along time and randomly drawn with replacement. The resulting behaviour of the error indicates the robustness of each analyzed model to quantify which model might be superior compared to others.

report_sample_uncertainty(self, percentiles: list = None)

Store raw results of uncertainty estimate and calculate aggregate statistics and store as raw data but also as markdown and latex.

calculate_block_mse(self, evaluate_competitors=True, separate_ahead=False, block_length='1m')

Transform data into blocks along time axis. Block length can be any frequency like ‘1m’ or ‘7d. Data are only split along time axis, which means that a single block can have very diverse quantities regarding the number of station or actual data contained. This is intended to analyze not only the robustness against the time but also against the number of observations and diversity ot stations.

create_error_array(self, data)

Calculate squared error of all given time series in relation to observation.

static create_full_time_dim(data, dim, sampling, start, end)

Ensure time dimension to be equidistant. Sometimes dates if missing values have been dropped.

load_competitors(self, station_name: str) → xarray.DataArray

Load all requested and available competitors for a given station. Forecasts must be available in the competitor path like <competitor_path>/<target_var>/forecasts_<station_name>_test.nc. The naming style is equal for all forecasts of MLAir, so that forecasts of a different experiment can easily be copied into the competitor path without any change.

Parameters

station_name – station indicator to load competitors for

Returns

a single xarray with all competing forecasts

calculate_feature_importance(self, create_new_bootstraps: bool, _iter: int = 0, bootstrap_type='singleinput', bootstrap_method='shuffle') → None

Calculate skill scores of bootstrapped data.

Create bootstrapped data if create_new_bootstraps is true or a failure occurred during skill score calculation (this will happen by default, if no bootstrapped data is available locally). Set class attribute bootstrap_skill_scores. This method is implemented in a recursive fashion, but is only allowed to call itself once.

Parameters

  • create_new_bootstraps – calculate all bootstrap predictions and overwrite already available predictions

  • _iter – internal counter to reduce unnecessary recursive calls (maximum number is 2, otherwise something went wrong).

create_feature_importance_bootstrap_forecast(self, bootstrap_type, bootstrap_method) → None

Create bootstrapped predictions for all stations and variables.

These forecasts are saved in bootstrap_path with the names bootstraps_{var}_{station}.nc and bootstraps_labels_{station}.nc.

calculate_feature_importance_skill_scores(self, bootstrap_type, bootstrap_method) → Dict[str, xarray.DataArray]

Calculate skill score of bootstrapped variables.

Use already created bootstrap predictions and the original predictions (the not-bootstrapped ones) and calculate skill scores for the bootstraps. The result is saved as a xarray DataArray in a dictionary structure separated for each station (keys of dictionary).

Returns

The result dictionary with station-wise skill scores

static get_distinct_branches_from_bootstrap_iter(bootstrap_iter)
rename_boot_var_with_branch(self, boot_var, bootstrap_type, branch_names=None, expected_len=0)
get_orig_prediction(self, path, file_name, prediction_name=None, reference_name=None)
static repeat_data(data, number_of_repetition)
_get_model_name(self)

Return model name without path information.

_load_model(self) → mlair.model_modules.AbstractModelClass

Load NN model either from data store or from local path.

Returns

the model

plot(self)

Create all plots.

Plots are defined in experiment set up by plot_list. As default, all (following) plots are enabled:

  • PlotBootstrapSkillScore

  • PlotConditionalQuantiles

  • PlotStationMap

  • PlotMonthlySummary

  • PlotClimatologicalSkillScore

  • PlotCompetitiveSkillScore

  • PlotTimeSeries

  • PlotAvailability

Note

Bootstrap plots are only created if bootstraps are evaluated.

calculate_test_score(self)

Evaluate test score of model and save locally.

train_ols_model(self)

Train ordinary least squared model on train data.

setup_persistence(self)

Check if persistence is requested from competitors and store this information.

make_prediction(self, subset)

Create predictions for NN, OLS, and persistence and add true observation as reference.

Predictions are filled in an array with full index range. Therefore, predictions can have missing values. All predictions for a single station are stored locally under <forecast/forecast_norm>_<station>_test.nc and can be found inside forecast_path.

_get_frequency(self) → str

Get frequency abbreviation.

_create_competitor_forecast(self, station_name: str, competitor_name: str) → xarray.DataArray

Load and format the competing forecast of a distinct model indicated by competitor_name for a distinct station indicated by station_name. The name of the competitor is set in the type axis as indicator. This method will raise either a FileNotFoundError or KeyError if no competitor could be found for the given station. Either there is no file provided in the expected path or no forecast for given competitor_name in the forecast file. Forecast is trimmed on interval start and end of test subset.

Parameters

  • station_name – name of the station to load data for

  • competitor_name – name of the model

Returns

the forecast of the given competitor

_create_observation(self, data, _, transformation_func: Callable, normalised: bool) → xarray.DataArray

Create observation as ground truth from given data.

Inverse transformation is applied to the ground truth to get the output in the original space.

Parameters

  • data – observation

  • transformation_func – a callable function to apply inverse transformation

  • normalised – transform ground truth in original space if false, or use normalised predictions if true

Returns

filled data array with observation

_create_ols_forecast(self, input_data: xarray.DataArray, ols_prediction: xarray.DataArray, transformation_func: Callable, normalised: bool) → xarray.DataArray

Create ordinary least square model forecast with given input data.

Inverse transformation is applied to the forecast to get the output in the original space.

Parameters

  • input_data – transposed history from DataPrep

  • ols_prediction – empty array in right shape to fill with data

  • transformation_func – a callable function to apply inverse transformation

  • normalised – transform prediction in original space if false, or use normalised predictions if true

Returns

filled data array with ols predictions

_create_persistence_forecast(self, data, persistence_prediction: xarray.DataArray, transformation_func: Callable, normalised: bool) → xarray.DataArray

Create persistence forecast with given data.

Persistence is deviated from the value at t=0 and applied to all following time steps (t+1, …, t+window). Inverse transformation is applied to the forecast to get the output in the original space.

Parameters

  • data – observation

  • persistence_prediction – empty array in right shape to fill with data

  • transformation_func – a callable function to apply inverse transformation

  • normalised – transform prediction in original space if false, or use normalised predictions if true

Returns

filled data array with persistence predictions

_create_nn_forecast(self, nn_output: xarray.DataArray, nn_prediction: xarray.DataArray, transformation_func: Callable, normalised: bool) → xarray.DataArray

Create NN forecast for given input data.

Inverse transformation is applied to the forecast to get the output in the original space. Furthermore, only the output of the main branch is returned (not all minor branches, if the network has multiple output branches). The main branch is defined to be the last entry of all outputs.

Parameters

  • nn_output – Full NN model output

  • nn_prediction – empty array in right shape to fill with data

  • transformation_func – a callable function to apply inverse transformation

  • normalised – transform prediction in original space if false, or use normalised predictions if true

Returns

filled data array with nn predictions

static _create_empty_prediction_arrays(target_data, count=1)

Create array to collect all predictions. Expand target data by a station dimension.

static create_fullindex(df: Union[xarray.DataArray, pandas.DataFrame, pandas.DatetimeIndex], freq: str) → pandas.DataFrame

Create full index from first and last date inside df and resample with given frequency.

Parameters

  • df – use time range of this data set

  • freq – frequency of full index

Returns

empty data frame with full index.

static create_forecast_arrays(index: pandas.DataFrame, ahead_names: List[Union[str, int]], time_dimension, ahead_dim='ahead', index_dim='index', type_dim='type', **kwargs)

Combine different forecast types into single xarray.

Parameters

  • index – index for forecasts (e.g. time)

  • ahead_names – names of ahead values (e.g. hours or days)

  • kwargs – as xarrays; data of forecasts

Returns

xarray of dimension 3: index, ahead_names, # predictions

_get_internal_data(self, station: str, path: str) → Union[xarray.DataArray, None]

Get internal data for given station.

Internal data is defined as data that is already known to the model. From an evaluation perspective, this refers to data, that is no test data, and therefore to train and val data.

Parameters

station – name of station to load internal data.

_get_external_data(self, station: str, path: str) → Union[xarray.DataArray, None]

Get external data for given station.

External data is defined as data that is not known to the model. From an evaluation perspective, this refers to data, that is not train or val data, and therefore to test data.

Parameters

station – name of station to load external data.

_combine_forecasts(self, forecast, competitor, dim=None)

Combine forecast and competitor if both are xarray. If competitor is None, this returns forecasts and vise versa.

calculate_bias_free_error_metrics(self)
calculate_error_metrics(self) → Tuple[Dict, Dict, Dict, Dict]

Calculate error metrics and skill scores of NN forecast.

The competitive skill score compares the NN prediction with persistence and ordinary least squares forecasts. Whereas, the climatological skill scores evaluates the NN prediction in terms of meaningfulness in comparison to different climatological references.

Returns

competitive and climatological skill scores, error metrics

static calculate_average_skill_scores(scores, counts)
static calculate_average_errors(errors)
report_feature_importance_results(self, results)

Create a csv file containing all results from feature importance.

report_error_metrics(self, errors, tag=None)
store_errors(self, errors)