The Walk-Forward Loop performs the back-testing, but how do we go about interpreting its results? The very nature of the walk-forward loop makes the test data set approach (test the performance on a left-out portion of the data) unattractive.

The answer is quite simple actually: apply standard machine learning techniques to the actual vs the forecasted results and quantify the success of the classification. The extra step here is loading the data.

import numpy as np
import pandas as pd

from sqlalchemy import create_engine

def get_models(db_url):
    ee = create_engine(db_url)
    res = pd.read_sql_table("models", ee)
    return res
    
def get_forecasts(db_url, model_id):
    ee = create_engine(db_url)
    query = (" SELECT ts,fore,details "
                 " FROM forecasts "
                 " WHERE model = " + str(model_id) + " "
                 " ORDER BY ts ASC")
    res = pd.read_sql_query(query, ee, index_col = ['ts'], parse_dates=['ts'])
    return res

def main():
    # The database URL
    db_url = "sqlite:///ml.sqlite"

    # Load the forecasts
    fore = get_forecasts(db_url, 1)

    # Load the actual data
    file_path = "all_data.bin"
    all_data = None
    with open(file_path, 'rb') as ff:
        all_data = pickle.load(ff)
    ho = all_data['HO2']

    # Concatenate the actual and the forecasted data
    dd = pd.concat([ho['full']['entry'],fore['fore']],axis=1).dropna()
    dd.columns = ['actual','fore']

if __name__ == "__main__":
    main()

One way to quantify the quality of the classification is to apply scikit learn’s functions to the loaded data:

cm = sklearn.metrics.confusion_matrix(dd['actual'].as_matrix(), dd['fore'].as_matrix(), labels=[-1,0,1])

For more advanced analysis we can utilize the information from the details column. Remember, this column contains JSON-encoded detailed information about the forecasts. Namely, the class probabilities for each forecast. It could be useful to answer questions like “Are forecasts with higher probability improving the classification”.