Model Evaluation

Overview

This guide systematically evaluates weather model performance for energy market applications, emphasizing quantitative validation and operational testing.

Evaluation Steps

Focus on metrics most relevant to your operations, such as power or price forecasts powered by Jua. If starting with weather data, prioritize real-life observations over analysis datasets like ERA5, as these often contain internal biases.

Step 1: Model Assessment

1. Review model performance reports (EPT-1.5, EPT-2)

2. Decision: Proceed with evaluation.

We recommend starting with our technical report to understand the model's capabilities before proceeding with your own evaluation.

Step 2.1: Evaluate Your Impact Forecast

1. Generate generation or demand forecasts using model data

2. Compare with your existing provider

3. Decision: Proceed to trade evaluation.

Step 2.2: Observations Benchmark

1. The best results are usually achieved by generating an impact forecast, but if you aim to have a weather comparison, we recommend comparing initial performance vs. station observations

2. Evaluate basic accuracy metrics

3. Decision: Continue to detailed testing?

Step 3: Trading Signal Test and PnL Backtest

1. Simulate historical market scenarios and strategies and conduct paper trading tests.

2. Decision: Is ROI there?

Example: Weather Model Testing Framework for Spot Markets

Approach 1: Generation or Demand Forecast

Prerequisites:

• Existing operational forecast using the current forecast provider

• New Impact forecast powered by Jua

Key Metrics:

• Wind Power Forecast accuracy (12-36hr horizon)

• Solar Power Forecast accuracy (12-36hr horizon)

• Demand forecast accuracy (12-36hr horizon)

• Price forecast accuracy (12-36hr horizon)

Evaluation Approach, example Day-Ahead Germany:

1. Jua EPT-1.5 & 3rd party forecast based Wind Power Forecast performance against realized power data in Germany

2. Compare each model performance against actuals

3. Calculate improvement metrics between models

Approach 2: Ground Truth Validation

Prerequisites:

• Weather station data

• Defined forecast window (typically 12-36hr for day-ahead markets)

Testing Protocol:

1. Forecast Evaluation:

   • Compare ECMWF HRES or your preferred model 12-36hr forecasts against ground truth

   • Compare Jua 12-36hr forecasts against ground truth

2. Performance Analysis:

   • Calculate error metrics (RMSE, MAE, bias)

   • Analyze temporal patterns in forecast accuracy

   • Identify systematic biases or errors

Approach 3: Advanced Validation

Option A: Live Strategy Testing & Historical PnL Backtesting

1. Setup:

   • Historical market simulation with trading strategy

   • Trading strategy implementation

2. Analysis:

   • Calculate theoretical PnL

   • Stress test under different market conditions

   • Sensitivity analysis to forecast errors