willwheeler/SkForecast_and_Testing
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Page: forecast_comp.py
Pages
Home forecast.py forecast_comp.py forecast_output.json test.php
Clone
4
forecast_comp.py
willwheeler edited this page 2026-06-18 21:42:09 +00:00
Table of Contents

<--- home

forecast_comp.py — Multi-point forecast experiments

Console-only script that compares two backtest models for one target BACnet point:

  • Model A — target history only (same idea as a single point in forecast.py)
  • Model B — target + one or more extra points at each timestamp (exogenous inputs)

Prints MAE for both and whether the extras helped. Does not write JSON and does not change charts.

Use this to decide “should we include FanSts / mode / another RTU point when predicting room temp?” before changing production settings in forecast.py.

Forecast · test.php · Home

On this page

What it is (and is not)

forecast_comp.py forecast.py
Purpose Compare model A vs B Ship forecasts to dashboard
Output Console text only forecast_output.json
Models per run 2 (alone vs combined) 1 per point in JOBS
Exogenous inputs Yes (WITH_POINTS) No (target lags only)
Future prediction No Yes (FUTURE_STEPS)
forecast_comp.py test.php
Runs in browser No Yes
Shows MAE on screen No (terminal only) No (MAE in JSON, not UI yet)

Prerequisites

Same stack as forecast.py:

pip install pandas scikit-learn skforecast

Sample data file referenced by JS_FILE must exist (default: js/rtu-sample.js).

Run from project root:

python forecast_comp.py

Quick run

  1. Open forecast_comp.py

  2. Set JS_FILE, TARGET, WITH_POINTS, LAGS

  3. Run:

     python forecast_comp.py

  4. Read MAE lines and Warnings before drawing conclusions

Configuration

All settings are at the top of the file (no CLI args).

Constant Default (example) Meaning
JS_FILE "js/rtu-sample.js" Sample JS with var xxxData = { ... };
TARGET "RaTmp" Point you want to predict
WITH_POINTS ["FanCmdOvr"] Extra series fed into model B only
LAGS 26 How many past target readings each step uses

Rules:

  • WITH_POINTS must not be empty — script exits if the list is blank
  • Every name in TARGET and WITH_POINTS must exist as a key in the JS file
  • Names are exact BACnet keys (case-sensitive), same as in sample / getData JSON

Changing the experiment

JS_FILE = "js/rtu-sample.js"
TARGET = "ServRmTmp"
WITH_POINTS = ["FanSts", "HVACMode"]
LAGS = 25

Re-run for each combination you want to compare. One run = one target + one set of extras.

Algorithm step by step

1. Load sample JS

Same parser as forecast.py: read file, regex extract JSON object after =.

2. Build time series

For TARGET and each name in WITH_POINTS:

  • x → pandas datetime (milliseconds)
  • y → float
  • Resample to median step; interpolate gaps
  • Result: one Series per column

3. Align timestamps

df = concat(TARGET, WITH_POINTS).dropna()

Only timestamps where every column has a value are kept.
Console prints: Aligned readings: N

If N is much smaller than raw row count, points may be sampled at different times — alignment dropped rows.

4. Train / test split (80 / 20)

Slice Rows Used for
First 80% *_train Fit models
Last 20% *_test Score MAE (never seen during fit)

Same split logic as forecast.py backtest.

5. Quality checks

check_data_quality() runs before MAE is printed. See Quality warnings.

6. Model A — target only

ForecasterRecursive(estimator=Ridge(), lags=LAGS)
alone.fit(y=y_train)
pred_alone = alone.predict(steps=len(y_test))

Predict the hidden 20% using only past values of TARGET.

7. Model B — target + extras

combined.fit(y=y_train, exog=extras_train)
pred_combined = combined.predict(steps=len(y_test), exog=extras_test)

Same lags on the target; at each step the model also sees the known exog columns for the test period (Fan command, mode, etc.).

8. Compare MAE

mae_alone    = mean_absolute_error(y_test, pred_alone)
mae_combined = mean_absolute_error(y_test, pred_combined)

Lower MAE = better average fit on the held-out tail.

Reading the output

Typical good run

File: js/rtu-sample.js
Target: RaTmp
Also using: FanCmdOvr
Aligned readings: 412

Backtest on last 20% of sample data:
  RaTmp only:                    MAE = 1.24
  RaTmp + [FanCmdOvr]:           MAE = 0.89

Extra points helped — about 0.35 better on average.

Verdict logic

Message Condition
Extra points helped mae_alone - mae_combined > 0.05
Extra points did not help difference < -0.05
About the same between -0.05 and +0.05

The 0.05 threshold avoids calling noise a win on noisy BAS data.

When combined MAE is ~0

Note: combined MAE ~0 — re-read warnings above before celebrating.

Almost always means flat data, duplicate columns, or a trivial relationship — not a production-ready model.

Quality warnings

Printed under Warnings (read before trusting MAE):

Warning Meaning
Test slice is empty Split or alignment left no test rows — fix data or file
TARGET is flat in test slice One value entire window → MAE can be 0 with no skill
TARGET barely moves std < 0.05 in test slice — MAE hard to interpret
'X' is identical to TARGET WITH point duplicates target → fake improvement
'X' is almost the same as TARGET corr > 0.995 — not independent signal
'X' almost fully determines TARGET Each exog value maps to one target → lookup, not forecast
'X' is flat in test slice Exog constant in backtest — model B gets no extra info

Rule: If warnings appear, treat MAE differences as hints for further investigation, not proof to deploy.

Example experiments

RTU room temp vs fan command

JS_FILE = "js/rtu-sample.js"
TARGET = "ServRmTmp"
WITH_POINTS = ["FanSts"]

Question: does knowing fan state improve room temp prediction on the last 20%?

Return air vs fan override

TARGET = "RaTmp"
WITH_POINTS = ["FanCmdOvr"]

(Default-style experiment in the repo.)

Multiple extras (one run)

TARGET = "ServRmTmp"
WITH_POINTS = ["FanSts", "HVACMode", "ServRmTmpSpt"]

Model B uses all listed columns as exog (not the target duplicated — setpoint is a different signal). Watch for warnings if setpoint tracks temp too closely.

Veris power (different file)

JS_FILE = "js/veris-sample.js"
TARGET = "P"
WITH_POINTS = ["Ia"]

Checks whether phase current helps predict total power on the same meter.

vs forecast.py

Question Use
“What will P / Ia / room temp do next 100 steps?” forecast.py → JSON → test.php
“Does adding FanCmdOvr help predict RaTmp?” forecast_comp.py
“What LAGS should we use?” Try both; comp for exog decisions, forecast.py for shipped JSON
“Show accuracy on a chart” Not yet — MAE is console / JSON only

forecast_comp.py does not replace forecast.py. A winning WITH_POINTS list would need to be implemented in a future version of forecast.py (exog support) before the dashboard benefits.

Troubleshooting

Error / symptom Cause Fix
Set at least one name in WITH_POINTS Empty list Add at least one point name
TARGET 'X' not found in … Typo or wrong JS file Grep sample JS for exact key
WITH_POINTS 'X' not found Extra not on device Pick a key that exists in that file
Aligned readings very low Points misaligned in time Normal for sparse points; try related points on same device
Both MAE identical Exog adds no signal Try different WITH_POINTS or longer sample
Combined much worse Exog noise or misalignment Remove extras; check warnings
IgnoredArgumentWarning in console Flat predictions from skforecast Suppressed in script; read MAE + warnings instead

Related

  • forecast.py — forecast.py, JSON schema, full pipeline
  • test.php — where shipped forecasts appear (overlay only)
  • Home — repo overview

forecast_comp.py wiki — multi-point backtest experiments.