diff --git a/forecast_comp.py.md b/forecast_comp.py.md index 6577d53..2be6aba 100644 --- a/forecast_comp.py.md +++ b/forecast_comp.py.md @@ -1,3 +1,285 @@ <--- [[home]] -will finish tomorrow \ No newline at end of file +# forecast_comp.py — Multi-point forecast experiments + +Console-only script that compares two backtest models for **one target BACnet point**: + +- **Model A** — target history only (same idea as a single point in `forecast.py`) +- **Model B** — target + one or more **extra points** at each timestamp (exogenous inputs) + +Prints **MAE** for both and whether the extras helped. **Does not write JSON** and **does not change charts**. + +Use this to decide *“should we include FanSts / mode / another RTU point when predicting room temp?”* before changing production settings in `forecast.py`. + +[[Forecast]] · [[test.php]] · [[Home]] + +--- + +## On this page + +- [What it is (and is not)](#what-it-is-and-is-not) +- [Prerequisites](#prerequisites) +- [Quick run](#quick-run) +- [Configuration](#configuration) +- [Algorithm step by step](#algorithm-step-by-step) +- [Reading the output](#reading-the-output) +- [Quality warnings](#quality-warnings) +- [Example experiments](#example-experiments) +- [vs forecast.py](#vs-forecastpy) +- [Troubleshooting](#troubleshooting) + +--- + +## What it is (and is not) + +| | forecast_comp.py | forecast.py | +|---|------------------|-------------| +| **Purpose** | Compare model A vs B | Ship forecasts to dashboard | +| **Output** | Console text only | `forecast_output.json` | +| **Models per run** | 2 (alone vs combined) | 1 per point in `JOBS` | +| **Exogenous inputs** | Yes (`WITH_POINTS`) | No (target lags only) | +| **Future prediction** | No | Yes (`FUTURE_STEPS`) | + +| | forecast_comp.py | test.php | +|---|------------------|----------| +| **Runs in browser** | No | Yes | +| **Shows MAE on screen** | No (terminal only) | No (MAE in JSON, not UI yet) | + +--- + +## Prerequisites + +Same stack as `forecast.py`: + + pip install pandas scikit-learn skforecast + +Sample data file referenced by `JS_FILE` must exist (default: `js/rtu-sample.js`). + +Run from **project root**: + + python forecast_comp.py + +--- + +## Quick run + +1. Open `forecast_comp.py` +2. Set `JS_FILE`, `TARGET`, `WITH_POINTS`, `LAGS` +3. Run: + + python forecast_comp.py + +4. Read MAE lines and **Warnings** before drawing conclusions + +--- + +## Configuration + +All settings are at the **top of the file** (no CLI args). + +| Constant | Default (example) | Meaning | +|----------|-------------------|---------| +| `JS_FILE` | `"js/rtu-sample.js"` | Sample JS with `var xxxData = { ... };` | +| `TARGET` | `"RaTmp"` | Point you want to predict | +| `WITH_POINTS` | `["FanCmdOvr"]` | Extra series fed into model B only | +| `LAGS` | `26` | How many past target readings each step uses | + +Rules: + +- **`WITH_POINTS` must not be empty** — script exits if the list is blank +- Every name in `TARGET` and `WITH_POINTS` must exist as a key in the JS file +- Names are **exact** BACnet keys (case-sensitive), same as in sample / `getData` JSON + +### Changing the experiment + + JS_FILE = "js/rtu-sample.js" + TARGET = "ServRmTmp" + WITH_POINTS = ["FanSts", "HVACMode"] + LAGS = 25 + +Re-run for each combination you want to compare. One run = one target + one set of extras. + +--- + +## Algorithm step by step + +### 1. Load sample JS + +Same parser as `forecast.py`: read file, regex extract JSON object after `=`. + +### 2. Build time series + +For `TARGET` and each name in `WITH_POINTS`: + +- `x` → pandas datetime (milliseconds) +- `y` → float +- Resample to median step; interpolate gaps +- Result: one **Series** per column + +### 3. Align timestamps + + df = concat(TARGET, WITH_POINTS).dropna() + +Only timestamps where **every** column has a value are kept. +Console prints: `Aligned readings: N` + +If `N` is much smaller than raw row count, points may be sampled at different times — alignment dropped rows. + +### 4. Train / test split (80 / 20) + +| Slice | Rows | Used for | +|-------|------|----------| +| First **80%** | `*_train` | Fit models | +| Last **20%** | `*_test` | Score MAE (never seen during fit) | + +Same split logic as `forecast.py` backtest. + +### 5. Quality checks + +`check_data_quality()` runs **before** MAE is printed. See [Quality warnings](#quality-warnings). + +### 6. Model A — target only + + ForecasterRecursive(estimator=Ridge(), lags=LAGS) + alone.fit(y=y_train) + pred_alone = alone.predict(steps=len(y_test)) + +Predict the hidden 20% using only past values of **TARGET**. + +### 7. Model B — target + extras + + combined.fit(y=y_train, exog=extras_train) + pred_combined = combined.predict(steps=len(y_test), exog=extras_test) + +Same lags on the target; at each step the model also sees the **known** exog columns for the test period (Fan command, mode, etc.). + +### 8. Compare MAE + + mae_alone = mean_absolute_error(y_test, pred_alone) + mae_combined = mean_absolute_error(y_test, pred_combined) + +Lower MAE = better average fit on the held-out tail. + +--- + +## Reading the output + +### Typical good run + + File: js/rtu-sample.js + Target: RaTmp + Also using: FanCmdOvr + Aligned readings: 412 + + Backtest on last 20% of sample data: + RaTmp only: MAE = 1.24 + RaTmp + [FanCmdOvr]: MAE = 0.89 + + Extra points helped — about 0.35 better on average. + +### Verdict logic + +| Message | Condition | +|---------|-----------| +| **Extra points helped** | `mae_alone - mae_combined > 0.05` | +| **Extra points did not help** | difference < **-0.05** | +| **About the same** | between -0.05 and +0.05 | + +The **0.05** threshold avoids calling noise a win on noisy BAS data. + +### When combined MAE is ~0 + + Note: combined MAE ~0 — re-read warnings above before celebrating. + +Almost always means flat data, duplicate columns, or a trivial relationship — not a production-ready model. + +--- + +## Quality warnings + +Printed under **Warnings (read before trusting MAE):** + +| Warning | Meaning | +|---------|---------| +| **Test slice is empty** | Split or alignment left no test rows — fix data or file | +| **TARGET is flat in test slice** | One value entire window → MAE can be 0 with no skill | +| **TARGET barely moves** | std < 0.05 in test slice — MAE hard to interpret | +| **'X' is identical to TARGET** | WITH point duplicates target → fake improvement | +| **'X' is almost the same as TARGET** | corr > 0.995 — not independent signal | +| **'X' almost fully determines TARGET** | Each exog value maps to one target → lookup, not forecast | +| **'X' is flat in test slice** | Exog constant in backtest — model B gets no extra info | + +**Rule:** If warnings appear, treat MAE differences as hints for further investigation, not proof to deploy. + +--- + +## Example experiments + +### RTU room temp vs fan command + + JS_FILE = "js/rtu-sample.js" + TARGET = "ServRmTmp" + WITH_POINTS = ["FanSts"] + +Question: does knowing fan state improve room temp prediction on the last 20%? + +### Return air vs fan override + + TARGET = "RaTmp" + WITH_POINTS = ["FanCmdOvr"] + +(Default-style experiment in the repo.) + +### Multiple extras (one run) + + TARGET = "ServRmTmp" + WITH_POINTS = ["FanSts", "HVACMode", "ServRmTmpSpt"] + +Model B uses **all** listed columns as exog (not the target duplicated — setpoint is a different signal). Watch for warnings if setpoint tracks temp too closely. + +### Veris power (different file) + + JS_FILE = "js/veris-sample.js" + TARGET = "P" + WITH_POINTS = ["Ia"] + +Checks whether phase current helps predict total power on the same meter. + +--- + +## vs forecast.py + +| Question | Use | +|----------|-----| +| “What will P / Ia / room temp do next 100 steps?” | **forecast.py** → JSON → **test.php** | +| “Does adding FanCmdOvr help predict RaTmp?” | **forecast_comp.py** | +| “What LAGS should we use?” | Try both; comp for exog decisions, forecast.py for shipped JSON | +| “Show accuracy on a chart” | Not yet — MAE is console / JSON only | + +**forecast_comp.py does not replace forecast.py.** A winning `WITH_POINTS` list would need to be **implemented** in a future version of `forecast.py` (exog support) before the dashboard benefits. + +--- + +## Troubleshooting + +| Error / symptom | Cause | Fix | +|-----------------|-------|-----| +| `Set at least one name in WITH_POINTS` | Empty list | Add at least one point name | +| `TARGET 'X' not found in …` | Typo or wrong JS file | Grep sample JS for exact key | +| `WITH_POINTS 'X' not found` | Extra not on device | Pick a key that exists in that file | +| Aligned readings very low | Points misaligned in time | Normal for sparse points; try related points on same device | +| Both MAE identical | Exog adds no signal | Try different WITH_POINTS or longer sample | +| Combined much worse | Exog noise or misalignment | Remove extras; check warnings | +| `IgnoredArgumentWarning` in console | Flat predictions from skforecast | Suppressed in script; read MAE + warnings instead | + +--- + +## Related + +- [[Forecast]] — `forecast.py`, JSON schema, full pipeline +- [[test.php]] — where shipped forecasts appear (overlay only) +- [[Home]] — repo overview + +--- + +*forecast_comp.py wiki — multi-point backtest experiments.* \ No newline at end of file