Update forecast_comp.py

2026-06-18 21:25:33 +00:00
parent 58af35ea3e
commit e39120ec56
1 changed files with 283 additions and 1 deletions
@@ -1,3 +1,285 @@
 <--- [[home]]

-will finish tomorrow
+# forecast_comp.py — Multi-point forecast experiments
+
+Console-only script that compares two backtest models for **one target BACnet point**:
+
+- **Model A** — target history only (same idea as a single point in `forecast.py`)
+- **Model B** — target + one or more **extra points** at each timestamp (exogenous inputs)
+
+Prints **MAE** for both and whether the extras helped. **Does not write JSON** and **does not change charts**.
+
+Use this to decide *“should we include FanSts / mode / another RTU point when predicting room temp?”* before changing production settings in `forecast.py`.
+
+[[Forecast]] · [[test.php]] · [[Home]]
+
+---
+
+## On this page
+
+- [What it is (and is not)](#what-it-is-and-is-not)
+- [Prerequisites](#prerequisites)
+- [Quick run](#quick-run)
+- [Configuration](#configuration)
+- [Algorithm step by step](#algorithm-step-by-step)
+- [Reading the output](#reading-the-output)
+- [Quality warnings](#quality-warnings)
+- [Example experiments](#example-experiments)
+- [vs forecast.py](#vs-forecastpy)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## What it is (and is not)
+
+| | forecast_comp.py | forecast.py |
+|---|------------------|-------------|
+| **Purpose** | Compare model A vs B | Ship forecasts to dashboard |
+| **Output** | Console text only | `forecast_output.json` |
+| **Models per run** | 2 (alone vs combined) | 1 per point in `JOBS` |
+| **Exogenous inputs** | Yes (`WITH_POINTS`) | No (target lags only) |
+| **Future prediction** | No | Yes (`FUTURE_STEPS`) |
+
+| | forecast_comp.py | test.php |
+|---|------------------|----------|
+| **Runs in browser** | No | Yes |
+| **Shows MAE on screen** | No (terminal only) | No (MAE in JSON, not UI yet) |
+
+---
+
+## Prerequisites
+
+Same stack as `forecast.py`:
+
+    pip install pandas scikit-learn skforecast
+
+Sample data file referenced by `JS_FILE` must exist (default: `js/rtu-sample.js`).
+
+Run from **project root**:
+
+    python forecast_comp.py
+
+---
+
+## Quick run
+
+1. Open `forecast_comp.py`
+2. Set `JS_FILE`, `TARGET`, `WITH_POINTS`, `LAGS`
+3. Run:
+
+        python forecast_comp.py
+
+4. Read MAE lines and **Warnings** before drawing conclusions
+
+---
+
+## Configuration
+
+All settings are at the **top of the file** (no CLI args).
+
+| Constant | Default (example) | Meaning |
+|----------|-------------------|---------|
+| `JS_FILE` | `"js/rtu-sample.js"` | Sample JS with `var xxxData = { ... };` |
+| `TARGET` | `"RaTmp"` | Point you want to predict |
+| `WITH_POINTS` | `["FanCmdOvr"]` | Extra series fed into model B only |
+| `LAGS` | `26` | How many past target readings each step uses |
+
+Rules:
+
+- **`WITH_POINTS` must not be empty** — script exits if the list is blank
+- Every name in `TARGET` and `WITH_POINTS` must exist as a key in the JS file
+- Names are **exact** BACnet keys (case-sensitive), same as in sample / `getData` JSON
+
+### Changing the experiment
+
+    JS_FILE = "js/rtu-sample.js"
+    TARGET = "ServRmTmp"
+    WITH_POINTS = ["FanSts", "HVACMode"]
+    LAGS = 25
+
+Re-run for each combination you want to compare. One run = one target + one set of extras.
+
+---
+
+## Algorithm step by step
+
+### 1. Load sample JS
+
+Same parser as `forecast.py`: read file, regex extract JSON object after `=`.
+
+### 2. Build time series
+
+For `TARGET` and each name in `WITH_POINTS`:
+
+- `x` → pandas datetime (milliseconds)
+- `y` → float
+- Resample to median step; interpolate gaps
+- Result: one **Series** per column
+
+### 3. Align timestamps
+
+    df = concat(TARGET, WITH_POINTS).dropna()
+
+Only timestamps where **every** column has a value are kept.  
+Console prints: `Aligned readings: N`
+
+If `N` is much smaller than raw row count, points may be sampled at different times — alignment dropped rows.
+
+### 4. Train / test split (80 / 20)
+
+| Slice | Rows | Used for |
+|-------|------|----------|
+| First **80%** | `*_train` | Fit models |
+| Last **20%** | `*_test` | Score MAE (never seen during fit) |
+
+Same split logic as `forecast.py` backtest.
+
+### 5. Quality checks
+
+`check_data_quality()` runs **before** MAE is printed. See [Quality warnings](#quality-warnings).
+
+### 6. Model A — target only
+
+    ForecasterRecursive(estimator=Ridge(), lags=LAGS)
+    alone.fit(y=y_train)
+    pred_alone = alone.predict(steps=len(y_test))
+
+Predict the hidden 20% using only past values of **TARGET**.
+
+### 7. Model B — target + extras
+
+    combined.fit(y=y_train, exog=extras_train)
+    pred_combined = combined.predict(steps=len(y_test), exog=extras_test)
+
+Same lags on the target; at each step the model also sees the **known** exog columns for the test period (Fan command, mode, etc.).
+
+### 8. Compare MAE
+
+    mae_alone    = mean_absolute_error(y_test, pred_alone)
+    mae_combined = mean_absolute_error(y_test, pred_combined)
+
+Lower MAE = better average fit on the held-out tail.
+
+---
+
+## Reading the output
+
+### Typical good run
+
+    File: js/rtu-sample.js
+    Target: RaTmp
+    Also using: FanCmdOvr
+    Aligned readings: 412
+
+    Backtest on last 20% of sample data:
+      RaTmp only:                    MAE = 1.24
+      RaTmp + [FanCmdOvr]:           MAE = 0.89
+
+    Extra points helped — about 0.35 better on average.
+
+### Verdict logic
+
+| Message | Condition |
+|---------|-----------|
+| **Extra points helped** | `mae_alone - mae_combined > 0.05` |
+| **Extra points did not help** | difference &lt; **-0.05** |
+| **About the same** | between -0.05 and +0.05 |
+
+The **0.05** threshold avoids calling noise a win on noisy BAS data.
+
+### When combined MAE is ~0
+
+    Note: combined MAE ~0 — re-read warnings above before celebrating.
+
+Almost always means flat data, duplicate columns, or a trivial relationship — not a production-ready model.
+
+---
+
+## Quality warnings
+
+Printed under **Warnings (read before trusting MAE):**
+
+| Warning | Meaning |
+|---------|---------|
+| **Test slice is empty** | Split or alignment left no test rows — fix data or file |
+| **TARGET is flat in test slice** | One value entire window → MAE can be 0 with no skill |
+| **TARGET barely moves** | std &lt; 0.05 in test slice — MAE hard to interpret |
+| **'X' is identical to TARGET** | WITH point duplicates target → fake improvement |
+| **'X' is almost the same as TARGET** | corr &gt; 0.995 — not independent signal |
+| **'X' almost fully determines TARGET** | Each exog value maps to one target → lookup, not forecast |
+| **'X' is flat in test slice** | Exog constant in backtest — model B gets no extra info |
+
+**Rule:** If warnings appear, treat MAE differences as hints for further investigation, not proof to deploy.
+
+---
+
+## Example experiments
+
+### RTU room temp vs fan command
+
+    JS_FILE = "js/rtu-sample.js"
+    TARGET = "ServRmTmp"
+    WITH_POINTS = ["FanSts"]
+
+Question: does knowing fan state improve room temp prediction on the last 20%?
+
+### Return air vs fan override
+
+    TARGET = "RaTmp"
+    WITH_POINTS = ["FanCmdOvr"]
+
+(Default-style experiment in the repo.)
+
+### Multiple extras (one run)
+
+    TARGET = "ServRmTmp"
+    WITH_POINTS = ["FanSts", "HVACMode", "ServRmTmpSpt"]
+
+Model B uses **all** listed columns as exog (not the target duplicated — setpoint is a different signal). Watch for warnings if setpoint tracks temp too closely.
+
+### Veris power (different file)
+
+    JS_FILE = "js/veris-sample.js"
+    TARGET = "P"
+    WITH_POINTS = ["Ia"]
+
+Checks whether phase current helps predict total power on the same meter.
+
+---
+
+## vs forecast.py
+
+| Question | Use |
+|----------|-----|
+| “What will P / Ia / room temp do next 100 steps?” | **forecast.py** → JSON → **test.php** |
+| “Does adding FanCmdOvr help predict RaTmp?” | **forecast_comp.py** |
+| “What LAGS should we use?” | Try both; comp for exog decisions, forecast.py for shipped JSON |
+| “Show accuracy on a chart” | Not yet — MAE is console / JSON only |
+
+**forecast_comp.py does not replace forecast.py.** A winning `WITH_POINTS` list would need to be **implemented** in a future version of `forecast.py` (exog support) before the dashboard benefits.
+
+---
+
+## Troubleshooting
+
+| Error / symptom | Cause | Fix |
+|-----------------|-------|-----|
+| `Set at least one name in WITH_POINTS` | Empty list | Add at least one point name |
+| `TARGET 'X' not found in …` | Typo or wrong JS file | Grep sample JS for exact key |
+| `WITH_POINTS 'X' not found` | Extra not on device | Pick a key that exists in that file |
+| Aligned readings very low | Points misaligned in time | Normal for sparse points; try related points on same device |
+| Both MAE identical | Exog adds no signal | Try different WITH_POINTS or longer sample |
+| Combined much worse | Exog noise or misalignment | Remove extras; check warnings |
+| `IgnoredArgumentWarning` in console | Flat predictions from skforecast | Suppressed in script; read MAE + warnings instead |
+
+---
+
+## Related
+
+- [[Forecast]] — `forecast.py`, JSON schema, full pipeline
+- [[test.php]] — where shipped forecasts appear (overlay only)
+- [[Home]] — repo overview
+
+---
+
+*forecast_comp.py wiki — multi-point backtest experiments.*