# TraderAlign Sync Engine & Interactive Trade Review — Architecture Design Document

| | |
|---|---|
| **Status** | Proposed — not implemented. No code, schema, or UI described in this document exists yet. |
| **Document type** | Architecture design reference |
| **Scope** | MT5 Sync Engine (trades, candles, account telemetry) + Interactive Trade Review ("View Trade") + Replay/Practice + Reflection + Colex behavioural comparison |
| **Created** | 2026-07-27 |
| **Origin** | Synthesized from a multi-session architectural review conducted against the live TraderAlign codebase and schema |
| **Supersedes** | The original "Upload Screenshot" plan for per-trade visual review (not deleted, superseded in direction — see [Background](#background--how-this-idea-evolved)) |

> **Read this document before starting any implementation work on trade replay, MT5 sync, or interactive chart review.** It captures the reasoning, the alternatives that were rejected and why, and the phased plan the team agreed on — not just the conclusion. Implementers should not skip straight to the roadmap without reading the reasoning sections; several of the decisions here (Service vs. EA, WebRequest vs. sockets/DLL, manual-first sequencing) were arrived at by ruling out plausible-looking alternatives, and re-litigating them from scratch later wastes the investigation already done.

---

## Table of Contents

1. [Executive Summary](#executive-summary)
2. [Background — How This Idea Evolved](#background--how-this-idea-evolved)
3. [Vision: The TraderAlign Sync Engine](#vision-the-traderalign-sync-engine)
4. [Architectural Investigation: Why MT5 Cannot Behave Like a Cloud API](#architectural-investigation-why-mt5-cannot-behave-like-a-cloud-api)
5. [MT5 Automation Mechanism Comparison](#mt5-automation-mechanism-comparison)
6. [Recommended Sync Architecture](#recommended-sync-architecture)
7. [Automatic Synchronization — Target User Experience](#automatic-synchronization--target-user-experience)
8. [Data Architecture](#data-architecture)
9. [Interactive Trade Review — "View Trade"](#interactive-trade-review--view-trade)
10. [Practice / Replay System](#practice--replay-system)
11. [Reflection Stage](#reflection-stage)
12. [Colex AI — Three-Source Behavioural Comparison](#colex-ai--three-source-behavioural-comparison)
13. [Why "Sync Engine" Instead of "MT5 Reconciliation" / "Screenshot Upload" / "Manual Import"](#why-sync-engine-instead-of-mt5-reconciliation--screenshot-upload--manual-import)
14. [Development Roadmap](#development-roadmap)
15. [Risks](#risks)
16. [Open Questions](#open-questions)
17. [Conclusion & Architectural Recommendation](#conclusion--architectural-recommendation)

---

## Executive Summary

TraderAlign originally planned a simple **Upload Screenshot** feature so traders could visually capture each trade in the Journal. Through architectural review, a substantially more valuable direction emerged: instead of a static image, TraderAlign can **reconstruct the trade itself** from real historical market data and the trader's own Journal fields — an interactive chart the trader can zoom, pan, and review, rather than a picture that goes stale the moment it's uploaded.

That reconstruction requires historical OHLC candle data TraderAlign does not currently possess anywhere in its schema. Three architectural questions had to be answered before any of this could be scoped:

1. **Where does the candle data come from, without taking on a commercial market-data vendor as a costly, ongoing dependency?**
   Answer: the trader's own MT5 terminal is the best source — it's free, it's tick-accurate to the broker the trader actually used, and TraderAlign already runs a working upload/parse pipeline (`mt5-upload.php`) that this idea extends rather than replaces.

2. **Can this be made fully automatic, so the trader never manually exports or uploads anything again after a one-time setup?**
   Answer: yes, with an honest caveat — MT5 is a desktop terminal, not a cloud API, so "automatic" means "runs unattended once the terminal is left running (locally or on a low-cost VPS)," not "zero local footprint forever." No architecture, however sophisticated, removes that constraint; it is a property of the MetaTrader platform itself.

3. **What's the actual mechanism, once we accept that constraint?**
   Answer: a two-part MQL5 component — a one-time **Script** for historical backfill, and a background **Service** (not an Expert Advisor) for ongoing incremental sync — communicating over standard `WebRequest()` HTTPS calls into TraderAlign's existing PHP ingestion architecture. This is broker-agnostic by construction and requires no DLLs, no raw sockets, and no broker-side API relationships.

This document names that synchronization layer the **TraderAlign Sync Engine** — a foundational platform service, not a feature bolted onto Journal. Once built, it becomes shared infrastructure for Journal (View Trade, Replay), Reports, Analytics, and Colex AI alike, the same way `AnalyticsEngine` already is today.

The full vision — View Trade → Practice/Replay → Reflection → Colex three-source behavioural comparison — is judged **architecturally sound, genuinely differentiated, and worth building**, but large enough that it must be sequenced deliberately. The recommended sequence deliberately does **not** start with automatic MT5 sync, even though that is the eventual goal — see [Development Roadmap](#development-roadmap) for the reasoning.

---

## Background — How This Idea Evolved

The Journal already captures rich structured data for every trade: symbol, timestamps, direction, entry/SL/TP, R:R, strategy, session, confidence/mood/energy, checklist adherence, notes, and outcome. The original plan was to let traders additionally attach a **screenshot** of the trade — a static image capturing what the chart looked like at the time.

Screenshots have real, permanent advantages worth stating plainly, because the direction below does not eliminate them:

- **100% coverage.** A screenshot works for literally any instrument, any broker, any obscure synthetic index a prop firm invents.
- **Zero licensing or data-sourcing risk.** It's the user's own capture of their own screen.
- **Zero ongoing infrastructure dependency.** No data pipeline, no external source, no sync mechanism.

What screenshots cannot do is what triggered this investigation: they're static, they can't be zoomed/panned/re-examined at a different timeframe, they carry no structured data Colex or any other module can reason over, and they do nothing to support the idea that eventually became **Practice/Replay** — simulating the trade decision *before* the outcome is known, for deliberate practice.

The reframing explored across this review: instead of a picture of the trade, reconstruct the **trade itself** — real historical candles, with the trader's own Entry/SL/TP overlaid, in an interactive chart. That single idea cascades into a much larger, more valuable concept once its natural extensions are considered: if the chart is real and interactive, the trader can replay it starting *before* the outcome, make a fresh decision blind, and compare that decision against what they actually did and what they said afterward — turning a passive review feature into an active training and behavioural-analysis system.

This document treats that full cascade — View Trade, Replay, Reflection, Colex comparison — as one connected vision, built on one shared data foundation: the **Sync Engine**.

---

## Vision: The TraderAlign Sync Engine

The Sync Engine is a **foundational platform service**, not a Journal feature. It is responsible for keeping TraderAlign synchronized with a trader's real MT5 environment, and it should be designed from day one to serve every module that benefits from real broker-verified data, not just the trade-review experience that motivated it.

### What it synchronizes

| Category | Examples |
|---|---|
| Trade data | Historical closed trades, newly closed trades, open positions, modifications |
| Market data | Historical candles (backfill), ongoing candles (new closed bars), traded symbols |
| Account telemetry | Balance, equity, margin, drawdown, server time, account currency/leverage |
| Broker metadata | Broker/server name, account number (for matching, not display), connection health |

### Who consumes it

| Module | How it benefits |
|---|---|
| **Journal** | View Trade / Replay draw directly from Sync Engine candle + trade data instead of a manual upload per trade |
| **Reports** | Broker-verified data closes the Stage 4 (Broker Reconciliation) gap that currently sits empty for every account in the live database — see [Data Architecture](#data-architecture) |
| **Analytics (`engine.php`)** | Reconciliation-gated fields (`is_reconciled`, `actual_broker_risk_pct`, slippage, holding time) become populated instead of permanently empty, for accounts that opt into sync |
| **Colex AI** | Richer, verified context — real account equity/drawdown at time of a question, not just self-reported figures |
| **Future modules** | Anything that currently depends on manual statement upload (broker reconciliation) or self-reported account state inherits live data for free once built against the Sync Engine rather than against a one-off pipeline |

### Architectural principle

> **Nothing in this system should be designed as "the MT5 pipeline for Replay."** It should be designed as "the platform's synchronization layer, which Replay happens to be the first major consumer of." This distinction matters concretely: table design, ingestion endpoints, and delta-tracking should not assume Replay is the only caller, or the second consumer (Reports' Stage 4, most likely) will require a rework rather than a reuse.

This mirrors a pattern TraderAlign already applies successfully elsewhere: `AnalyticsEngine::computeFourStageRaw()` is a single source of truth consumed independently by Journal Trades and Reports, specifically so the two surfaces can never drift apart. The Sync Engine should be designed with the same discipline from the outset.

---

## Architectural Investigation: Why MT5 Cannot Behave Like a Cloud API

This section documents the investigation's central finding, because it corrects an assumption that shaped the earliest version of this idea (a "connect MT5" flow imagined as something like OAuth-linking a bank account).

**MetaTrader 5 is a desktop terminal application, not a hosted cloud service.** MetaQuotes does not offer a public API that lets a third-party retail app pull an arbitrary user's account history or market data without something running on that user's own machine, logged into that account, with the terminal active. This is a hard platform constraint, not an engineering gap TraderAlign can architect around:

- There is no MetaQuotes-hosted endpoint a server can call with "give me user X's MT5 history."
- The **MT5 Manager API** does provide full server-side access to account/trade/tick data — but it is granted by *individual brokers* to their own infrastructure partners, not to third-party retail apps, and is scoped to one broker's own MT5 server. Pursuing it would require a commercial relationship with every broker/prop-firm TraderAlign's users trade through — dozens of distinct firms, per the real `prop_firm_name` values already in TraderAlign's data. Not viable for a broker-agnostic consumer product.
- **The `MetaTrader5` Python package** (sometimes proposed as a bridge) communicates with a *running terminal on the same machine* via local IPC — it is not a remote API either. Using it server-side would mean TraderAlign hosting and logging into a live MT5 terminal instance per user, with that user's real broker credentials, on TraderAlign's own infrastructure. This is the architecture of a trade-copier or signal-copying platform, not a data-sync feature, and was explicitly rejected: disproportionate operational and legal exposure (holding live broker sessions/credentials for every user) for what is fundamentally a "read some history" need.

**Conclusion: some component must run on the trader's own side, inside or alongside a running MT5 terminal, for any of this to work — automatically or otherwise.** The best achievable experience is "invisible after a one-time setup, provided the terminal is left running" — not "zero local footprint." Given that many active EA-running retail traders already run MT5 on a small VPS for uptime reasons unrelated to TraderAlign, this is a smaller ask for TraderAlign's actual user base than it would be for a general consumer audience — but it must still be communicated honestly in the product UX, not glossed over as a pure account-linking flow.

---

## MT5 Automation Mechanism Comparison

MetaTrader 5 exposes several distinct mechanisms capable of reaching trade/candle/account data. All of them ultimately call the same underlying terminal functions (`CopyRates()` for candles, `HistorySelect()`/`HistoryDealGetX()` for trade history, `AccountInfoDouble()`/`AccountInfoString()` for account telemetry) — the choice between them is not about *capability*, it is about **fit, resilience, security posture, and long-term maintainability**.

| Mechanism | Nature | Runs without terminal open? | Assessment |
|---|---|---|---|
| **MQL5 Script** | Runs once, on demand (user double-clicks / drags onto a chart) | No | ✅ Correct tool for **one-time historical backfill** — no persistent loop needed for a single bulk export |
| **MQL5 Expert Advisor (EA)** | Attached to one chart, reacts to ticks/timer | No | ⚠️ Works, but semantically wrong tool — EAs are built for "make a trading decision on this chart's ticks," not "sync data in the background." Forcing this role onto an EA (dummy chart, `OnTimer()` only) is a known pattern elsewhere in the MT5 ecosystem, but fights the primitive and reads to the user as an auto-trading robot rather than a background utility |
| **MQL5 Service** | Runs headless in the background, not attached to any chart, own `OnStart()` loop | No (still requires the terminal itself running) | ✅ **Recommended** — this is the MQL5 construct MetaQuotes purpose-built for chart-independent background utilities |
| **`WebRequest()`** | HTTPS call from MQL5 to a pre-whitelisted URL | N/A (transport, not a program type) | ✅ **Recommended transport** — used by the Script and the Service alike |
| **Raw sockets** (`SocketCreate`/`SocketConnect`/etc.) | Persistent TCP connection from MQL5 | N/A | ❌ Not recommended — would require TraderAlign to stand up and maintain a persistent connection-handling server component (a genuinely different class of infrastructure from the rest of the PHP/HTTPS stack), disproportionate to the actual data velocity (trade closes and new bars are low-frequency events, not a streaming workload) |
| **DLL imports** (`#import "x.dll"`) | MQL5 calling native code | N/A | ❌ Not recommended — every data point needed here is already reachable from plain MQL5 with zero DLL. A DLL import buys no functional capability and costs: a much bigger, more legitimately alarming user trust ask ("enable DLL imports on my live trading terminal"), platform/build fragility, many prop firms explicitly flag DLL usage, and a native-code artifact requiring a skillset (build, sign, version, maintain across MT5 releases) the rest of TraderAlign's stack has no footprint in |
| **MT5 Manager API** | Broker server-side infrastructure access | N/A | ❌ Not viable — broker-granted, broker-specific, requires a commercial relationship per firm. See prior section |

### Why WebRequest is preferred over the alternatives

`WebRequest()` is the only mechanism that (a) reaches every data point needed, (b) requires no DLL and no persistent server-side connection infrastructure, (c) uses the exact same HTTPS/JSON shape TraderAlign's PHP backend already speaks, and (d) has a single, well-understood, user-controlled security gate (the one-time URL whitelist in terminal settings) rather than a broad, alarming permission like DLL imports.

### Why sockets are unnecessary

The event frequency this system actually needs to handle — a trade closing, a new candle forming — does not require a persistent streaming connection. Periodic polling via a Service's `OnTimer()`, pushing small deltas over `WebRequest()`, is a proportionate match for the real workload and avoids introducing a wholly new server-side architecture (a long-running socket-handling daemon) alongside the rest of TraderAlign's stateless PHP request model.

### Why the solution remains broker-agnostic

`CopyRates()`, `HistorySelect()`, `AccountInfoX()`, and `WebRequest()` are all **terminal-level** APIs — they behave identically regardless of which broker's MT5 server the terminal happens to be connected to. One `.ex5` Service works across every MT5 broker and prop firm in TraderAlign's user base without any broker-specific integration work, which is precisely the property a MT5 Manager API approach or a broker-specific REST integration (e.g. OANDA's own API) would *not* have. The one broker-specific variable is instrument symbol suffixing (`EURUSD` vs. `EURUSD.a` vs. `EURUSDm`) — a normalization problem TraderAlign already has partial groundwork for (`normalizeSymbol()` in the Asset Class Edge Detection feature), and arguably easier to solve here than for a third-party data vendor, since the Service can report the terminal's own exact symbol string alongside every payload rather than requiring guesswork.

---

## Recommended Sync Architecture

```
 One-time setup:        MQL5 Script  →  bulk historical backfill (trades + candles)
                                          chunked WebRequest() calls, per symbol/window

 Ongoing, automatic:    MQL5 Service →  periodic delta sync (OnTimer loop)
                                          new trades / new closed candles / account snapshot
                                          WebRequest() → TraderAlign ingestion endpoint

 Server-side:           PHP ingestion (extends the existing mt5-upload.php pattern)
                            → trade payloads: private, per-user (existing reconciliation model)
                            → candle payloads: shared, deduplicated cache (see Data Architecture)
```

Trade sync and candle sync should be **handled by the same client-side agent, not separated** — both are read-only queries against the same running terminal, and a single sync cycle can gather trades, candles, and account telemetry together into one batched payload (or a small number of typed payloads), giving one place to handle authentication, retry/backoff, and delta-watermark tracking rather than duplicating that logic per data type.

They should, however, be **kept separate server-side**: trade and account data are private and strictly user-scoped (flowing through the same ownership model as today's reconciliation pipeline), while candle data is not user-private — it's raw historical market price, safe to deduplicate into one pool shared across every TraderAlign user regardless of who originally supplied it. This split matters for how the ingestion endpoint authorizes and routes each payload type, and should be a deliberate design decision, not an accident of table layout.

### Deferred: the local bridge application

A separate installed watcher app (distinct from MT5 itself) — either polling the terminal via local IPC, or watching files the Service writes into its own sandboxed `MQL5/Files/` folder and forwarding them with proper retry/offline-queueing — is a real reliability upgrade over a Service calling `WebRequest()` directly. It is **explicitly deferred**, not rejected: it adds a second installed artifact (code-signing, antivirus/SmartScreen trust, cross-platform builds, a second class of "why isn't it running" support ticket) that should only be built in response to an observed reliability problem with the direct-Service approach, not spent on speculatively.

---

## Automatic Synchronization — Target User Experience

The target experience, once built:

1. Trader completes a one-time setup: downloads and runs the TraderAlign backfill Script once, attaches the TraderAlign sync Service, and whitelists TraderAlign's domain in terminal settings (a single MT5 settings action).
2. From that point forward, as long as the terminal (or a VPS running it) stays active, the Service syncs new trades, new candles, and account telemetry on its own timer — no further action from the trader.
3. Journal's **View Trade** works immediately for any trade whose symbol/window has been synced — no per-trade export, no per-trade upload.
4. **Replay** works against the same synced data.
5. **Colex** has real, broker-verified context (equity, drawdown, reconciled outcomes) available whenever relevant, without the trader having to describe their account state manually.
6. If sync ever stops (terminal closed, VPS down, whitelist revoked), this should **fail visibly, not silently** — a "last synced" indicator surfaced in the product (naturally placed alongside Settings/Dashboard) so staleness is obvious rather than discovered only when View Trade unexpectedly has no recent data.

This experience is the **end state**, reached via the phased roadmap below — it is explicitly not proposed as the Version 1 deliverable. See [Development Roadmap](#development-roadmap) for why.

---

## Data Architecture

New, additive-only tables (nothing here modifies `trades`, `trading_accounts`, or any table governed by the existing P&L Rule or Lifecycle Rule):

| Table (proposed) | Purpose | Scope |
|---|---|---|
| `market_data_candles` | `(symbol, timeframe, candle_time, open, high, low, close, volume, source, fetched_at)` — the shared historical/ongoing candle cache | Shared across all users, not per-account. Once a candle exists for a given symbol/timeframe/time, it never needs to be fetched again by anyone. |
| `market_data_symbol_map` | `(internal_instrument, asset_class, provider_or_source, external_symbol, is_supported)` | Normalization layer between TraderAlign's stored `trades.instrument` values and whatever symbol string a given MT5 terminal/broker reports |
| `sync_connections` | Per-user record of an active Sync Engine connection: connection status, last successful sync timestamp per data type, terminal/account identifier (for matching, never for holding credentials), broker/server name | Per-user |
| `sync_watermarks` | `(user_id, data_type, scope_key, last_synced_at_or_id)` — the delta-sync bookmark per (user, symbol/timeframe or account) so ongoing Service payloads only ever contain what's new | Per-user |
| `trade_replay_sessions` | `(id, trade_id [FK, read-only reference], user_id, status, replay_start_candle_time, started_at, ended_at)` | Per-user, strictly isolated from `trades` |
| `trade_replay_actions` | `(id, replay_session_id, action_type, action_payload JSON, candle_time, created_at)` — append-only event log, same shape as the existing `mql5_trades` event-log pattern | Per replay session |
| `trade_replay_sessions.reflection_responses` (JSON column) | Structured reflection Q&A, consistent with this schema's existing habit of JSON columns for flexible substructure (`rules`, `warnings_fired`, `edge_snapshot`) | Per replay session |

### Non-negotiable isolation rule

Per TraderAlign's existing **Lifecycle Rule** and **P&L Rule** (`CLAUDE.md`), replay is a hypothetical simulation and must never be capable of writing to `trades`, `challenge_pnl`, `broker_pnl`, `current_balance`, or `lifecycle_state`. Keeping replay data in fully separate tables, linked only by a read-only `trade_id` reference, makes this a structural guarantee rather than a convention that can be accidentally violated by a future change. Given how much of this codebase's history involves P&L-cascade bugs leaking real numbers into the wrong place, this isolation boundary should be treated as a hard constraint during implementation, not a nice-to-have.

### Relationship to Reports' existing Stage 4 gap

`CLAUDE.md` documents that **zero trades in the live database have ever been broker-reconciled** (`is_reconciled = 0` for all 322 real trades as of the last audit), leaving every Stage 4 (Broker Reconciliation) field in the 4-Stage Summary permanently empty. A working Sync Engine — even before Replay is built — would be the first real path to populating that gap, since Sync Engine trade data is exactly what `is_reconciled` was designed to represent. This should be considered a secondary, valuable beneficiary of Phase 2/3 below, independent of the Replay feature that originally motivated this investigation.

---

## Interactive Trade Review — "View Trade"

Replaces the "Upload Screenshot" entry point in the Journal with **View Trade**, available once synced (or manually imported) candle data exists for that trade's symbol/window.

**Experience:**
- Interactive chart (TradingView Lightweight Charts — see rationale below), correct symbol, correct historical candles surrounding the trade.
- Entry marker, Stop Loss and Take Profit rendered as price lines.
- TraderAlign's own Long/Short position visualization and Risk vs. Reward visualization, layered on top of the chart rather than relying on any charting library's built-in drawing tools — this keeps the visual language consistent with the rest of TraderAlign's design system (`assets/tokens.css`) rather than inheriting a third-party look.
- Trade statistics and Journal notes displayed alongside the chart, following the same "config-driven sectioned inspector" pattern already used by Journal Trades' existing Trade Details view.
- Zoom, pan, and timeframe switching, with Entry/SL/TP remaining correctly positioned across timeframe changes (implemented by resampling a single stored base timeframe upward, not fetching/caching every timeframe independently — see below).

### Charting library: TradingView Lightweight Charts (confirmed as the right choice)

Evaluated against klinecharts, ECharts, Chart.js + financial plugin, Highcharts Stock, and uPlot. Lightweight Charts remains the correct pick: purpose-built for exactly this shape of requirement (candlesticks + price lines for SL/TP + series markers for entry + smooth zoom/pan), Apache-2.0 licensed (free, self-hosted, no required attribution), ~45KB, and maintained directly by TradingView with strong documentation and adoption. Note this is a **different product** from TradingView's separate "Advanced Charting Library," which requires application/approval and carries its own usage terms — implementers should confirm they are pulling the `lightweight-charts` open-source package specifically.

### Timeframe switching without redundant fetches

Fetch/cache only the finest granularity actually needed (e.g. M1), and derive higher timeframes (M5, M15, H1, H4, D1) via standard OHLC resampling rather than treating each timeframe as a separate cached/fetched dataset. This reduces both Sync Engine payload volume and cache storage without any loss of accuracy, since resampling from a finer series is lossless for OHLC aggregation.

### Honest limitation to carry into the UI copy

Reconstructed candles reflect the data source's feed (the trader's own broker via Sync Engine, ideally — see below), not necessarily tick-for-tick identical to what the trader's own eyes saw at the exact moment of entry if the data came from any other source. Product copy should say "reconstructed from synchronized market data," not "the exact chart you saw," to preserve trust — this matters less once Sync Engine (rather than a third-party vendor) is the primary source, since at that point the data genuinely is the trader's own broker feed, but the phrasing discipline should hold regardless of source.

### Screenshot upload is not removed

Coverage gaps will exist regardless of data source (exotic instruments, futures/commodities with contract-rollover ambiguity, users who haven't completed Sync Engine setup). **Screenshot upload remains available as a universal, zero-dependency fallback** wherever View Trade's data isn't available — View Trade is the flagship experience for well-covered instruments, not a wholesale replacement that leaves gaps unaddressed.

---

## Practice / Replay System

From the View Trade screen, a trader may select **Practice This Setup**.

**Mechanics:**
- The chart loads approximately 30–100 candles *before* the original trade's entry — the trader sees only the market as it looked at that moment, with no knowledge of what happened next.
- **Future candles must never be sent to the client ahead of the simulated replay "clock" position.** This is a server-enforced constraint, not a UI-layer hide/show — if future candles are present in the browser's memory (even visually hidden), a trader can trivially inspect them via browser devtools and defeat the entire pedagogical premise. The server should only ever serve candles up to the current, validated replay step.
- The trader simulates the trade: Buy, Sell, Wait, Skip, move Stop Loss, move Take Profit, move to Break Even, take partial profits, or manually close — each action recorded as an event in `trade_replay_actions`.
- **No broker orders are ever placed.** Replay is a simulation recorded entirely within TraderAlign, isolated from the trader's real account, real balance, and real trade history — see the [isolation rule](#non-negotiable-isolation-rule) above.

---

## Reflection Stage

After a replay session concludes, the trader is prompted to reflect. Example prompts:

- Why did you take this trade?
- Why did you skip it?
- What confirmation were you waiting for?
- What would you improve?
- What mistake did you make?

Responses are captured as structured data (`reflection_responses` JSON on the replay session), not free text discarded after display — this is what makes Reflection usable as an input to Colex's comparison in the next stage, rather than a purely reflective UI exercise with no downstream value.

---

## Colex AI — Three-Source Behavioural Comparison

The distinguishing capability of this whole feature: Colex compares three genuinely independent sources of information about the same trade setup, rather than analyzing any one of them in isolation.

| Source | Contents |
|---|---|
| **1. Original Journal** | Entry/SL/TP, confidence/emotion, checklist adherence, notes, strategy, real outcome |
| **2. Replay Behaviour** | What the trader actually did during the blind replay — entered earlier/later, waited, skipped, closed early, held longer, moved SL/TP, changed R:R |
| **3. Reflection Notes** | The trader's own explanation, captured immediately after replay |

Colex looks for patterns such as: consistently waiting for stronger confirmation during replay than in the original trade, repeatedly closing winners early during replay, reflection notes that contradict replay behaviour, or measurable improvement in execution discipline over time.

### PHP as the AI Context Builder (reaffirms existing architecture, does not introduce a new pattern)

`CLAUDE.md` already states Colex "never queries the database directly — receives a prepared/bounded context package." This feature does not change that architecture; it is a direct extension of it. PHP is responsible for:

- Querying only the relevant Journal fields, replay actions, and reflection responses for the trade/session in question.
- Computing comparison statistics (timing deltas, R:R deltas, adherence deltas) — the same kind of aggregation-before-AI discipline already used throughout `AnalyticsEngine`.
- Assembling one structured, bounded JSON context object.

Colex's role is reasoning, coaching, explanation, and behavioural pattern recognition **over that prepared context** — never raw database access, never large unbounded trade history dumps. This keeps Colex's cost (`colex_usage.usage_units`) predictable and keeps the three-source comparison logic testable and versionable in PHP rather than opaque inside a prompt.

---

## Why "Sync Engine" Instead of "MT5 Reconciliation" / "Screenshot Upload" / "Manual Import"

These three framings were each considered and rejected as the primary architectural identity for this work, though pieces of each survive as components within the Sync Engine:

- **"It's just an extension of MT5 reconciliation."** Rejected as the framing (not the code) — the existing `mql5_trades`/`broker_reconciliations` pipeline is real, working infrastructure this design deliberately reuses, but naming the new system after it would anchor its scope to trade-history reconciliation alone, when candle sync and account telemetry sync are equally core and have different data-ownership models (shared vs. private) that "reconciliation" as a name doesn't suggest.
- **"It's a screenshot upload replacement."** Rejected — this frames the work as a Journal UI feature, obscuring that its real value (a shared candle cache, broker-verified account data, a delta-sync mechanism) benefits Reports, Analytics, and Colex independent of whether View Trade or Replay ever gets clicked.
- **"It's a manual candle import tool."** Rejected as the end state, though it is explicitly the correct **starting point** — see the roadmap below. Framing the whole system around manual import would undersell the automatic-sync vision and risk the architecture being built in a way that doesn't generalize cleanly to the Service-based automatic path later.

**"TraderAlign Sync Engine"** is the correct framing because it describes what the system *is* (a synchronization and caching layer between TraderAlign and a trader's real trading environment) rather than what triggered building it (a screenshot feature) or which of its consumers ships first (Replay). Table design, ingestion endpoints, and delta-tracking should be built against this broader identity from the start, specifically so Reports' Stage 4 gap, Analytics' reconciliation-gated fields, and any future module can adopt it without a rework.

---

## Development Roadmap

The roadmap below **deliberately reorders** the phase sequence sketched during earlier discussion. Automatic MT5 sync is the correct long-term destination, but it should not be Phase 1 or 2 — it is the riskiest, least controllable part of the whole system (new skillset in MQL5, new support surface, user trust barrier), and every module downstream of the candle cache (View Trade, Replay, Colex comparison) is *identical* regardless of whether candles arrived via manual upload or automatic Service push. Building the manual path first validates the entire shared foundation cheaply before committing to the harder, riskier automatic-sync engineering.

| Phase | Deliverable | Rationale |
|---|---|---|
| **Phase 0 — Sync Engine foundation + manual import** | `market_data_candles` shared cache, `market_data_symbol_map`, and a manual MT5 price-history import flow extending the existing `mt5-upload.php` upload/parse pattern (new export type: candle history, alongside the existing statement history). Scoped initially to forex majors + metals (TraderAlign's own real data shows these dominate: EURUSD and XAUUSD account for the bulk of trades in the live database). | Proves the entire ingestion → cache → consumption pipeline end-to-end with the lowest possible engineering risk and zero new skillset requirement. Everything built here is reused unchanged by every later phase. |
| **Phase 1 — Interactive View Trade (Stage 1)** | Chart rendering, Entry/SL/TP overlay, zoom/pan/timeframe switching, trade stats + notes panel, screenshot fallback for uncovered instruments. | Ships real user-facing value immediately on top of Phase 0's data, without waiting on MT5 automation. |
| **Phase 2 — Automatic trade synchronization** | MQL5 Script (backfill) + MQL5 Service (ongoing delta sync) for trade/account data, `WebRequest()` transport, `sync_connections`/`sync_watermarks` tables. Positioned as a Pro/Elite-tier convenience layer (fits the existing `feature_flags`/`tier_quotas` tier-gating infrastructure, consistent with `mt5-upload.php` already being Pro+-gated). | Only attempted once Phase 0/1 prove the foundation is sound and worth automating. |
| **Phase 3 — Automatic candle synchronization** | Extends the Phase 2 Service to also sync historical + ongoing candles, feeding the same Phase 0 cache. | Shares the Phase 2 client-side agent — not a separate mechanism (see [Recommended Sync Architecture](#recommended-sync-architecture)). |
| **Phase 4 — Replay / Practice engine** | `trade_replay_sessions`, `trade_replay_actions`, server-enforced future-candle isolation, the Buy/Sell/Wait/Skip/SL-TP-BE/partial-close/manual-close simulation UI. | Depends on Phase 0/1's chart infrastructure and (ideally) Phase 2/3's broader data coverage, though it can function on Phase 0's manually-imported coverage alone if sequenced earlier. |
| **Phase 5 — Reflection capture** | Structured post-replay reflection Q&A. | Small, low-risk, depends only on Phase 4. |
| **Phase 6 — Colex three-source behavioural comparison** | PHP context-builder assembling Journal + Replay + Reflection into bounded JSON; new Colex prompt/reasoning surface. | The highest-value, most differentiated capability — and entirely dependent on Phases 4/5 producing trustworthy data, which is why it is sequenced last. |
| **Phase 7 — Platform-wide Sync Engine adoption** | Reports' Stage 4 (Broker Reconciliation) populated for synced accounts; `AnalyticsEngine`'s reconciliation-gated fields (slippage, actual broker risk, holding time) become live instead of permanently empty; any future module built against Sync Engine data directly. | Realizes the "foundational platform service" vision stated at the outset — not a new feature, but the payoff of having built Phases 0–3 as shared infrastructure rather than Replay-specific plumbing. |

---

## Risks

**MT5 platform limitations**
- No true "zero footprint" automation exists; the terminal (or a VPS running it) must stay active for any sync to occur. This is a permanent platform constraint, not a solvable engineering problem.
- `WebRequest()` requires an explicit one-time user action (URL whitelisting) that cannot be automated for security reasons by MetaQuotes' own design.
- MQL5 syntax and available functions drift across terminal builds — the Script/Service will need ongoing maintenance and periodic revalidation against new MT5 releases.

**User trust and adoption**
- Enabling Algo Trading and whitelisting a URL on a live trading terminal is a materially bigger trust ask than a file upload. Expect a meaningful fraction of cautious or prop-firm-restricted users to decline regardless of how safe the mechanism actually is — the manual-import path (Phase 0) must remain permanently available, not just as a bridge to automation.
- Some prop firms explicitly restrict EA/DLL/external-communication usage during evaluation phases. A read-only sync Service is a different category from trading automation or signal copying, but this distinction will need clear disclosure, and some users on some firms may be told not to use it regardless.
- Distribution trust of the `.ex5` itself — a downloaded executable run inside a trading terminal is a legitimate thing for a cautious trader to be wary of. Consider open-sourcing the Script/Service or distributing through MetaQuotes' own MQL5 Market/CodeBase rather than a bare download link.

**Synchronization failures**
- If sync silently stops (terminal closed, VPS down, whitelist revoked, broker disconnection), View Trade/Replay data goes stale without any obvious signal to the user unless a "last synced" health indicator is explicitly built and surfaced (see [Automatic Synchronization](#automatic-synchronization--target-user-experience)).

**Broker and instrument differences**
- Symbol suffix variation across brokers (`EURUSD` vs. `EURUSD.a` vs. `EURUSDm`) requires an ongoing normalization layer (`market_data_symbol_map`).
- Commodities and index futures (e.g. `coffee`, `natgas`, `ger40` in TraderAlign's own real trade data) often involve contract-rollover ambiguity — there is frequently no single clean continuous historical series for these instruments, unlike spot forex. This is a data-quality problem independent of sync mechanism and should inform which instruments Phase 0/1 launch with.

**Historical data quality**
- Even broker-sourced candles are only as good as that specific broker's feed; minor discrepancies at fast-moving bars are expected and should be reflected honestly in UI copy rather than presented as ground truth.

**Maintenance of MQL5 code**
- MQL5 is a genuinely separate skillset from TraderAlign's PHP/MySQL stack, with weaker tooling (no real unit-testing framework, limited debugging support relative to PHP). The Script/Service becomes a permanent, distinct maintenance line item, not a one-time build.

**Security**
- No DLL imports required anywhere in the recommended design — deliberately, to avoid the security posture and prop-firm-flagging risk that comes with them.
- Sync payloads must never carry broker login credentials; the Sync Engine's authentication with TraderAlign should be a TraderAlign-issued token/API key scoped to that connection, entirely separate from the trader's MT5 login.

**Scalability**
- The shared `market_data_candles` cache is the mechanism that keeps this scalable — marginal cost of an additional user viewing an already-cached symbol/window is a database read, not new data acquisition. This property should be preserved carefully in implementation; any design that accidentally scopes candle storage per-user instead of shared would silently reintroduce the cost/scale problem this whole architecture exists to avoid.

**Support burden (new category for TraderAlign)**
- "Why isn't my data syncing" becomes a debugging conversation about the user's local terminal/VPS state — a class of support ticket TraderAlign has not previously had to handle, since everything else in the product today is fully server-side and within TraderAlign's own control.

---

## Open Questions

These should be investigated or explicitly decided before implementation begins on the relevant phase:

1. **Market data provider terms for Phase 0's cold-start seed.** If a bulk free historical-data source (e.g. Dukascopy- or HistData-style downloads) is used to pre-seed the cache for launch instruments, its terms need an actual legal read for "app-hosted, served to end users" use, not just "personal offline backtesting" use — these are commonly different tiers even for nominally free data.
2. **Broker MT5 ToS review.** A lightweight but real check of typical retail-broker terms regarding a customer exporting their own historical account/price data into a third-party analytics tool — expected to be low-risk (see the architectural investigation above) but not yet formally verified.
3. **Prop firm policy variance.** Should TraderAlign maintain a list of prop firms known to restrict EA/WebRequest usage, and gate or warn accordingly? Or leave this entirely to the trader's own judgment/disclosure?
4. **Instrument launch scope for Phase 0/1.** Confirmed as forex majors + metals based on TraderAlign's own real trade data — should this be revisited once real usage data from Phase 0 exists, or held fixed until Phase 2/3?
5. **Symbol mapping ownership.** Should `market_data_symbol_map` be manually curated by TraderAlign, crowd-sourced from users' own Sync Engine connections (the Service can report the terminal's literal symbol string), or both?
6. **Replay session pricing/tier placement.** Should Practice/Replay be a free-tier capability (to drive adoption and generate the reflection data Colex needs) or gated to Pro/Elite alongside automatic sync? This has direct implications for how quickly the crowd-sourced candle cache and reflection dataset both grow.
7. **Local bridge application — build trigger.** What specific reliability threshold (e.g. sync failure rate, support ticket volume) should trigger investing in the deferred local-bridge-app upgrade described in [Recommended Sync Architecture](#recommended-sync-architecture)?
8. **Retention/pruning policy for `trade_replay_actions`.** Given it's an append-only event log, does it need a retention policy at scale, or is data volume expected to stay small enough (bounded by real replay session count, not tick-level granularity) that no pruning is needed for the foreseeable future?
9. **Colex cost model for three-source comparison.** What's an acceptable `colex_usage.usage_units` budget per replay-comparison call, and does this need its own quota separate from Colex's existing general chat usage?
10. **Code-signing and distribution channel** for the `.ex5` Script/Service — self-signed, TraderAlign-signed, or distributed via MQL5 Market/CodeBase for built-in trust signaling?

---

## Conclusion & Architectural Recommendation

As lead architect for this proposal, having evaluated technical feasibility, user experience, scalability, maintainability, cost, and long-term product value together, my recommendation is:

**Yes — this should become a future direction for TraderAlign, built exactly in the phased order above, with the Sync Engine treated as core platform infrastructure rather than a Journal feature.**

The reasoning, weighed honestly against the alternative of not pursuing this at all:

- **Technical feasibility is real, not speculative.** Every piece of this — the shared candle cache, the MQL5 Script/Service split, `WebRequest()` transport, the replay isolation model, the Colex context-builder pattern — either directly extends infrastructure TraderAlign already runs in production (`mt5-upload.php`'s upload/parse pipeline, `AnalyticsEngine`'s single-source-of-truth pattern, Colex's prepared-context architecture) or is a well-understood, proven pattern from elsewhere in the trading-tools ecosystem. Nothing here requires an unproven technology bet.
- **The cost profile is fundamentally different from the commercial-data-vendor architecture originally assumed.** Recurring cost is near-zero by design: the shared cache means acquisition cost is paid once per (symbol, timeframe, time window) across the entire user base, not per user or per request, and the primary data source (traders' own MT5 terminals) has no licensing fee at all.
- **The user experience payoff compounds rather than plateaus.** Unlike most features that deliver a fixed amount of value, the Sync Engine's shared cache gets *more* valuable and *cheaper to serve* as adoption grows — a genuinely rare property, and one worth designing deliberately around rather than discovering by accident.
- **It is differentiated in a way that's hard to copy quickly.** The three-source Colex comparison (Journal vs. Replay vs. Reflection) depends on TraderAlign's existing rich checklist/emotion/strategy-adherence data model already being as deep as it is — a competitor without that foundation cannot replicate Phase 6 by simply adding a chart.
- **The risk is real but bounded, provided the roadmap is followed as sequenced.** The riskiest component (automatic MT5 sync via MQL5) is deliberately not on the critical path to shipping real user value — Phases 0–1 deliver View Trade on manually-imported data alone, meaning the harder, less controllable engineering (Phases 2–3) can be pursued without having bet the whole feature on it succeeding on the first attempt.

The one condition attached to this recommendation: **do not build Phases 2 and 3 (automatic MT5 sync) before Phase 0 and 1 are shipped and validated with real users.** The temptation, given that "fully automatic sync" is the headline vision, will be to start there. Resist it. Every module in Phases 4–7 depends on the same underlying candle cache and ingestion architecture regardless of whether that cache was filled by a manual import or an automatic Service — building the manual path first is not throwaway work in service of a later "real" version, it *is* the real foundation, exercised the cheap way first.
