Project Iris trades real money. A change to it has to be right before it goes live, and it has to stay right afterwards. Between 23 September and 4 October we shipped twenty-one updates. This post describes the path each of them took, and the alarms that watch the live account around the clock.
Before a change goes live
1. A rule, decided on evidence
A change to how Iris trades starts with a decision based on evidence: the real trades involved, the engine's own logs for those trades, and where possible a replay of recent trades with and without the change. Some proposals end here. In this period several ideas, including a limit on the whole trading day and faster exits for index trades, were replayed and dropped because they did not do better.
2. Written with tests that pin the rule
Every change comes with tests that state the rule in plain terms, such as "a break-even stop is never moved back" or "a brief spread spike changes nothing". The tests check the rule itself, not just that the code runs.
3. Reviewed by someone else, who tries to break it
A different reviewer checks every change. Besides reading it, the reviewer deliberately breaks the new code in small ways, for example by removing a safety check, and confirms that the tests then fail. If a test still passes when the code it protects is broken, it is not protecting anything.
4. Built together, checked as one
Approved changes are combined and built as a single update. Two changes that each pass on their own can still clash once combined, and this step has caught exactly that more than once. The combined build has to pass the full set of automated checks and the full test suite, and it has to compile without new warnings. The exact version that passed is the version that goes live. If anything else changes in the meantime, the build is made and checked again.
5. Live in a quiet window
Updates go live when the market is quiet or closed, and never in the run-up to major news. For example, on 2 October no update was allowed in the hours around the US jobs report.
After a change goes live
6. Checked after the restart
After each update we confirm three things: the program running on the trading server matches the build that passed, file by file; the engine's safety settings for the day came back exactly as they were; and the logs show no errors.
7. Watched around the clock
A set of live alarms runs several times an hour. Each one checks a behaviour, not just whether the engine is running. Most were added after a specific mistake, so that the same mistake cannot happen again unnoticed. Among them:
- Sizing: new trades open at the size they were meant to have. Each order carries the stop its size was calculated on. The engine and your dashboard agree on your capital, and no capital cap has come back.
- Recovery: every recovery leg is funded. A leg removed while its cluster was safe comes back if the cluster is at risk again. A fixed recovery plan keeps every planned leg.
- Stops: a break-even or trailing stop is not being refused over and over. A stop that has been committed is never moved away from price.
- News and FX: trades in affected instruments are tightened around high-impact news. FX orders removed for overlapping exposure come back.
- Health: the day's allowance is funded. Every engine's logs are flowing. The engine is not refusing trades when nothing has told it to stop. The programs running are the ones that were checked. Every family places at least one trade a day.
8. A scoreboard per build
Each build is scored on the trades it opened, so every update can be compared with the one before it on real trades. A score based on too few trades is flagged and is not read as a trend. With updates going live this often, most builds have only traded for a short time. These scores are for our own use, and they describe how a build behaved on a small sample of past trades. Past performance is not a reliable indicator of future results. The performance figures we publish are on the performance page, calculated with their own documented method.
Each alarm is a small, plain check. Here are two, simplified. They show the shape of the checks, not the production code.
def check_allowance_funded(engine):
"""A running engine must have a funded daily allowance."""
if engine.is_running() and engine.trading_day_open():
allowance = engine.daily_allowance()
if allowance is None or allowance <= 0:
raise Alarm("allowance_unfunded", engine=engine.name)
def check_order_sized_on_its_stop(order, tolerance):
"""The size must come from the stop the order actually carries."""
loss_at_stop = order.lots * abs(order.entry - order.stop) * order.value_per_point
if abs(loss_at_stop - order.intended_loss) > tolerance:
raise Alarm("size_does_not_match_stop", order=order.id)
Both checks started life as incidents: the first after the empty-allowance mornings described below, the second after lots were calculated on one stop and sent with another.
When it went wrong
Two of the alarms exist because of mistakes made in this period, and we would rather tell you about them than not.
- 24 and 25 September: on both days, an update made during the day left the engine with an empty daily allowance, so it opened no new trades for several hours. Open trades kept their stops throughout. Both causes were fixed on the same day, and an alarm now checks around the clock that a running engine has an allowance to trade with.
- 24 September: one update briefly installed an outdated build of the program. It was replaced within minutes. The check after every restart now compares each running program with the build that passed.
One rule, one place
Many of the defects fixed in this period had the same shape. One fact, such as when the trading day ends or how big a trade should be, was defined in two places, and the two slowly drifted apart. Every rule in Iris now has exactly one definition, and every part of the engine reads that one. New rules are added without on/off switches or parallel "shadow" versions, so what these posts describe is what runs.
What you need to do
Nothing. If something in your dashboard ever looks inconsistent with what is described here, please tell Support. The alarms are there so that reports like that are rarely needed, and when they miss something, we want to hear about it.
Notes
- Internal references: the live alarms are invariants INV1 to INV21; the restart fixes are #427 and #442.