# Backtesting LLM Strategies

Evaluating model-driven financial strategies against historical data, under strict controls against lookahead.

**Rests on:** Hypothesis Testing, Test Planning

*Applied Domains · AI in Finance*


## Why learn this

You are designing an AI system that gives financial advice. Before you let it suggest a single trade or write a single recommendation, you need to know: does this strategy actually work, or does it just look like it works because of luck or hidden shortcuts? Backtesting is how you find out. It lets you test your strategy on what already happened, before risking a rupee of real money. Master this, and you unlock the discipline of using AI in finance with integrity — ensuring your data reflects reality as it stood at the moment, that your results can be verified by others, and that you do not accidentally use future information to predict the past. Only then can you trust the models you build for analysis, forecasting, and trading support.


## The intuition

Imagine you have an idea for a restaurant. You think: 'If I open a place that serves only south Indian breakfast near a metro station, it will do well.' You can't know for sure until you open it. But you could look at what already happened: where the metro stations are, how many people pass by, what other breakfast places exist, and how crowded they get. That is backtesting — testing your idea against the record of what actually happened, without spending any real money.

Now imagine your idea is a strategy driven by a language model. For example: 'If the earnings call of a company is negative in tone, sell the stock for the next week.' To backtest it, you need to go through the past: for each past earnings call, decide what the model would have said (is the tone negative?), then check what the stock price actually did in the next week. If the strategy would have made money consistently, it might be a good idea. If it would have lost money, you just saved yourself a fortune.

But here is the catch — the record of the past is full of traps. Imagine you are testing that strategy, and you check the year 2021. You look at a company's earnings call in January 2021. The stock price dropped in February. You think: 'Great, the call was negative, and the stock dropped, my strategy works.' But what if the model you are using to judge the tone of the call was trained on data that included February 2021? Then the model might have learned 'this stock dropped after this call' — not because the tone was negative, but because it literally read the answer. You are cheating, and you don't even know it. That is called lookahead, and it is the single biggest trap in backtesting. The whole discipline of backtesting is about closing every trap so that your test is honest.

Another trap is overfitting. This happens when a strategy memorizes the specific quirks of the past data rather than learning a general rule. Imagine a student who crams for an exam by memorizing the exact answers to last year's test questions instead of understanding the subject. They will score 100% on the old questions but fail miserably on any new question. Similarly, a strategy might find a weird pattern in 2020-2021 data that worked purely by coincidence, but fails when applied to real-world situations where that pattern doesn't hold.

To guard against these traps and ensure robustness, we use walk-forward analysis. Instead of testing the strategy on one big chunk of past data, we split history into two parts: a 'learning' part and a 'testing' part. We train the strategy on the learning part, then test it on the immediately following testing part. Then we slide that window forward one step at a time. It is like checking if a driver who learned on Monday's route can successfully drive Tuesday's new route, then Wednesday's, and so on. If the strategy fails to adapt as the data changes, it is not robust, regardless of how well it did in earlier tests.


## An analogy

Think of a driving test. The examiner sits in the passenger seat and watches you drive around a set route. But imagine the examiner had seen you drive the exact same route before the test, and knew exactly where you would make a mistake. That would not be a fair test. The examiner's knowledge is contaminated.

Backtesting a strategy is like a driving test for money. You have a strategy (a driver). The historical data is the route. You want to see: does the strategy drive well? To make the test fair, you must make sure the strategy (and anything it uses to make decisions) has never seen the route before. If the model used to read the earnings call has read the future, the examiner has been tipped off.

This analogy goes deeper. Imagine a driver who knows the test route perfectly — they pass easily. But give them a different route and they crash. That is called 'overfitting' — the strategy works on the test route but fails everywhere else. Just like a driver, a strategy can memorize the past instead of learning a general rule. To catch that, you give the strategy a route it has never seen, and only then do you judge it.

Where the analogy breaks: In a driving test, the route is fixed and the examiner knows the rules. In backtesting, you design the route, the rules, and even the examiner. You have to be careful that you do not accidentally design the test so that your strategy looks good.


## Definition

Backtesting LLM strategies is the process of evaluating a strategy driven by a language model against historical data, under strict controls that prevent the model or the analyst from using information that was not available at the time the decision would have been made (a 'lookahead'), and ensuring the results are robust across different periods of time.


## How it connects

This concept connects to everything you are learning in the AI in Finance branch. It relies on your notes on Large Language Models (how the model works), but more importantly it connects to your notes on Point-in-Time Data (the records must be exactly as they were on the day, not as they are today), Regulatory Compliance Controls (a backtest that cannot be independently verified is not admissible), and Admissible Evidence Spans (every decision the model makes must be traceable to the data it was allowed to see). It also connects to your notes on Financial Sentiment and Earnings Call Analysis — those are the signals an LLM-driven strategy might use. Your notes on Market Risk matter here too, because a backtest that does not account for the risk of large losses is incomplete.


## The mathematics

The mathematics behind backtesting is mostly arithmetic and careful counting, not calculus. The branch of mathematics that governs it is statistics — specifically the ideas of sampling, probability, and hypothesis testing. You are asking: 'If I ran this strategy many times on random chunks of history, how often would it make money purely by chance?' That is a probability question.

The key operation is: take a historical period, divide it into two parts — a 'training' part and a 'testing' part. You let the strategy (or the model) learn from the training part. Then you evaluate it on the testing part, which it has never seen. This is called train/test splitting. The reasoning is straightforward: if a strategy genuinely works, it must work on data it has never seen. If it only works on the data it learned from, that is evidence of overfitting, not evidence of skill.

A second key operation is walk-forward testing. Instead of one split, you do many. You test on the first month, then roll forward and test on the second month, and so on. Each time, the model may be allowed to re-learn from all the data before that month. This simulates what would happen if you ran the strategy in real-time, re-training as you go. The result is a series of test results — one for each month — that you can count up: how many months were profitable? What was the average profit? And crucially, how many months had large losses? This is real arithmetic you can do by hand.

Let profit_i be the profit for month i in the test set. The total backtest profit is:

P = Σ profit_i

over all test months i.

The hit rate is:

H = (number of months with profit_i > 0) / (total number of months)

The average profit is:

A = P / N

where N is the number of test months.

The maximum drawdown (worst peak-to-trough loss) is found by tracking the cumulative profit C_i = C_{i-1} + profit_i, and then finding the largest drop from any peak to a later trough.

Each symbol: P is total profit, profit_i is one month's profit, H is hit rate, A is average monthly profit, C_i is the cumulative profit up to month i.


## A worked example

```
Imagine the year 2022, with 12 monthly periods. Let's say your LLM strategy decides at the start of each month whether to be 'in the market' (owning a certain stock) or 'out of the market' (holding only cash). At the end of each month, you record the profit or loss.

Let's assume you train the model on the first 6 months (January to June). Then you test it on July. You record the result. Then you re-train on January through July, and test on August. And so on. This is walk-forward testing.

Step 1: Train on Jan–Jun, test on Jul → profit: +2%   # The model was trained on the first half of the year, then applied to July. It made 2%.
Step 2: Train on Jan–Jul, test on Aug → profit: -1%   # The model was re-trained with July included, then applied to August. It lost 1%.
Step 3: Train on Jan–Aug, test on Sep → profit: +0.5%   # Now with August included, tested on September. It made half a percent.
Step 4: Train on Jan–Sep, test on Oct → profit: +1.5%   # With September included, it makes 1.5% on October.
Step 5: Train on Jan–Oct, test on Nov → profit: -0.5%   # In November it loses half a percent.
Step 6: Train on Jan–Nov, test on Dec → profit: +3%   # In December it makes a solid 3%.

Result: Total: 2% - 1% + 0.5% + 1.5% - 0.5% + 3% = +5.5% for the second half of the year. A positive result, but not spectacular. You also need to see that there were two losing months (August and November), which means the strategy is not a guaranteed winner. This arithmetic is the basis of all backtesting.

Now consider a naive backtest without walk-forward: you train on all of 2022 and then test on all of 2022. The model has already seen the price of December, so it might 'predict' December's rise perfectly — not because it understands the market, but because it memorized it. That is the lookahead trap in action.
```


## From scratch, in Python

Here is plain Python to walk-forward test a strategy. No external libraries.

```python
# Step 1: A list of monthly profits, one per month, for all of 2022.
# These numbers are hypothetical results of a strategy applied to each month.
monthly_profits = [1.0, -0.5, 0.8, 0.2, -1.2, 0.5, 2.0, -1.0, 0.5, 1.5, -0.5, 3.0]

# Step 2: We say the first 6 months are training data.
# In a real implementation, the model would be trained here.
# We do not use these months for testing.
train_months = 6

# Step 3: Extract the test months only.
test_profits = monthly_profits[train_months:]  # This is a list: [2.0, -1.0, 0.5, 1.5, -0.5, 3.0]

# Step 4: Loop over each test month, accumulating total profit.
total = 0.0
for p in test_profits:
    total = total + p

# Step 5: Count how many test months were profitable.
count = 0
for p in test_profits:
    if p > 0:
        count = count + 1

# Step 6: Compute hit rate.
hit_rate = count / len(test_profits)

# Step 7: Compute average monthly profit.
average = total / len(test_profits)

# Step 8: Compute cumulative profit and maximum drawdown.
cumulative = []
running = 0.0
for p in test_profits:
    running = running + p
    cumulative.append(running)

max_drawdown = 0.0
peak = cumulative[0]
for value in cumulative:
    if value > peak:
        peak = value
    drawdown = peak - value
    if drawdown > max_drawdown:
        max_drawdown = drawdown

# Print the results
print("Total profit:", total)
print("Hit rate:", hit_rate)
print("Average monthly profit:", average)
print("Maximum drawdown:", max_drawdown)
```

Output:
```
Total profit: 5.5
Hit rate: 0.6666666666666666
Average monthly profit: 0.9166666666666666
Maximum drawdown: 1.5
```


## The same thing, with a framework

In practice, you will not loop like this by hand. You will use a library like `backtrader`, `vectorbt`, or `pandas`.

For example, `vectorbt` can compute all these metrics in a few lines:

```python
import vectorbt as vbt

profits = vbt.Indicator.run(monthly_profits)
# ... more complex in practice
```

What it does underneath is exactly the loop you just wrote: it accumulates profits, counts wins, computes averages, and finds the maximum drawdown. The library is just a convenience wrapper around arithmetic.


## What the code does

The code starts by defining a list called `monthly_profits`. It contains 12 numbers — the result of a strategy for each month of the year. The first 6 are marked 'training' by setting the variable `train_months = 6`. The line `monthly_profits[train_months:]` is a list slice — it takes the elements from index 6 to the end. Python lists are zero-indexed, so the first element is at position 0, and position 6 is the 7th month (July).

The `for` loop is a Python construct that walks through each item in the list. For each item, it adds it to `total`. The second loop counts how many items are positive, which is the hit rate. `len(test_profits)` gives the number of items in the list, so dividing the count by that gives a fraction.

The final loop tracks the running cumulative sum, stored in the `cumulative` list. Then a second pass over that list finds the peak and the largest drop from a peak — that is the maximum drawdown. This is the worst stretch of losing money in the test period, which matters in any financial strategy. The `print` statements output the results so you can see them plainly.

The framework version hides all this arithmetic, but it is the same math underneath.


## Common misconceptions

**Language models are trained on text, so they don't know prices** — The model looks at today's prices, but that's fine because it's a language model, not a trader. It just reads text and outputs text. It doesn't know the stock price, so my backtest can't have lookahead.

Language models are trained on text scraped from the entire internet at a point in time. That text may contain a news article from December 2022 that mentions the stock price from November 2022. When you ask the model to evaluate a January 2022 earnings call, you must ensure the model's internal knowledge does not include December 2022 data. If it does, the model 'knows' the future. In fact: You must control the model's knowledge. Use a model with a clear training cutoff, and verify that the text you feed it in the backtest is exactly what it would have seen on that day. This is what your notes call point-in-time discipline.

**Transaction costs don't matter much** — I don't need to worry about transaction fees and slippage; my backtest showed 150% profit, so I can buy my new flat.

In a realistic market, every trade has a cost: the spread between the buy price and the sell price, brokerage fees, and slippage from moving the price yourself. If your backtest ignores these, you can easily see profits that vanish once real costs are applied. In fact: The backtest must subtract a realistic estimate of transaction costs and slippage from each trade. The discipline in AI in Finance demands that 'costs are fiction' if they are not modelled. A backtest that ignores costs is a fiction.

**Training and testing on the same data is acceptable** — My model is amazing — I trained it on 5 years of data and then tested it on the same 5 years, and it was right 90% of the time!

This is the textbook definition of overfitting. The model has simply memorised the price history. It will likely fail catastrophically on new data. In fact: The minimum discipline is out-of-sample testing: split the data, use the first part solely for training and the second solely for testing, and never look at the test results while you are tuning the model. Walk-forward is even better. This is called 'out-of-sample' testing in your notes.

**A profitable backtest proves the strategy works** — My walk-forward backtest gave me a 5.5% return over six months, so my strategy is guaranteed to make money when I go live.

The return is within a range of luck. Even a random strategy has a 20% chance of making money over six months. You need to measure how much the strategy could lose (maximum drawdown) and how consistent the gains are (hit rate). In fact: You need to report not just the profit, but the drawdown, the hit rate, and ideally a statistical test (like a t-test on the monthly profits) to see if the returns are likely real and not random noise. Your notes on Market Risk echo this: 'VaR and expected shortfall are standard measures'.


## In short

Backtesting LLM strategies is about honesty under pressure. To trust a strategy, you must test it on data it has never seen, account for every cost, avoid the lookahead trap (the model must never see data from the future), and measure not just profit but risk. The minimum discipline is out-of-sample and walk-forward testing. The tools are arithmetic, loops, and libraries like pandas, but the hardest part is discipline.


## Where this is used

Backtesting is not a niche activity — it is the standard gatekeeper for any AI system that touches money. Here is where it shows up across the field.

**Large language models (LLMs) in finance.** When a bank or hedge fund uses an LLM to read earnings calls, news, and filings and then generate a 'buy' or 'sell' signal, the entire strategy is backtested before it is deployed. The developer writes a pipeline: pull historical documents, run the LLM to produce a score for each, then feed that score into a trading rule. Walk-forward testing is used to make sure the LLM — which was trained on internet data that may include future prices — is not peeking. If the backtest shows profits, the strategy might be deployed; if not, it is sent back to the drawing board. The mechanism is exactly the arithmetic from this page: cumulative profit, hit rate, maximum drawdown. A developer would write code using `pandas` to manage the historical price data and a model API to get the LLM's scores.

**Small language models running on devices (TinyML, on-prem).** When a model must run on a phone or a laptop, it cannot be as large as GPT-4. The strategy must be backtested on the *small* model, not a large one, because the small model may make more errors. The backtest still uses the same walk-forward method, but with a lighter hand: a developer might code the model in ONNX or TensorFlow Lite, deploy it in a simulator, and run the historical data through it. The cost of errors is measured the same way, and the strategy is judged on the same metrics. This is common in retail trading apps that filter signals on a phone.

**Retrieval-augmented generation (RAG).** Many financial AI assistants use RAG to pull in relevant documents — say, past SEC filings or news articles — before generating an answer. When the assistant is used to *advise* on trades, the retrieval pipeline itself must be backtested. The mechanism: given a historical question, the retriever must pull only documents that existed at that date. A backtest would replay past questions and check whether the retrieved set is point-in-time correct and whether asking the LLM to reason over those documents improves downstream returns. The developer works with a vector database (e.g., FAISS) and an LLM API, and the backtest grid-searches over retrieval windows and document sources. This is part of the discipline of 'admissible evidence spans' in your library.

**Agentic systems that plan, call tools, and act in loops.** The most sophisticated financial agents use an LLM to break a task into steps, call tools (like a stock quote API or a balance sheet fact-checker), and act based on the results. But an agent can be exploited — it may call a 'current price' tool and get a price from 2021 while testing on 2021 data, which is lookahead. Backtesting these agents requires a *simulated environment*: you give the agent a fake clock and fake tools that only return data as of the simulated date. The developer writes a harness that steps the agent through each historical day, capturing its actions and their outcomes. The backtest measures not just final profit but whether the agent's tool calls were valid for the date. Without this, an agent can cheat by 'calling' for future data. This is a hard, open problem in AI safety, and backtesting is the main line of defence.


---

Read this on the web: https://basavapurushottam.com/learn-ai/backtesting-llm-strategies
