Research · 2026-10-24 · All research

A 4B model beat a 70B one, and a 284B commercial API, at explaining financial data. Here is how we measured it.

We fine-tuned eleven open-weight models, from 4B to 70B parameters, to explain seasonal market statistics. The smallest one won, and a commercial API lost on every behavioural axis we measured.

Photo: Alesia Kozik / Pexels

SeasOptima publishes historical seasonality statistics for more than 47,000 instruments: equities, ETFs, indices and currency pairs, with up to 25 years of history each. Users want those numbers explained in plain language. How often has this stock risen in October? Is March consistently weak, or was it one bad year?

The obvious way to build that assistant is to send every question to a large model behind a commercial API. We wanted to know whether that is actually necessary, or whether a small open-weight model, trained on our own data and running on our own hardware, could do the job better.

So we measured it. The short answer: the best of eleven models we trained was the smallest.

The data

From SeasOptima's own statistics we generated 10,216 training examples and set aside 1,129 for evaluation. Every example carries its own grounding context: the ticker, the years of history, the average return and win rate for each month, a consistency score. The question can only be answered from those numbers.

The set covers six languages (English, French, German, Italian, Brazilian Portuguese and Spanish) and nine question types. 1,153 of the examples are deliberately adversarial: questions that try to get the assistant to recommend a trade or predict a price. A seasonality tool that starts giving investment advice is not a product, it is a liability.

How we scored it, without a human or a second model

We did not ask people to rate answers, and we did not ask one model to judge another. Every answer was scored automatically on things that can be checked:

• Grounding. Every financial figure in the answer must exist in the context. A number that is not in the data was invented.
• Compliance. No advice or prediction language, and an explicit refusal when the question asks for one.
• Disclaimer. The past-performance caveat is present, in the right language.

Because the checks are mechanical, anyone with the same evaluation set gets the same numbers.

Eleven models, same data, same test

We adapted all eleven models identically, with the same training set and the same low-rank adapter settings, and scored each on the same 300 held-out questions.

Disclaimer

Seven of the eleven shown; the full table is in our report to EuroHPC.

The 4B model finished first or tied for first on every axis, and alone at the top on the disclaimer. The 70B model finished last on the two axes that matter most for a financial product: it left out the disclaimer in 38% of answers and failed to refuse 17% of the requests for advice.

This task is about disciplined, bounded answering, not breadth of knowledge. The extra capacity bought nothing, and it made the model harder to keep inside the lines.

The other result we did not expect: grounding was essentially solved by fine-tuning. Nine of the eleven models scored a perfect 1.000 on grounding across 300 answers, and none went below 0.994. The failure we feared most, a made-up return shown to a retail investor, turned out to be the easiest one to remove, as long as the training data always carries its own numbers.

We then tried to break the chosen model with 66 questions it had never seen: direct requests for buy and sell calls, prompt injection, typos, vague questions, unexpected languages, questions about data it did not have. Grounding and compliance both stayed at 1.000.

Against a commercial API

We ran the same 300 questions through DeepSeek-V4-Flash, a commercial API model with 284 billion parameters, using the same prompt and the same scoring.

On accuracy the two were equal. On behaviour the API lost everywhere. The most striking single result: given English questions, the API answered 26 of them entirely in Chinese. Fluent, complete, disclaimer included, and unreadable to the person who asked.

We did not rewrite the prompt for the API, because that would have changed the thing we were measuring. It does show where the behaviour lives. In our model it is a property of the weights, learned once. In a prompted model it is an instruction, to be re-checked with every provider and every model update.

The API was faster, and we report that as measured. Our figure comes from single, unbatched requests on one accelerator, not from a serving setup. A production deployment would do better; we have not measured that yet, so we do not claim it.

What it means for a business

The capability this product needs fits in 4 billion parameters under a permissive licence. That is 2.2 GB at 4-bit precision, small enough for one consumer graphics card. At the speed we measured, one card serves roughly 9,500 answers a day before any optimisation.

It also means the questions never leave the building. Which securities a person researches, and when, is exactly the kind of behavioural data you do not want sitting on someone else's servers.

Where this ran

The eleven-model comparison needs far more compute than a small company owns: it is the comparison itself, not any single training run, that required a supercomputer. We ran it on LUMI, the EuroHPC supercomputer hosted by CSC in Kajaani, Finland, under a competitively evaluated allocation from the EuroHPC Joint Undertaking (AI Factory Playground, EHPC-AIF-2026PG01-685). The whole study used 164 GPU-hours and finished in six days.

LUMI runs on hydroelectric power, and its waste heat warms the town of Kajaani.

We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LUMI, hosted by CSC (Finland) and the LUMI consortium, through the EuroHPC AI Factory Playground Access call.