The Evolution from LLM-Centric Systems to RAG Architectures and the Optimal Model Stack for Financial Analysis & Research

1. The Core Problem: Why LLM-Only Approaches Break at Scale

Early AI platforms (including ESG research systems) were built using general-purpose LLM APIs such as GPT and Perplexity. These systems:

• Generated research queries
• Retrieved information via AI-powered search
• Produced summarised responses
• Extracted structured insights
• Generated final reports

This worked well for exploratory research and narrative outputs. However, as use cases scaled to institutional-grade analysis across hundreds of companies and filings, key limitations emerged:

Key Constraints of LLM-Centric Architectures

Retrieval and reasoning were coupled

LLMs performed search, selection, reasoning, and generation in a single step.
This created:

• No visibility into source selection
• No auditability
• Weak traceability of conclusions

Search became the bottleneck

LLM-powered search returns summaries, not full evidence:

• Critical disclosures often omitted
• Quantitative data lost
• Important context never retrieved

Structured extraction was inconsistent

Extracting:

• metrics
• policies
• targets
• governance structures

from conversational outputs led to:

• variability across runs
• poor reproducibility
• weak dataset quality

Document intelligence was fundamentally missing

Critical information lives inside:

• Annual reports
• Sustainability reports
• Regulatory filings

These contain:

• multi-column layouts
• financial tables
• footnotes and overrides

LLM search APIs are not designed to parse these structures reliably

2. The Architectural Shift: From LLMs → RAG

The breakthrough was not a better model—it was a better architecture.

New Workflow (RAG-Based)

Instead of:

“Ask model → get answer”

The system became:

Retrieve → Extract → Validate → Generate

What RAG Changes Fundamentally

Retrieval happens first

• Full evidence is captured before reasoning
• Eliminates hallucination from missing context

Evidence is complete, not summarised

• Direct access to filings and documents
• Richer context for downstream models

Responsibilities are separated. Each stage becomes independently optimisable:

• Retrieval
• Extraction
• Validation
• Generation

Outputs become auditable

• Every extracted value can be traced to source
• Enables institutional-grade confidence

 

3. The Hidden Truth: Model Choice Is Not the Main Driver

A critical insight from both documents:

Most failures come from parsing and retrieval—not model intelligence

Even top-tier models fail if:

• wrong section is retrieved
• tables are misparsed
• chunking is poor

So performance depends on four layers:

1. PDF parsing / layout understanding
2. Retrieval precision (RAG quality)
3. Model reasoning
4. Structured output discipline

 

4. Model Roles in a Modern RAG System

Rather than one “best” model, different models dominate different layers.

General LLMs (GPT-class)

Role: Reasoning + structured output

Strengths:

• Highest financial reasoning accuracy (~88–90%)
• Strong table/JSON generation
• Consistent outputs

Weakness:

• Dependent on upstream retrieval quality

Best use:

• Normalisation
• multi-year alignment
• final table generation

Claude (Anthropic)

Role: Long-context reasoning + document interpretation

Strengths:

• Handles large documents
• Strong multi-step extraction
• Excellent contextual understanding

Weakness:

• Slightly weaker numerical precision

Best use:

• filings ingestion
• complex disclosures
• cross-document synthesis

Gemini (Google)

Role: Document parsing and extraction

Strengths:

• Best-in-class PDF + layout understanding
• Strong OCR and table reconstruction
• Top performance in document extraction benchmarks

Weakness:

• Less consistent reasoning
• weaker structured outputs without control

Best use:

• extracting tables from filings
• parsing complex layouts

Perplexity

Role: Retrieval assistant

Strengths:

• fast retrieval
• citation support

Weakness:

• not designed for structured extraction
• inconsistent outputs

Best use:

• ad hoc research queries

Not suitable for:

• production data pipelines 

5. The Real Architecture: How Best-in-Class Systems Work

Optimal Pipeline

1. Document ingestion
   • PDF → layout-aware parser (Gemini / DocAI)
2. Data separation
   • Extract tables and text independently
3. Storage
   • structured JSON + indexed chunks
4. Retrieval (RAG)
   • vector database / semantic search
5. Reasoning layer
   • GPT or Claude:
     • align years
     • normalise metrics
     • resolve inconsistencies
6. Validation layer
   • cross-check against source
7. Output generation
   • structured tables + narrative

6. Why Hybrid Systems Win

A key synthesis across both documents:

“No single model wins end-to-end”

• Gemini → best extraction (front-end)
• GPT / Claude → best reasoning + structuring (back-end)

Another critical insight:

• Vision/document models → best at structure
• LLMs → best at meaning

So:

Best performance = hybrid system combining both

7. Why Financial Data & Documents Are Hard

Both perspectives converge on this:

Financial filings are adversarial to AI systems because they include:

• inconsistent naming conventions
• multi-column formatting
• nested tables
• footnotes that override numbers
• mixed narrative + quantitative content

This is why:

• extraction is difficult
• consistency is hard
• deterministic outputs require careful design

8. Final Synthesis

What changed?

The evolution was not:

“better models”

It was:

from monolithic AI → modular AI systems

What matters most now

1. Retrieval quality > model choice
2. Parsing accuracy > reasoning accuracy
3. Architecture > prompt engineering

Best-in-class setup today

• Parsing: Gemini / DocAI
• Retrieval: RAG (vector DB)
• Reasoning: GPT or Claude
• Validation: independent layer
• Output: structured + traceable

Bottom line

• LLMs alone are sufficient for exploration
• RAG systems are required for production
• Hybrid architectures are required for accuracy at scale