Comparing LLM Providers for Project Finance

The Core Trade-off Framework: Accuracy, Latency, and Cost

The fundamental challenge in deploying LLMs at scale is balancing the quality of the model's output with the speed and cost of generating it. A great starting point is to adopt the philosophy used by the model providers themselves: first, optimize for the accuracy you need, and then optimize for latency and cost.

To get a deep dive into this framework, we'll watch a presentation from OpenAI's development team. It provides an excellent structure for thinking about these trade-offs.

Part 1: Defining and Measuring "Accuracy" for Project Finance

For a general-purpose chatbot, "accuracy" might mean providing helpful, coherent answers. For your project finance agents, the definition is far more stringent. It involves capabilities like:

• Numerical Modeling: Performing calculations correctly (e.g., DSCR, IRR).
• Temporal Reasoning: Understanding sequences of events (e.g., construction milestones vs. operational periods).
• Scenario Planning: Evaluating outcomes under different assumption sets.
• Terminology Understanding: Correctly interpreting finance-specific jargon.

Standard benchmarks like MMLU provide a general sense of a model's knowledge, but they don't capture performance on these specialized financial tasks. Fortunately, academic research provides more domain-specific insights.

The XFINBENCH results show that even the most advanced models struggle with certain financial reasoning tasks, and no single model is the best at everything. This underscores the most important principle of model evaluation: benchmarks are a guide, but you must test on your own data.

The OpenAI video provides a practical framework for this.

For your project finance agents, an "eval" could be a set of questions about a deal memo where you already know the answers (a "golden dataset"). You can then measure how often a model provides the correct answer, helping you choose between providers and prompting strategies.

Part 2: Analyzing Latency and Cost

Once you have a model that meets your accuracy threshold, the next step is to see if you can achieve the same result faster and cheaper.

Latency (Speed)

Latency is critical for user-facing applications. An agent that takes 10 seconds to answer a question feels broken. Total request latency can be broken down into:

1. Time to First Token (TTFT): How long it takes to process the prompt and start generating a response.
2. Time Between Tokens (TBT): The speed at which subsequent tokens are generated. Total output time is TBT multiplied by the number of output tokens.

Cost

LLM usage is typically billed per token, with different rates for input (prompt) and output (generation). Costs can add up quickly, especially with large context windows or high-volume tasks.

To see how these concepts apply in the real world, let's look at a practical comparison guide.

Part 3: The Strategic Choice: Proprietary vs. Open-Source

So far, we've focused on proprietary models available via an API (e.g., from OpenAI, Anthropic, Google). An alternative is to host an open-source model (e.g., from Meta, Mistral) on your own infrastructure.

Proprietary Models (API-based):

• Pros: Easy to use, no infrastructure to manage, access to the absolute state-of-the-art.
• Cons: Pay-per-use can be expensive, data privacy concerns (your data is sent to a third party), less control over the model.

Open-Source Models (Self-hosted):

• Pros: Full data privacy and control, no per-token API costs (only infrastructure costs), ability to deeply fine-tune.
• Cons: Requires significant engineering effort to set up and maintain, infrastructure can be expensive, typically lag slightly behind the frontier proprietary models in performance.

For someone with your CS background, the technical aspects of self-hosting are important. Latency isn't just a function of the model; it's also determined by the serving infrastructure.

A Practical Framework for Model Selection

Let's synthesize everything into a decision-making framework for building your project finance agents. The best practice is not to choose one model for everything, but to implement a multi-model routing strategy.

1. Define the Task & Requirements: What is the agent doing?
   • Task: Simple classification (e.g., "Is this a PPA?").
   • Requirements: High volume, low cost, latency is not critical.
2. Select a Candidate Model:
   • A cheap, fast model is ideal. DeepSeek or a small, self-hosted open-source model would be a good fit. Using Claude or o1 would be massive overkill.
3. Test and Evaluate:
   • Run a test set of 1,000 documents through the candidate model and measure its accuracy. If it meets your 99% accuracy target, you have your model.
4. Build a Router:
   • Create a "router" or "supervisor" agent that directs tasks to the appropriate specialist model.

The LLM Comparison Guide offers excellent heuristics for this.

Conclusion

You now have a comprehensive framework for one of the most vital decisions in building AI systems: choosing the right LLM. This isn't a one-time choice but a continuous process of evaluation as models evolve and your application's needs change.

Key Takeaways:

• The Iron Triangle: Model selection is a constant trade-off between accuracy, latency, and cost.
• Accuracy is Domain-Specific: General benchmarks are a starting point, but you must evaluate models on tasks relevant to your domain, like the financial reasoning capabilities tested in XFINBENCH.
• Optimize Systematically: Start by hitting your accuracy target, then use techniques like prompt caching, batch processing, and choosing smaller models to optimize for latency and cost.
• Proprietary vs. Open-Source: This is a key strategic decision balancing ease-of-use and frontier performance against control, privacy, and long-term cost.
• Use a Multi-Model Strategy: The most effective and cost-efficient systems use a router to delegate tasks to different models—cheaper models for simple tasks and more powerful models for complex, critical ones.

In this lesson, we discussed the concept of a "router agent" that directs tasks to the right model. In our next lesson, we will put this into practice. We will tackle the learning outcome: Build a simple router agent that selects the appropriate tool based on user queries. You will learn how to build an agent that can dynamically choose between different functions or even other agents, a foundational step towards orchestrating the complex, multi-agent systems you aim to create.