Local LLM Setup with Ollama

Hello! Welcome to the second module of our course.

In the previous module, we established the theoretical foundations of LLM security. We analyzed how tokenization and in-context learning create vulnerabilities, and we concluded by diagramming how Retrieval-Augmented Generation (RAG) introduces a potent "Back Door" for indirect prompt injection attacks.

Now, we shift from theory to practice. This lesson kicks off our work on the vulnerable chatbot you'll use for your demo. Our learning outcome is to set up a local development environment with Ollama and a small language model (e.g., Llama 3 8B, Phi-3). This local setup gives you complete control over the model and environment, which is perfect for reproducible security experiments.

1. The "Why": Running Language Models Locally

Before we install anything, let's address the key considerations for running LLMs on your own machine. Unlike calling a cloud API, a local setup makes you responsible for resource management. The feasibility and performance of running a model depend heavily on your hardware.

The most important factors are:

• GPU and VRAM: The model's parameters are loaded into the GPU's Video RAM. This is the primary bottleneck. More VRAM allows you to run larger, more powerful models.
• Quantization: This is a technique to reduce the memory footprint of a model by lowering the precision of its weights (e.g., from 16-bit to 4-bit numbers). This allows large models to fit into consumer-grade GPUs with a manageable trade-off in accuracy.
• System RAM and CPU: System RAM is needed to support the GPU and handle data processing, while a decent CPU helps with loading the model and preparing data.

To get a clear picture of these hardware requirements, let's watch a short video.

With that context, let's set up the software that makes this all manageable: Ollama.

2. Installing and Configuring Ollama

Ollama is a fantastic tool that streamlines the process of downloading, managing, and running local LLMs. It packages models into a consistent format and exposes them through a local API server, which is exactly what we need to build our FastAPI application.

The following video gives a great tour of the entire process, from installation to programmatic use. We will walk through it step-by-step.

A More Robust Setup for Linux

Given your deep experience with Linux administration and SRE, you'll appreciate that running Ollama as a managed background service is a more robust approach than just launching it from a terminal. The official documentation provides instructions for setting it up as a systemd service.

Alternative: Running Ollama with Docker

Your proficiency with Docker offers another excellent, self-contained way to run Ollama. Encapsulating the Ollama server in a Docker container isolates its dependencies and makes it easy to manage as part of a larger application stack using docker-compose.

While we won't implement the Caddy proxy shown here, running the core Ollama service via Docker is a highly viable alternative to a direct systemd installation. You can find the official Ollama Docker image on Docker Hub.

3. Downloading and Testing a Model

With Ollama installed and its service running, the next step is to download a model. We'll use Llama 3 8B, a powerful and versatile model that is small enough for local experimentation.

You can also see all the models you've downloaded by running ollama list.

4. Programmatic Interaction with Python

The command line is useful for quick tests, but our goal is to build an application that uses the model. Ollama exposes an HTTP API that allows any application to send it prompts and receive responses. We will interact with this API using the official ollama Python library.

First, let's set up a Python virtual environment for our project and install the necessary package.

# Create a new directory for your project
mkdir llm-security-demo
cd llm-security-demo




# Create and activate a Python virtual environment
python -m venv venv
source venv/bin/activate  # On Windows, use `venv\Scripts\activate`




# Install the Ollama library
pip install ollama

Now, let's see how to use this library.

Now, create a Python file (e.g., test_model.py) and add the following code to verify your setup:

import ollama

try:
    response = ollama.chat(
        model='llama3:8b',
        messages=[
            {
                'role': 'user',
                'content': 'In one sentence, why is local-first AI development useful for security research?',
            },
        ]
    )
    print(response['message']['content'])
except Exception as e:
    print(f"An error occurred: {e}")
    print("Please ensure the Ollama application or service is running.")

Run this script from your terminal (python test_model.py). If everything is configured correctly, you should see a response generated by your local Llama 3 model.

Conclusion

Congratulations! You now have a fully functional local LLM development environment. This is a significant first step and the foundation upon which we'll build and test our entire prompt injection demo.

Key Takeaways:

• Local LLMs are practical: Thanks to techniques like quantization, we can run powerful models like Llama 3 8B on consumer hardware.
• Ollama is your control plane: It handles the download, management, and serving of local models, exposing them via a simple API.
• Robustness matters: For a stable development environment, running Ollama as a systemd service (on Linux) or within a Docker container is preferable to running it manually.
• Python is the interface: The ollama Python library provides a clean, high-level way to integrate local models into your applications.

In our next lesson, we will build on this foundation. The topic is "Explain the core components of LangChain: chains, prompt templates, and output parsers." Now that we have a running model, we need a framework to structure our interactions with it. LangChain will provide the tools to build the logic of our vulnerable RAG chatbot.