Generative Models: Modalities and Tools

A Map of the Generative AI World

At its core, generative AI is a collection of different technologies specialized for creating different types of content. The most common way to categorize these models is by the modality of their output.

Let's start with a high-level overview. The following mind map illustrates the main categories of generative AI models that have become prominent.

As you can see, the generative AI ecosystem is not monolithic. It's a vibrant field with specialized tools for a wide range of creative and technical tasks. We will now explore these key territories one by one.

Exploring the Territories: A Tour of GenAI Modalities

While there are many niche modalities, we will focus on the most mature and widely used categories: text, image, and code. We will also touch upon the emerging fields of audio and video generation.

For a dynamic and engaging walkthrough of these different types, the following video provides an excellent summary.

Now, let's look at each of these modalities in more detail, identifying the representative tools you'll need to know.

Let's formalize that categorization:

• Text-Based Models: These are the most famous type of generative AI. Often called Large Language Models (LLMs), they are trained on vast amounts of text data to generate human-like writing, from emails and summaries to poetry and complex arguments.

  • Representative Tools: GPT-4o (OpenAI), Claude 3 (Anthropic), Gemini (Google), LLaMA 3 (Meta), Mistral.

• Image-Based Models: These models typically generate novel images from text descriptions (a process known as text-to-image). They have revolutionized digital art, design, and marketing. Given your interest in pencil sketching and artwork, you may find the creative potential here particularly compelling.

  • Representative Tools: Midjourney, DALL-E 3 (OpenAI), Stable Diffusion (Stability AI), Adobe Firefly.

• Code-Based Models: As a programmer and technical trainer, you'll recognize the immense utility of this category. These models, which are often specialized LLMs, assist developers by generating code snippets, completing functions, writing documentation, and even finding bugs.

  • Representative Tools: GitHub Copilot, Amazon CodeWhisperer, Replit AI.

• Audio and Video Models: These are rapidly emerging fields. Audio models can generate music in various styles (e.g., Suno, MusicGen) or create realistic voiceovers. Video models can create short video clips from text prompts (e.g., Sora, RunwayML), though this technology is still less accessible to the public.

Under the Hood: Modalities vs. Model Architectures

So far, we've categorized models by what they create (their modality). However, for someone with your technical background, it's also useful to understand how they create it, which relates to their underlying model architecture. Think of modality as the artistic medium (oil paint, watercolor) and the architecture as the technique (impressionism, pointillism).

The learning outcome for this lesson focuses on modality, but knowing the key architectural differences will deepen your understanding and help you explain these concepts to advanced students.

To summarize the key relationships from that reading:

• Transformers are the dominant architecture for text and code generation due to their powerful "self-attention" mechanism, which excels at understanding context in sequential data.
• Diffusion Models are the current state-of-the-art for high-quality image generation. They work by starting with random noise and gradually refining it into a coherent image based on the prompt.
• Generative Adversarial Networks (GANs) were a breakthrough architecture for image generation before diffusion models became popular. They involve a "generator" and a "discriminator" network competing against each other to produce realistic images. They are still used, especially where generation speed is critical.

The Global Landscape and the Rise of Multimodality

The field of generative AI is not just a collection of academic concepts; it's a bustling global ecosystem of companies and research labs. This map gives you a sense of the scale and geographic distribution of the major players.

A key trend visible in this landscape is the move towards multimodal AI. Instead of using separate tools for text, images, and code, the new frontier is single models that can seamlessly process and generate content across different modalities.

Models like Google's Gemini and OpenAI's GPT-4o are at the forefront of this trend. They can analyze an image, have a spoken conversation about it, and write code to describe it—all within a single interaction. For your course design, this means future learning activities can be more integrated, blending different types of creative and analytical tasks.

Your Project: A "Collection Quest"

Now, let's translate this knowledge into a practical step for your course design. One classic game mechanic is the "collection quest," where a player has to find and gather a set of distinct items. We can adapt this to create a simple introductory activity.

Starter Quest: Design a simple "collection quest" for your beginner GenAI course.

• Objective: Students must find and test one free, publicly available GenAI tool from three different modalities (e.g., one for text, one for images, and one for code).
• Task: For each tool, they must perform a simple task and document the result. For example:
  • Text: Ask the tool to explain a complex topic in simple terms.
  • Image: Generate an image of a "robot learning in a library."
  • Code: Ask for a Python snippet to read a CSV file.
• Deliverable: A short document or presentation comparing their experience with each tool. What felt different about interacting with each modality? Which was most intuitive? Which was most surprising?

Just sketch out the instructions for this quest. This will serve as a foundational activity in your course blueprint, helping your students build their own mental map of the GenAI landscape.

Conclusion

In this lesson, we moved from the "what is it?" of generative AI to the "what are the different kinds?" We've mapped out the field and given you the vocabulary to navigate it.

Key Takeaways:

• Generative AI models are primarily categorized by modality: the type of content they create (text, image, code, audio, video).
• Each modality has well-known representative tools: ChatGPT for text, Midjourney for images, and GitHub Copilot for code are iconic examples.
• Underlying these modalities are different model architectures. Transformers dominate text and code, while Diffusion models are the current standard for high-quality images.
• The industry is rapidly moving towards multimodal models that can understand and generate content across several modalities in a single, unified experience.

In our next lesson, we will equip ourselves with the essential vocabulary of GenAI. We'll define key terms like tokens, parameters, inference, and context window. This will give you the precise language needed to explain how these models function at a fundamental level—a critical skill for any technical educator.