AI Foundation Models: How They Work and More

What are Foundation Models in Generative AI?

AI foundation model definition: The term foundation models was introduced by the Stanford University Institute for Human-Centered Artificial Intelligence (HAI). This organization conducts research on foundation models and artificial intelligence (AI) models more generally with an emphasis on ensuring that the technology maintains a focus on the public good.

HAI defined foundation models in generative AI as large, pre-trained machine learning models—usually neural networks—that serve as a foundation for a range of tasks.

A few key traits set foundational AI models apart:

• General-purpose. They are not built for just one task—they’re adaptable for many AI applications.
• Used across domains. Similarly, they are not specifically deployed in any one area and can be used to solve problems across knowledge domains.
• Pre-trained at scale. They are trained on huge amounts of data and demand massive computational resources.
  

Think of foundational models in AI as the “engines” or “brains” powering various generative AI tools. Just like a basic engine can be customized for high performance or a human brain can be trained into an expert with a specialty, a foundation model in generative AI can be shaped into a much more specific tool.

How Do Foundation Models Work in AI?

AI foundational models are large-scale models trained with self-supervised learning on broad, diverse datasets. To understand how foundation models are different from the traditional AI models, consider their key characteristics, how they work, and how they can be adapted for downstream tasks with fine-tuning, prompt engineering, and model alignment.

Key characteristics of a foundation model in AI include:

Massive scale. Foundation models in generative AI are trained on billions to trillions of parameters. They use enormous and often diverse datasets—web pages, books, code, images, and other sources. They demand high-performance computing infrastructure such as clusters of GPUs/TPUs.

General-purpose utility. Unlike traditional ML models, foundation AI models are not task-specific. They can perform many tasks with little or no extra training with the help of prompting and fine-tuning techniques.

Transferable knowledge. Mastery of general patterns and knowledge such as language structure and world facts allow AI foundation models to apply their knowledge across different domains and tasks.

Self-supervised learning. Because gen AI foundation models are trained without manually labeled data, they learn by predicting parts of their input—filling in missing words or predicting the next token in a sentence, for example. (Transfer learning and labeled data can later be used to fine-tune foundation models in AI for more specific tasks.)

Architecture. Most AI foundation models are based on transformer architecture, which understands relationships between pieces of data such as words in a sentence or pixels in an image using attention mechanisms.

Some AI foundation models have additional—and in some cases, unexpected—capabilities. For example, a foundation model in gen AI may have multimodal capabilities and the ability to handle multiple types of data such as text, images, video, and audio. And as model size and training scale increase, generative AI foundation models may exhibit unexpected skills that were not explicitly programmed, such as reasoning, basic arithmetic, code generation.

There are a few basics of the training process for foundation models in gen AI that remain constant:

It starts with inputting large datasets scraped from the internet. The trainer inputs text, code, images, or other data focused on a specific objective, such as learning to predict or reconstruct data.

For example, a large language model will learn to predict the best responses to a prompt like this: “The man sat down on the _____” while an image generating model will learn to respond to a prompt like this: “Generate an image of a white hen sitting in a nest.”

Next, fine-tuning, prompt engineering, and model alignment each play a different role in customizing, controlling, or improving AI foundation models.

Fine-tuning. Fine-tuning involves training a foundation model further on a smaller, targeted dataset such as legal documents or medical records for a new, narrower domain and adjusting its parameters to specialize it for new tasks. Full fine-tuning updates all of the model’s parameters, while parameter-efficient fine-tuning only updates small portions of the model to reduce resource usage.

Prompt engineering. This step involves tailoring inputs to get the desired outputs from an AI foundational model without changing its internal parameters. There are a few types of prompt engineering that can achieve these kinds of results:

• Zero-shot prompting directly asks for the desired result: “Translate this sentence into French. . .” produces a direct translation.
• Few-shot prompting provides examples in the prompt to get the model started: “Translate like the following: ‘Hello’ → ‘Bonjour’, ‘Goodbye’ → ‘Au revoir’,. . .” and so on gives the model enough to work with that the results are accurate.
• Chain-of-thought prompting asks the model to reason step by step, sort of like a word problem from math class: “I went to the market and bought 10 oranges. I gave 2 to my neighbor and 2 to my friend. Then I ate 1 orange. I then went and bought 5 more oranges. How many oranges did I have left? Let’s think step by step.” Instead of a simple number as the answer, the model will reply with its reasoning in the answer, like this: “First, you started with 10 oranges. You gave away 2 oranges to your neighbor and 2 to your friend, so you had 6 left. Then you ate 1 orange, so you had 5 left. Finally, you bought 5 more oranges, so you would have 10 oranges left.

Model alignment. This final step ensures that the behavior of foundation models of generative AI is safe, helpful, honest, and aligned with human values. Alignment may be constitutional, meaning that the model uses AI-generated feedback, values, and principles to guide its training without direct human supervision, or it may be based on reinforcement learning from human feedback (RLHF). In RLFH, humans rank model responses and the model is trained based on those rankings. This kind of fine-tuning reinforces learning to produce better, safer answers.

Types of Foundation Models in Generative AI

In foundation models and generative AI, each type of model is designed to generate or understand different kinds of data. AI foundation models may be text-based, image-based, audio-based, code-based, or multimodal.

Text-based AI foundation models generate or understand natural language. These transformer-based, decoder-only models are trained on books, websites, Wikipedia, forums, code, and other sources of text data.

Image-based AI foundation models generate or interpret images from text or other images. They may be transformer-based, diffusion models, or convolutional neural network (CNN) hybrids, and they are trained on image-caption pairs and image datasets.

Audio-based AI foundation models generate, understand, or transcribe speech or sound. They may be transformer-based or encoder-decoder models, and they are trained on audio transcripts, music datasets, and speech data.

Code-based AI foundation models generate and understand programming languages. Like LLMs, they are transformer-based, but they are trained on public code repositories such as GitHub.

Multimodal AI foundation models handle multiple types of data at once. They are built as multi-encoder or fusion transformers with modality-specific inputs and trained on pairs or triplets of text, image, audio, and video.

AI Foundation Models Examples and Use Cases

Examples of text-based AI foundation models include GPT (OpenAI), PaLM (Google), Claude (Anthropic), LLaMA (Meta), and Mistral. These large language models (LLMs) are best-suited for use in text generation apps that compose or summarize; chatbots and assistants; and translation, question answering, and reasoning apps.

Other specific use cases for these text-based AI foundation models include:

• Business analytics (GPT-3/GPT-4)
• Ethical AI assistants (Claude Anthropic)
• Google Workspace tool integrations (PaLM/Gemini)
• Legal document review with guardrails (Claude Anthropic)
• Multimodal search and captioning (Gemini)
• On-device AI, private/local chatbots and assistants (LLaMA 2/3)
• Research and customization by enterprises (LLaMA 2/3)
• Reasoning-heavy tasks such as math and logic (PaLM/Gemini)
• Safe and controlled enterprise chatbots (Claude Anthropic)

Some examples of image-based AI foundation models include DALL·E (OpenAI), Midjourney, Stable Diffusion open-source image generation, and Imagen (Google). They are best-suited for applications that translate visual concepts into high-quality, creative images, such as text-to-image generation, image editing or enhancement, and artistic creation and design tools.

Other specific use cases for these image-based AI foundation models include:

• AI-assisted design tools (Stable Diffusion)
• Book covers, album art, branding (Midjourney)
• Creative marketing visuals and ads (DALL·E 3)
• Custom image generation (Stable Diffusion)
• Fashion and interior design mockups (Midjourney)
• High-quality concept art and visuals (Midjourney)
• Illustration and concept art (DALL·E 3)
• Storyboarding and design prototypes (DALL·E 3)
• Visual content in video games and XR (Stable Diffusion)

Examples of audio-based AI foundation models include Whisper (OpenAI), AudioLM (Google), MusicLM, and Bark. They are ideal for understanding and generating rich, nuanced audio content and are well-suited for transcription (speech-to-text), voice synthesis (text-to-speech), music generation, and sound design applications.

Other specific use cases for these audio-based AI foundation models include:

• Accessibility tools such as live captions (Whisper)
• Generative music for videos, games, or ambient settings (MusicLM)
• Multilingual transcription for media companies (Whisper)
• Soundtrack prototyping (MusicLM)
• Tools for musicians and producers (MusicLM)
• Transcription of podcasts, interviews, and meetings (Whisper)

Examples of code-based AI foundation models include Codex (OpenAI), the open-source code LLMs CodeLLaMA and StarCoder, and coding variants of Gemini. These models are adept with understanding syntax, logic, and structure of code, making them ideal for code generation and completion, automated bug fixing, code documentation, and translation between coding languages.

Other specific use cases for these code-based AI foundation models include:

• AI pair programming (Codex)
• Building small apps or scripts from text prompts (Codex)
• Code documentation, explanation, refactoring (Codex)
• IDE integration for auto-completion (StarCoder/Code LLaMA)
• Open-source coding copilots (StarCoder/Code LLaMA)
• Secure enterprise software engineering (StarCoder/Code LLaMA)

Examples of multimodal AI foundation models include GPT-4 with vision, Gemini, CLIP, Flamingo, and Kosmos-1. These models can understand context across multiple formats, enabling more intelligent and intuitive AI systems, suiting them well to describe or caption images or videos, engage in visual question answering (VQA), generate images with detailed prompts, and serve as interactive assistants with vision/audio input.

Other specific use cases for these multimodal AI foundation models include:

• Accessibility tools such as describing images to blind users (GPT-4 with Vision)
• Complex reasoning with text, images, audio (Gemini)
• Content moderation and visual search (CLIP)
• Educational tutors that interpret visuals (Gemini)
• Image-based question answering (GPT-4 with Vision)
• Image captioning (CLIP)
• Image classification by natural language (CLIP)
• Reading charts, graphs, screenshots (GPT-4 with Vision)
• Real-time multimodal search (Gemini)

WEKA and AI Foundational Models

Training foundational models at scale requires an infrastructure that can keep pace with massive data throughput, unpredictable I/O patterns, and the relentless demand for performance from GPUs. NeuralMesh™ by WEKA is purpose-built to meet these challenges, delivering a high-performance, software-defined storage solution that eliminates I/O bottlenecks and saturates GPU pipelines with data. Its distributed, parallel architecture provides ultra-low latency and linear scalability, enabling organizations to train larger models faster, with greater efficiency and predictability—whether running in the cloud, on-premises, or in hybrid environments.

What sets NeuralMesh apart is its ability to support massive, multi-petabyte datasets with fine-grained parallelism and consistent performance, even under extreme load. By decoupling data services from underlying infrastructure and optimizing metadata operations at scale, NeuralMesh ensures fast checkpointing, seamless data ingestion, and real-time streaming during training cycles. This results in higher GPU utilization, reduced training runtimes, and the ability to experiment with bigger models and larger batch sizes—all while lowering the total cost of AI infrastructure.