Generative AI for Professionals

Executive Summary

Deep learning creates narrow expert models for specific tasks. Large language models (LLMs) predict the next token using massive internet text, creating base
models that are then fine-tuned into assistants (ChatGPT, Claude). Key limitation: assistant models imitate training data without verification - unreliable for
math/logic. Reasoning models (o1, DeepSeek) use reinforcement learning to generate extended thinking chains, improving accuracy on logical problems at higher
cost. Extend capabilities with tools (search, code execution). Best practices: verify outputs, use reasoning models for logic, assistant models for general
tasks, leverage prompting techniques.

Deep Learning

The goal of deep learning is to create a mathematical function that can mimic expert behavior on specific tasks. This is achieved by using large amounts of labeled data to update the parameters of a neural network.
The network learns patterns from examples, adjusting millions or billions of parameters through training to improve accuracy on the target task.

Data Types: Images / Audio / Video / Raw text

Tasks: Object detection / Image classification / Sentiment analysis / Language translation / Next word generation (language modeling)

Limitations: Training requires significant computational time due to the large number of parameters. The learned relationships are often opaque (black box problem), and models tend to be narrow experts, excelling at specific tasks but lacking generalization across domains.

Large Language Models

The goal of large language models like GPT is to create a general expert capable across several tasks, moving beyond narrow specialization. These are still neural networks, but they're based on the Transformer architecture published by Google DeepMind in 2016, rescoped to predict the next word in a sequence.

Base Model Training Process

Data Collection: Massive amounts of text are gathered from the internet. Datasets like Fineweb provide examples of this scale.

Tokenization: Text is converted into sequences of symbols called tokens. Starting with a stream of bytes, Byte Pair Encoding (BPE) merges the most common token pairs in the data. Continuing this process allows individual tokens to hold more information, with some tokens representing entire words. This increases the total vocabulary (GPT-2 used approximately 50,000 tokens).

Neural Network Training: A fixed set of tokens is passed to the model based on its context length. The model outputs a probability distribution over all possible tokens. Since the next token is available in the data itself, the dataset is self-labeled. The training goal is to have the model predict high probability for the correct next token and low probability for others. After training, the model learns to produce sequences that simulate what it saw in training data.

Base Model Characteristics

Base models learn general representations and acquire substantial knowledge. They can be "prompted" into becoming assistants by using input sequences to guide future token behavior. However, they are not true assistants, are not moderated, and can hallucinate information.

Special Tokens: The <|endoftext|> token is introduced in training data to implicitly teach the model when a sequence should end. Through training, the model learns to assign high probability to this token when a sequence seems complete.

Sampling Parameters: The output probability distribution can be modified using sampling parameters such as temperature top-p top-k.

See this interactive demonstration to understand how various parameters work.

Assistant Model Training

After a base model is trained on internet data, high-quality question-and-answer datasets are collected through human labelers and experts. These datasets, such as Open Assistant, are enhanced with special tokens to organize conversations into a protocol separating system, user, and assistant tags. The model must learn to mimic assistant responses.

This data is tokenized as before, and training resumes using the base model as a starting point. OpenAI observed that the new model retains the general knowledge of the base model but adopts the persona of a helpful assistant, making it suitable for applications like chatbots.

Scaling Observations

Empirically, assistant model performance increases with model size, leading companies to train increasingly larger models.

This means that you can get much better out of box model performance by choosing a bigger (more parameters) assistant model.

Prompting

Assistant model behavior can be modified through prompting. Specifically, assigning roles, giving clear instructions, providing context, and offering several examples significantly improves the model's ability to behave as expected.

Critical Risks

Assistant models are token simulators, meaning there is no reflection or sanity checks. Output from these models cannot be trusted as source of truth. They may appear like experts, but their outputs are based on imitating training data, not critical thought.

Common Failures: Models fail with spelling and math. Some issues stem from tokenization choices, while others are inherent to the nature of LLMs as probabilistic token generators.

Reasoning models

Increasing size of assistant model does not inherently make them better at logical problems.

Several methods were attempted to train assistant models better, but without success. The breakthrough came from using reinforcement learning. OpenAI succeeded with their o1 model, though details were initially unclear. Later in 2025, DeepSeek published a paper outlining exactly how to train a reasoning model.

DeepSeek's Approach

The central idea was to use problems from a verifiable domain (math) and make the model generate several outputs, then imitate the outputs that were successful.

Data: Math problems such as those from GSM8K Dataset.

Starting Point: A pretrained assistant model.

Output Format: A thinking model instructed to put its reasoning within <think> tags and the final answer within <answer> tags.

Key Discovery: DeepSeek observed that the model by itself learns to output a large number of tokens as part of its thinking during training. This is something researchers had previously tried to encourage through prompts like "let's think step by step," but now the model develops this behavior through reinforcement learning.

Significance

Reasoning models represent "true" machine intelligence because machines can learn to come up with novel solutions given enough training, rather than simply pattern matching from training data.

Advantages: Reasoning models are more likely to produce correct answers to logical tasks and can develop multi-step reasoning strategies.

Disadvantages: These models produce a large number of tokens during the reasoning process, which leads to longer wait times and increased cost of inference.

Tools and Extensions

Beyond prompting and reasoning models, we can optimize model performance by giving it access to tools.

Search Tools: Fetch latest information from the internet and place it in the model's context, addressing knowledge cutoff limitations.

Code Interpreter Tools: Run code and provide the model with execution output, enabling accurate computational tasks.

Note: Models themselves cannot search the internet or run code. Programmers must write software to integrate search APIs or code runtimes along with the model. Models can be taught (through training data or in-context learning) to use additional special tokens to trigger search, code execution, or other tools.

Key terms

Tokens: Basic units of text (words, subwords, or characters) that models process. Tokenization converts raw text into these discrete units.

Context Length: The maximum number of tokens a model can process at once. Modern models range from 4K to 200K+ tokens.

Model Parameters: The learned weights in the neural network, typically billions in modern LLMs. More parameters generally mean more capacity but slower inference.

Model Type: Classification into Base Model (raw next-token predictor), Assistant Model (fine-tuned for helpfulness), or Reasoning Model (trained with reinforcement learning for logical tasks).

Best Practices and Recommendations

Use Prompting Techniques: Optimize model output by assigning clear roles, providing explicit instructions, offering relevant context, and including examples (few-shot prompting).

Leverage Tools: Use search tools for up-to-date information that exceeds the model's knowledge cutoff. Use code interpreters to offload calculations and tasks that models inherently struggle with, such as arithmetic.

Choose the Right Model: Use thinking models for logical problems requiring multi-step reasoning. For general assistance and conversational tasks, standard assistant models are more cost-effective.

Warning: Don't take assistant model outputs as source of truth. Outputs can be convincing but incorrect, with no built-in verification or fact-checking.

Future Directions

Multimodal Capabilities: Current models can process images, audio, and video by converting them into tokens alongside text. This enables vision tasks, speech processing, and multimedia understanding.

Agentic Systems: The future involves models with tools talking to other models, enabling multi-agent collaboration and complex workflow automation.

Local LLMs: Running models on personal devices offers privacy and cost benefits, with open-source alternatives becoming increasingly capable.

Enhanced Multimodal: Continued improvements in image generation, real-time video understanding, and audio synthesis are expected.

Additional Links

Papers

• GPT-2 Paper (2019) - Language Models are Unsupervised Multitask Learners - foundational work on base models
• GPT-3 Paper (2020) - Language Models are Few-Shot Learners - scaling laws and emergent capabilities
• InstructGPT Paper (2022) - Training language models to follow instructions with human feedback (RLHF)
• DeepSeek-R1 Paper (2025) - Technical paper detailing how to train reasoning models using reinforcement learning

Datasets

• Open Assistant - High-quality conversational dataset for training assistant models
• UltraChat - Large-scale dialogue dataset with interactive visualization
• OLMo Hard-Coded - Curated dataset from Allen Institute for AI
• FineWeb - Large web-crawled dataset for pretraining language models
• Big-Math-RL-Verified - Math problems with verified solutions for training reasoning models

Useful Demos

• Teachable Machine - GUI app for training simple machine learning models without code
• LLM Probability Distribution - Interactive visualization of how LLMs sample tokens based on probability
• BBycroft LLM Visualization - Visual explanation of how transformers and LLMs work internally
• Tiktokenizer - Interactive tool to see how text gets tokenized by different models
• SolidGoldMagikarp - Exploration of unusual tokens and their effects on LLM behavior

All Materials

• Complete Course Slides (ZIP, 85MB)
• Code Repository Access (GitHub)

Recommended Tools

RICEGen template

<endoftext\> 😉