University of Maryland · Fall 2026

Multimodal
foundation
models.

A research-driven course on the models that connect language, vision, audio, and action—from the transformer’s first principles to capable multimodal systems.

Explore the schedule ↓Official listing ↗Piazza ↗

WhenTue / Thu · 12:30–1:45 PM

WhereCSI 3117

InstructorJia-Bin Huang

#	Date	Lecture	Slides	Supp video
Module 01 Transformer foundations
L01	Tue · Sep 1	Introduction Course logistics Multimodal foundation model applications	Slides ↗
L02	Thu · Sep 3	Transformer Tokenization Sequence-to-sequence architecture	Slides ↗
L03	Tue · Sep 8	Position embedding Absolute position Relative position Rotary position embeddings	Slides ↗
L04	Thu · Sep 10	Attention design Key-value cache Multi-head attention Multi-query attention Grouped-query attention Multi-head latent attention DeepSeek Sparse attention (DSA)	Slides ↗
L05	Tue · Sep 15	Flash Attention GPUs Tiling algorithm Online softmax FlashAttention	Slides ↗
L06	Thu · Sep 17	Linear attention Linear attention Chunkwise parallel training Test-time regression perspective Delta update rule	Slides ↗
L07	Tue · Sep 22	Mixture of Experts 1 Feedforward layers Activations Load balancing Router z-loss	Slides ↗
L08	Thu · Sep 24	Mixture of Experts 2	Slides ↗
L09	Tue · Sep 29	Residual connections Residuals Hyper connections Attention residuals Gated residual	Slides ↗
M1	Thu · Oct 1	Midterm 1 — in-class exam	—
Module 02 Large Language Models
L11	Tue · Oct 6	Pretraining and scaling laws Next-token prediction Data, model, and compute scaling Chinchilla-optimal training	Slides ↗
L12	Thu · Oct 8	Prompting and PEFT Low-Rank Adaptation LoRA+ Quantized Low-Rank Adaptation Vector-based Random Matrix Adaptation Weight-Decomposed Low-Rank Adaptation	Slides ↗
L13	Tue · Oct 13	Fall Break	—
L14	Thu · Oct 15	Post-training Reinforcement Learning from Human Feedback Proximal Policy Optimization Direct Preference Optimization	Slides ↗
L15	Tue · Oct 20	Efficient training Mixed precision training ZeRO	Slides ↗
L16	Thu · Oct 22	Large-scale training 5D parallelism	Slides ↗
L17	Tue · Oct 27	Efficient inference Quantization Speculative decoding PagedAttention
Module 03 Multimodal models
L18	Thu · Oct 29	Large multimodal models Example state-of-the-art models	—
L19	Tue · Nov 3	Self-supervised representation learning SimCLR DINO Masked autoencoders CLIP JEPA
M2	Thu · Nov 5	Midterm 2 — in-class exam	—
L21	Tue · Nov 10	Diffusion Variational autoencoders Diffusion training Guidance Latent diffusion
L22	Tue · Nov 17	Flow matching Flow matching foundations MeanFlow Efficient sampling
Module 04 Systems & applications
L23	Thu · Nov 19	Applications · robot learning Vision-language-action Policy learning World models	—
L24	Tue · Nov 24	Agentic AI 1 — reasoning & tool use Test-time scaling Tool calling and function APIs Web search, code execution, and computer use	—
—	Nov 25–29	Thanksgiving recess	—
L25	Tue · Dec 1	Agentic AI 2 — agent harnesses Planning and orchestration Memory and state Tracing, evaluations, and guardrails	—
M3	Thu · Dec 3	Midterm 3 — in-class exam	—
L26	Tue · Dec 8	Applications · video & audio Multimodal models for video Multimodal models for audio	—
L27	Thu · Dec 10	Applications · 3D Multimodal models for 3D understanding 3D creation	—

Date

Lecture

Slides

Supp video

Module 01 Transformer foundations

L01

Tue · Sep 1

Introduction