К содержимому
learnspaceYOUR NEXT CHAPTER
ПРОСТРАНСТВО ОБУЧЕНИЯ
ГлавнаяКаталог курсовМоё обучениеCoursera

Знания без границ

Учитесь у лучших университетов и компаний мира.

Открыть Coursera
Интеграция
Пространство университета
Моё пространствоСтраница курса
↵
ЯЛичный кабинетСтудент
© 2026 LearnSpaceКаждый день — возможность узнать больше.Помощь
Train Large Language Models Faster - Parallelism Deep Dive · LearnSpace
Назад в каталог
courseraIT и технологии

Train Large Language Models Faster - Parallelism Deep Dive

Курс от Packt
Средний≈ 17.2 чАнглийский
О курсеНавыкиПрограммаПреподаватели

О курсе

This course features Coursera Coach! A smarter way to learn with interactive, real-time conversations that help you test your knowledge, challenge assumptions, and deepen your understanding as you progress through the course. This course focuses on accelerating the training of large language models (LLMs) through parallelism strategies. By exploring techniques such as data, model, and hybrid parallelism, you will learn how to optimize training processes for faster results. The course breaks down complex topics in a structured way, starting with an introduction to parallel computing and scaling laws, before diving into hands-on applications using popular libraries like PyTorch and DeepSpeed. You will also gain practical experience running parallelism strategies on multi-GPU systems and exploring fault tolerance techniques to ensure reliable training. The course integrates theoretical concepts with real-world examples to provide a comprehensive understanding of LLM training. Throughout the course, you will explore various types of parallelism—data, model, pipeline, and tensor parallelism—and their applications in LLMs. You’ll work with datasets like MNIST and WikiText, gaining hands-on experience implementing parallel strategies to optimize training speed. The course culminates in an exploration of advanced checkpointing strategies and fault tolerance methods, ensuring you understand how to recover from system failures during training. This course is perfect for learners interested in optimizing machine learning workflows and accelerating AI model development. A background in machine learning or deep learning is recommended, and the course is suitable for intermediate learners seeking to deepen their knowledge of LLM training strategies. By the end of the course, you will be able to implement and compare various parallelism techniques for LLM training, run distributed training on multi-GPU environments, apply fault tolerance strategies, and understand advanced topics in parallel computing.

Навыки, которые вы освоите

Model TrainingModel Optimization

Программа курса

16 модулей · 129 учебных материалов

01Introduction4 материалов

Introduction

Introduction & What Is This Course AboutВидеоFull Course ResourceЧтениеCourse StructureВидеоDEMO - What You'll Build in This CourseВидео
02Strategies for Parallelizing LLMS - Deep Dive

Учитесь у экспертов

Packt - Course Instructors

Преподаватель курса

Train Large Language Models Faster - Parallelism Deep Dive
В каталоге вашей программы

Инвестируйте в себя

Новые знания — в удобное для вас время.

Начать на Coursera

Обучение откроется на Coursera
в новой вкладке

Обучение на Coursera

≈ 17.2 ч

16 модулей

Язык: Английский

Часть программы вашего университета
6 материалов

Strategies for Parallelizing LLMS - Deep Dive

What is Parallelism and Why it MattersВидеоUnderstanding the Single GPU StrategyВидеоUnderstanding the Parallel Strategy and AdvantagesВидеоParallelism vs Single GPU - SummaryВидеоUnderstanding Parallelism in Large Language Model TrainingDIALOGUEStrategies for Parallelizing LLMS - Deep Dive - AssessmentЗадание
03IT Fundamental Concepts12 материалов

IT Fundamental Concepts

IT Fundamentals - IntroductionВидеоIntroduction to Cloud Computing and Traditional ITВидеоWhat is a Computer - CPU and RAM OverviewВидеоData Storage and File SystemsВидеоOS File System StructureВидеоLAN IntroductionВидеоWhat is the InternetВидеоInternet Communication Deep DiveВидеоUnderstanding Servers and ClientsВидеоGPUs - OverviewВидеоIT Fundamentals: Hardware, Networking, and the CloudDIALOGUEIT Fundamental Concepts - AssessmentЗадание
04GPU Architecture for LLM Training Deep Dive4 материалов

GPU Architecture for LLM Training Deep Dive

GPU Architecture for LLM TrainingВидеоWhy this Architecture ExcelsВидеоUnderstanding GPU Architecture and Large Language Model TrainingDIALOGUEGPU Architecture for LLM Training Deep Dive - AssessmentЗадание
05Deep and Machine Learning - Deep Dive13 материалов

Deep and Machine Learning - Deep Dive

Machine and Deep Learning IntroductionВидеоDeep and Machine Learning - Overview and BreakdownВидеоDeep Learning Key AspectsВидеоDeep Neural Networks - Deep DiveВидеоThe Single Neuron Computation - Deep DiveВидеоWeightsВидеоActivation Functions - Deep DiveВидеоDeep Learning - SummaryВидеоMachine Learning Introduction - ML vs DLВидеоLearning Types and Full ML & DL Analogy ExampleВидеоDL and ML Comparative Capabilities - SummaryВидеоUnderstanding Machine Learning and Deep Learning FundamentalsDIALOGUEDeep and Machine Learning - Deep Dive -AssessmentЗадание
06Large Language Models - Fundamentals of AI and LLMs7 материалов

Large Language Models - Fundamentals of AI and LLMs

IntroductionВидеоThe Transformer Architecture FundamentalsВидеоThe Self-Attention Mechanism - AnalogyВидеоThe Transformer Architecture AnimationВидеоThe Transformer Library - Deep DiveВидеоUnderstanding Transformer Architecture & Self-AttentionDIALOGUELarge Language Models - Fundamentals of AI and LLMs - AssessmentЗадание
07Parallel Computing Fundamentals & Parallelism in LLM Training4 материалов

Parallel Computing Fundamentals & Parallelism in LLM Training

Parallel Computing Introduction - Key ConceptsВидеоParallel Computing Fundamentals and Scaling Laws - Deep DiveВидеоFundamentals of Parallel Computing: Nodes, Scaling Laws, and ArchitectureDIALOGUEParallel Computing Fundamentals & Parallelism in LLM Training - AssessmentЗадание
08Types of Parallelism in LLM Training - Data, Model, and Hybrid Parallelism13 материалов

Types of Parallelism in LLM Training - Data, Model, and Hybrid Parallelism

Types of Parallelism in LLM TrainingВидеоData Parallelism - How It WorksВидеоData Parallelism Advantages for LLM TrainingВидеоReal-world Example - Data Parallelism in GPT-3 TrainingВидеоModel Parallelism and Tensor Parallelism and Layer Parallelism - Deep DiveВидеоLLM Relevance and ImplementationВидеоModel vs Data ParallelismВидеоKey Differences Highlighted - Data vs Model ParallelismВидеоData vs Model ParallelismВидеоHybrid Parallelism - AnimationВидеоHybrid Parallelism - What is It and MotivationВидеоUnderstanding Parallel Computing Strategies in Large Language Model TrainingDIALOGUETypes of Parallelism in LLM Training - Data, Model, and Hybrid Parallelism - AssessmentЗадание
09Types of Parallelism - Pipeline and Tensor Parallelism13 материалов

Types of Parallelism - Pipeline and Tensor Parallelism

Pipeline Parallelism OverviewВидеоPipeline Parallelism Key Concepts and How it Works - Step by StepВидеоPipeline Bubbles Key ConceptsВидеоPipeline Schedules Key ConceptsВидеоActivation Recomputation - Overview and IntroductionВидеоNeural Network and Activation and Backward and Forward Passes - Full DiveВидеоUnderstanding Activation Recomputation vs Standard Training - Deep DiveВидеоDemo - Activation Recomputation VisualizationВидеоActivation Recomputation vs Standard ApproachВидеоBenefits of Activation Recomputation and Implementation StrategiesВидеоPipeline Parallelism Implementation Frameworks and Key TakeawaysВидеоUnderstanding Pipeline Parallelism and Activation Recomputing in Large Language Model TrainingDIALOGUETypes of Parallelism - Pipeline and Tensor Parallelism - AssessmentЗадание
10Tensor Parallelism - Deep Dive10 материалов

Tensor Parallelism - Deep Dive

What is Tensor Parallelism and Why - BenefitsВидеоTensor Parallel Pizza Making AnalogyВидеоTensors and Partitioning Strategies - Deep DiveВидеоTensor Communication Patterns - Deep DiveВидеоDevice Mesh Communication Pattern - Deep DiveВидеоHow Components Work Together in Distributed LLM TrainingВидеоUnderstanding Tensor Parallelism with LEGO Bricks Animation DemoВидеоPutting it All Together - All Strategies in LLM TrainingВидеоUnderstanding and Applying Tensor Parallelism in Distributed LLM TrainingDIALOGUETensor Parallelism - Deep Dive - AssessmentЗадание
11HANDS-ON: Strategies for Parallelism - Data Parallelism Deep Dive13 материалов

HANDS-ON: Strategies for Parallelism - Data Parallelism Deep Dive

Strategies for Parallelizing LLMs - Hands-on IntroductionВидеоPytorch - LLM Training Library OverviewВидеоThe Transformers Library - OverviewВидеоNumpy OverviewВидеоTorchVision and TorchDistributed OverviewВидеоDeepSpeed and Megatron-LM - OverviewВидеоDatasets and Why this ToolkitВидеоHANDS-On: Data Parallelism - Training a Small Model - MNIST DatasetВидеоTesting Pseudo Data Parallelism Trained ModelВидеоHANDS-ON: Data Parallelism - Colab - Full DemoВидеоData Parallelism - Simulated Parallelism on GPU TakeawaysВидеоUnderstanding and Implementing Data Parallelism in Machine LearningDIALOGUEHANDS-ON: Strategies for Parallelism - Data Parallelism Deep Dive - AssessmentЗадание
12HANDS-ON: Data Parallelism w/ WikiText Dataset & DeepSpeed Mem. Optimization5 материалов

HANDS-ON: Data Parallelism w/ WikiText Dataset & DeepSpeed Mem. Optimization

Hands-on: Data Parallelism - Wikitext-2 DatasetВидеоDeepSpeed - Full DiveВидеоHands-on: Data Parallelism with DeepSpeed OptimizationВидеоImplementing Data Parallelism and Memory Optimization with DeepSpeedDIALOGUEHANDS-ON: Data Parallelism w/ WikiText Dataset & DeepSpeed Mem. Optimization - AssessmentЗадание
13Running TRUE Parallelism on Multiple GPU Systems - Runpod.io7 материалов

Running TRUE Parallelism on Multiple GPU Systems - Runpod.io

Setup Runpod.io Environment OverviewВидеоRunpod SSH SetupВидеоSetting up Runpod Parallelism in JupyterNotebookВидеоHANDS-ON - Parallelism with IMDB Dataset - Deep Dive - True ParallelismВидеоRunpod CleanupВидеоSetting Up and Running Multi-GPU Training on runpod.ioDIALOGUERunning TRUE Parallelism on Multiple GPU Systems - Runpod.io - AssessmentЗадание
14Fault Tolerance and Scalability & Advanced Checkpointing Strategies - Deep Dive12 материалов

Fault Tolerance and Scalability & Advanced Checkpointing Strategies - Deep Dive

Fault Tolerance Introduction & Types of Failures in Distributed LLM TrainingВидеоStrategies for Fault ToleranceВидеоCheckpointing in LLM Training - AnimationВидеоBasic Checkpointing in LLM TrainingВидеоIncremental Checkpointing in LLM TrainingВидеоAsynchronous Checkpointing in LLM TrainingВидеоMulti-level Checkpointing in LLM Training - AnimationВидеоCheckpoint Storage Considerations - Deep DiveВидеоImplementing a Hybrid Approach - Performance, Failure, Optimizations - Full DiveВидеоCheckpoint Storage Strategy - SummaryВидеоUnderstanding Fault Tolerance and Checkpoint Storage in Large Language Model TrainingDIALOGUEFault Tolerance and Scalability & Advanced Checkpointing Strategies - Deep Dive - AssessmentЗадание
15Advanced Topics and Emerging Trends3 материалов

Advanced Topics and Emerging Trends

Advanced Topics and Emerging TrendsВидеоUnderstanding Large Language Model Parallelization StrategiesDIALOGUEAdvanced Topics and Emerging Trends - AssessmentЗадание
16Wrap up and Next Steps3 материалов

Wrap up and Next Steps

Course Summary and Next StepsВидеоFull Course Practice AssessmentЗаданиеFull Course AssessmentЗадание