Courses

Course

Computer Science - Parallel Computing

Indian Institute of Technology Delhi

This first course in parallel programming is designed for students without prior experience in this field but requires knowledge of data structures and operating systems. The course covers the following key areas:

  • Introduction to parallel programming paradigms
  • Understanding parallel architecture through case studies
  • Hands-on experience with OpenMP and MPI for shared memory and message passing
  • Exploration of algorithmic techniques, including CUDA for GPU programming
  • In-depth discussions on memory consistency and performance issues

With the increasing number of cores in modern processors, efficient programming for these architectures is vital for future technological advancements. The course consists of lectures and significant practical components, ensuring students gain valuable experience on compute clusters, multi-core CPUs, and massive-core GPUs.

Course Lectures
  • Mod-01 Lec-01 Introduction
    Mod-01 Lec-01 Introduction
    Dr. Subodh Kumar
    Play

    This module provides an introduction to the fundamentals of parallel computing. Students will learn about the necessity of parallel programming in the context of modern computing. Key topics include:

    • The evolution of parallel computing
    • Applications of parallelism in real-world scenarios
    • Basic concepts and terminology

    By the end of this module, students will have a solid foundation that prepares them for more advanced topics in subsequent lectures.

  • Mod-01 Lec-02 Parallel Programming Paradigms
    Mod-01 Lec-02 Parallel Programming Paradigms
    Dr. Subodh Kumar
    Play

    This module explores various parallel programming paradigms that empower developers to effectively utilize multiple processing units. Key areas covered include:

    1. Data parallelism
    2. Task parallelism
    3. Pipeline parallelism
    4. Shared memory vs. distributed memory models

    Students will engage in discussions and hands-on exercises to grasp how these paradigms influence program structure and performance.

  • Mod-01 Lec-03 Parallel Architecture
    Mod-01 Lec-03 Parallel Architecture
    Dr. Subodh Kumar
    Play

    In this module, students will learn about the architectural principles that underlie parallel computing systems. The focus will be on:

    • Understanding multi-core and many-core architectures
    • Memory hierarchies and their impact on performance
    • Interconnection networks and their role in parallelism
    • Scalability issues in architecture design

    This knowledge is crucial for writing efficient parallel programs that take full advantage of underlying hardware capabilities.

  • Mod-01 Lec-04 Parallel Architecture (case studies)
    Mod-01 Lec-04 Parallel Architecture (case studies)
    Dr. Subodh Kumar
    Play

    This module presents case studies that illustrate the practical applications of parallel architecture. Students will analyze:

    • Real-world parallel computing applications
    • Performance analysis of existing systems
    • Lessons learned from successful and unsuccessful implementations

    By examining these case studies, students will gain insights into best practices and pitfalls in parallel computing.

  • Mod-01 Lec-05 Open MP
    Mod-01 Lec-05 Open MP
    Dr. Subodh Kumar
    Play

    This module introduces OpenMP, a widely used API in parallel programming. Students will learn:

    • Basic syntax and constructs of OpenMP
    • How to parallelize loops and sections of code
    • Strategies for managing shared and private data

    Hands-on examples and exercises will reinforce learning, enabling students to write parallel code effectively using OpenMP.

  • Mod-01 Lec-06 Open MP(Contd.)
    Mod-01 Lec-06 Open MP(Contd.)
    Dr. Subodh Kumar
    Play

    This module continues the exploration of OpenMP, delving deeper into its advanced features and capabilities. Topics include:

    • Nested parallelism
    • Dynamic tasking
    • Synchronization techniques and reduction operations

    Students will participate in practical coding sessions to apply these advanced features in real-world scenarios.

  • Mod-01 Lec-07 Open MP(Contd..)
    Mod-01 Lec-07 Open MP(Contd..)
    Dr. Subodh Kumar
    Play

    This module further extends the study of OpenMP, focusing on more complex programming patterns. Students will explore:

    • Task dependencies and scheduling
    • Using OpenMP in conjunction with other parallel frameworks
    • Performance tuning techniques

    Through hands-on projects, students will learn to optimize their parallel applications for better performance.

  • Mod-01 Lec-08 Open MP&PRAM Model of Computation
    Mod-01 Lec-08 Open MP&PRAM Model of Computation
    Dr. Subodh Kumar
    Play

    In this final module, students will examine the PRAM model of computation and its relevance in parallel programming. Key aspects include:

    • Understanding the PRAM model's characteristics
    • Comparing PRAM with other models of computation
    • Applications of the PRAM model in algorithm design

    This foundational knowledge will help students appreciate theoretical underpinnings of parallel algorithms and their practical implications.

  • Mod-01 Lec-09 PRAM
    Mod-01 Lec-09 PRAM
    Dr. Subodh Kumar
    Play

    The PRAM (Parallel Random Access Machine) model is an essential concept in parallel computing. This module introduces students to the formal definition of PRAM and its significance in understanding parallel algorithms. Key topics include:

    • Basic structure and components of the PRAM model
    • Types of PRAM: EREW, CREW, and CRCW
    • Examples of algorithms suitable for PRAM
    • How PRAM relates to real-world parallel computing scenarios

    By the end of this module, students will have a solid understanding of the PRAM model and its applications in designing efficient parallel algorithms.

  • Mod-01 Lec-10 Models of Parallel Computation,Complexity
    Mod-01 Lec-10 Models of Parallel Computation,Complexity
    Dr. Subodh Kumar
    Play

    This module explores various models of parallel computation and their complexity. Students will learn about:

    • Different models of computation: PRAM, BSP, and more
    • The concept of computational complexity in parallel contexts
    • How to analyze the performance of parallel algorithms
    • Key theorems and principles in parallel complexity

    By exploring these topics, students will gain insights into how different models impact parallel performance and algorithm efficiency.

  • Mod-01 Lec-11 Memory Consistency
    Mod-01 Lec-11 Memory Consistency
    Dr. Subodh Kumar
    Play

    Memory consistency is crucial for ensuring correct execution of parallel programs. This module covers:

    • Fundamentals of memory consistency models
    • Impact of memory consistency on parallel programming
    • Examples of different consistency models
    • Challenges in maintaining memory consistency

    Students will learn how various models affect program behavior and performance in multi-threaded environments.

  • Mod-01 Lec-12 Memory Consistency & Performance Issues
    Mod-01 Lec-12 Memory Consistency & Performance Issues
    Dr. Subodh Kumar
    Play

    This module expands on the topic of memory consistency, highlighting performance issues that arise in parallel systems. Key areas of focus include:

    • Performance overheads related to memory consistency
    • Strategies for optimizing memory access in parallel applications
    • Trade-offs between consistency and performance
    • Case studies showcasing real-world applications

    Students will learn how to identify and mitigate performance bottlenecks caused by memory consistency constraints.

  • Mod-01 Lec-13 Parallel Program Design
    Mod-01 Lec-13 Parallel Program Design
    Dr. Subodh Kumar
    Play

    This module covers the fundamentals of parallel program design. Students will explore key concepts that include:

    • Principles of designing efficient parallel algorithms
    • Approaches to decomposing tasks for parallel execution
    • Common pitfalls in parallel program design
    • Tools and frameworks for developing parallel applications

    By the end of this module, participants will be equipped to create robust and efficient parallel programs.

  • Mod-01 Lec-14 Shared Memory & Message Passing
    Mod-01 Lec-14 Shared Memory & Message Passing
    Dr. Subodh Kumar
    Play

    This module introduces students to shared memory and message passing paradigms in parallel computing. Important topics include:

    • Difference between shared memory and message passing models
    • Examples of shared memory architectures
    • Implementation techniques for message passing
    • Challenges and advantages of each paradigm

    Understanding these paradigms is essential for effective parallel programming and optimizing resource utilization.

  • Mod-01 Lec-15 MPI
    Mod-01 Lec-15 MPI
    Dr. Subodh Kumar
    Play

    This module focuses on the Message Passing Interface (MPI), a standardized method for communication in parallel computing. Key areas covered include:

    • Introduction to MPI and its purpose
    • Key MPI functions and their applications
    • Strategies for effective message passing
    • Real-world examples of MPI usage in parallel applications

    Students will gain practical skills in employing MPI for developing distributed applications, enhancing their parallel programming capabilities.

  • Mod-01 Lec-16 MPI(Contd.)
    Mod-01 Lec-16 MPI(Contd.)
    Dr. Subodh Kumar
    Play

    This module continues the exploration of MPI, delving into more advanced concepts and techniques. Students will cover:

    • Advanced MPI functions and their use cases
    • Performance tuning for MPI applications
    • Debugging and profiling tools for MPI
    • Case studies on high-performance computing with MPI

    By the end of this module, students will be adept at maximizing performance and troubleshooting MPI-based applications.

  • Mod-01 Lec-17 MPI(Contd..)
    Mod-01 Lec-17 MPI(Contd..)
    Dr. Subodh Kumar
    Play

    This module continues the exploration of the Message Passing Interface (MPI), a standard for parallel programming. Students will delve deeper into MPI's capabilities, examining various communication techniques essential for effective parallel processes.

    Key topics include:

    • Advanced MPI functions and their applications
    • Optimizing communication patterns
    • Debugging and performance analysis tools for MPI
  • Mod-01 Lec-18 Algorithmic Techniques
    Mod-01 Lec-18 Algorithmic Techniques
    Dr. Subodh Kumar
    Play

    This module introduces algorithmic techniques crucial for parallel programming. Students will learn about various algorithms that leverage concurrency to enhance performance.

    Topics to be covered include:

    • Parallel sorting algorithms
    • Graph algorithms with parallel approaches
    • Workload balancing and task scheduling techniques
  • Mod-01 Lec-19 Algorithmic Techniques(Contd.)
    Mod-01 Lec-19 Algorithmic Techniques(Contd.)
    Dr. Subodh Kumar
    Play

    This module continues the discussion on algorithmic techniques, focusing on further optimization strategies and examples of successful implementations in various scenarios.

    Students will explore:

    • Advanced concepts in parallel algorithms
    • Case studies of real-world applications
    • Performance comparison of different algorithmic approaches
  • Mod-01 Lec-20 Algorithmic Techniques(Contd..)
    Mod-01 Lec-20 Algorithmic Techniques(Contd..)
    Dr. Subodh Kumar
    Play

    This module further elaborates on algorithmic techniques, emphasizing real-time application of learned concepts in practical scenarios.

    Key learning points include:

    • Hands-on implementation of algorithms
    • Analysis of algorithm performance in parallel contexts
    • Integration of theoretical knowledge into practical programming tasks
  • Mod-01 Lec-21 CUDA
    Mod-01 Lec-21 CUDA
    Dr. Subodh Kumar
    Play

    This module introduces CUDA (Compute Unified Device Architecture), a parallel computing platform and programming model developed by NVIDIA. Students will learn how to leverage GPU computing to accelerate applications.

    Key topics include:

    • Understanding the CUDA architecture
    • Writing basic CUDA kernels
    • Memory management in CUDA
  • Mod-01 Lec-22 CUDA(Contd.)
    Mod-01 Lec-22 CUDA(Contd.)
    Dr. Subodh Kumar
    Play

    This module continues the exploration of CUDA, enhancing students' skills in writing more complex kernels and optimizing performance for specific applications.

    Topics covered include:

    • Advanced kernel development techniques
    • Performance profiling and optimization strategies
    • Best practices for memory usage and data transfer
  • Mod-01 Lec-23 CUDA(Contd..)
    Mod-01 Lec-23 CUDA(Contd..)
    Dr. Subodh Kumar
    Play

    This module further expands on CUDA, with a focus on implementing real-world applications and case studies that utilize CUDA for performance enhancements.

    Students will learn about:

    • Case studies of CUDA applications in various fields
    • Integrating CUDA with other programming frameworks
    • Debugging and optimizing CUDA applications
  • Mod-01 Lec-24 CUDA(Contd...)
    Mod-01 Lec-24 CUDA(Contd...)
    Dr. Subodh Kumar
    Play

    This module concludes the CUDA series, emphasizing the latest developments and future directions in GPU programming. Students will explore emerging trends and technologies.

    Topics include:

    • Recent advancements in CUDA
    • Future trends in GPU computing
    • Preparing for industry demands in parallel computing
  • Mod-01 Lec-25 CUDA(Contd....)
    Mod-01 Lec-25 CUDA(Contd....)
    Dr. Subodh Kumar
    Play

    This module continues the exploration of CUDA programming, focusing on advanced features and optimization techniques. Students will learn how to:

    • Implement parallel algorithms effectively using CUDA.
    • Optimize memory usage and performance in GPU applications.
    • Utilize CUDA libraries for enhanced computational efficiency.

    By the end of this module, students will have practical experience in enhancing CUDA applications and understanding best practices for parallel programming.

  • Mod-01 Lec-26 CUDA(Contd.....)
    Mod-01 Lec-26 CUDA(Contd.....)
    Dr. Subodh Kumar
    Play

    This module further extends students' knowledge of CUDA programming, emphasizing real-world applications and problem-solving strategies. Topics covered will include:

    • Debugging techniques for CUDA applications.
    • Profiling and analyzing GPU performance.
    • Case studies on CUDA applications in various industries.

    Students will gain hands-on experience through coding assignments and projects, reinforcing their understanding of effective CUDA programming.

  • Mod-01 Lec-27 CUDA(Contd......)
    Mod-01 Lec-27 CUDA(Contd......)
    Dr. Subodh Kumar
    Play

    This module continues the CUDA journey by introducing more complex concepts and techniques. Students will explore:

    • Dynamic parallelism in CUDA.
    • Optimizing data transfer between host and device.
    • Memory coalescing and its impact on performance.

    Through practical assignments, students will implement dynamic parallelism and understand its advantages for performance improvement.

  • Mod-01 Lec-28 Algorithms,Merging & Sorting
    Mod-01 Lec-28 Algorithms,Merging & Sorting
    Dr. Subodh Kumar
    Play

    This module introduces students to essential algorithms for merging and sorting in parallel computing. Key topics include:

    • Parallel sorting algorithms and their applications.
    • Efficient merging techniques in distributed systems.
    • Analyzing time complexity of parallel algorithms.

    Students will implement these algorithms in practical exercises, gaining insights into their performance and scalability.

  • Mod-01 Lec-29 Algorithms,Merging & Sorting(Contd.)
    Mod-01 Lec-29 Algorithms,Merging & Sorting(Contd.)
    Dr. Subodh Kumar
    Play

    This module continues the exploration of merging and sorting algorithms, delving into more complex scenarios. Topics covered will include:

    • Advanced parallel sorting techniques such as bitonic and sample sort.
    • Handling large data sets and optimizing memory usage.
    • Case studies demonstrating real-world applications.

    Students will work on projects that challenge them to implement and optimize these algorithms in practical settings.

  • Mod-01 Lec-30 Algorithms,Merging & Sorting(Contd..)
    Mod-01 Lec-30 Algorithms,Merging & Sorting(Contd..)
    Dr. Subodh Kumar
    Play

    This module further builds upon the concepts of merging and sorting, focusing on the integration of these techniques in parallel applications. Key areas of study include:

    • Implementation of hybrid sorting algorithms.
    • Performance comparison of various sorting techniques.
    • Optimizing algorithm performance through parallelization.

    Students will engage in coding exercises that allow them to apply these techniques effectively in real-world scenarios.

  • Mod-01 Lec-31 Algorithms,Merging & Sorting(Contd...)
    Mod-01 Lec-31 Algorithms,Merging & Sorting(Contd...)
    Dr. Subodh Kumar
    Play

    This module continues to advance students' knowledge of algorithms, focusing on further complexities and advanced techniques. Students will cover:

    • Data structures that enhance algorithm efficiency.
    • Techniques for reducing computational overhead.
    • Practical applications of advanced merging and sorting algorithms.

    Students will engage in collaborative projects that emphasize teamwork and the application of theoretical knowledge in practical situations.

  • Mod-01 Lec-32 Algorithms,Merging & Sorting(Contd....)
    Mod-01 Lec-32 Algorithms,Merging & Sorting(Contd....)
    Dr. Subodh Kumar
    Play

    This final module wraps up the course by summarizing the key concepts learned throughout the course. Students will review:

    • Key challenges and solutions in parallel computing.
    • Recent advancements in parallel programming.
    • Future trends and directions in the field.

    Students will also present their final projects, demonstrating their comprehensive understanding of parallel computing principles.

  • Mod-01 Lec-33 Lower Bounds Lock Free Synchronization,Load Stealing
    Mod-01 Lec-33 Lower Bounds Lock Free Synchronization,Load Stealing
    Dr. Subodh Kumar
    Play

    This module focuses on the critical concepts of lower bounds, lock-free synchronization, and load stealing in parallel computing. Students will learn about:

    • Theoretical foundations of lower bounds in parallel algorithms.
    • Techniques for achieving lock-free synchronization, which is crucial for building efficient concurrent data structures.
    • Load stealing strategies that help in balancing workloads across multiple cores and enhancing overall system performance.

    Through hands-on exercises, participants will implement these concepts and analyze their impact on performance in various computing environments.

  • Mod-01 Lec-34 Lock Free Synchronization,Graph Algorithms
    Mod-01 Lec-34 Lock Free Synchronization,Graph Algorithms
    Dr. Subodh Kumar
    Play

    This module delves into the intersection of lock-free synchronization and graph algorithms in parallel programming. Key topics include:

    • Understanding how lock-free data structures can enhance graph algorithm performance.
    • Implementing and analyzing key graph algorithms such as breadth-first search and depth-first search in a lock-free manner.
    • Exploring the challenges and solutions related to synchronization when processing graphs in a parallel environment.

    Students will gain practical experience by coding these algorithms and assessing their efficiency in multi-threaded settings.

About | Terms | Privacy | Contact Us | Schools
© 2009-2025 Courses.com. All rights reserved.