The runtime includes these parts:

• activation_checkpointing
• comm
• compression
• data_pipeline
• fp16
• pipe
• swap_tensor
• zero

The entry for the runtime is in https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/engine.py

Elasticity? not supported with model parallelism

DeepSpeed uses gradient accumulation to extract pipeline parallelism.

DeepSpeed Architecture(from https://www.youtube.com/watch?v=zqsOEzKZX2Y&t):

There are major two parts: runtime and ops.

DeepSpeed runtime study: http://aheader.org/2021/10/02/deepspeed-runtime-study/

DeepSpeed ops study: