Fully Sharded Data Parallel

Related topics

• Fully Sharded Data Parallel is an alternative to pipeline parallelism From the official documentation: “Achieving maximum performance in Multislice environments requires considering how to shard across the multiple slices. There are typically three choices (data parallelism, fully-sharded data parallelism and pipeline parallelism).”