Accelerate documentation

๐Ÿค— accelerate์˜ ์‹œํ€€์Šค ๋ณ‘๋ ฌ

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

๐Ÿค— accelerate์˜ ์‹œํ€€์Šค ๋ณ‘๋ ฌ

์ด ๊ฐ€์ด๋“œ์—์„œ๋Š” ๐Ÿค—accelerate์—์„œ ์‹œํ€€์Šค ๋ณ‘๋ ฌ์„ฑ์„ ์‚ฌ์šฉํ•˜๋Š” ๊ธฐ๋ณธ ์‚ฌํ•ญ์„ ๋‹ค๋ฃน๋‹ˆ๋‹ค.

๋งค์šฐ ๊ด€๋ จ์ด ์žˆ๋Š” Context Parallelism๋„ ์ฐธ๊ณ ํ•˜์„ธ์š”.

์™œ ์‹œํ€€์Šค ๋ณ‘๋ ฌ์„ฑ์ธ๊ฐ€?

๋Œ€ํ˜• ์–ธ์–ด ๋ชจ๋ธ, ๊ทธ๋ฆฌ๊ณ  ์ตœ๊ทผ์—๋Š” ์ถ”๋ก  ๋ชจ๋ธ์˜ ๋“ฑ์žฅ์œผ๋กœ ์‹œํ€€์Šค ๊ธธ์ด๊ฐ€ ๋น ๋ฅด๊ฒŒ ์ฆ๊ฐ€ํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์— attention์˜ 2์ฐจ ๋ฉ”๋ชจ๋ฆฌ ๋ณต์žก๋„๊ฐ€ ๋”ํ•ด์ง€๋ฉด์„œ, ๊ธด ์‹œํ€€์Šค๋กœ ๋ชจ๋ธ์„ ํ•™์Šตํ•˜๋Š” ๋” ํšจ์œจ์ ์ธ ๋ฐฉ์‹์— ๋Œ€ํ•œ ํ•„์š”์„ฑ์ด ์ปค์กŒ์Šต๋‹ˆ๋‹ค. ์‹œํ€€์Šค ๊ธธ์ด๊ฐ€ 128k์ผ ๋•Œ, vanilla attention ๊ตฌํ˜„์„ ๊ฐ€์ •ํ•˜๋ฉด bf16 ์ •๋ฐ€๋„์—์„œ attention matrix์˜ ๋ฉ”๋ชจ๋ฆฌ ์š”๊ตฌ๋Ÿ‰์€ 128k * 128k * 2 bytes * num_heads = ~32 GB * num_heads์ž…๋‹ˆ๋‹ค. ๋ฌผ๋ก , ์ด๋Ÿฌํ•œ attention ๊ฐ€์ค‘์น˜๋ฅผ ๋ฌผ๋ฆฌ์ ์œผ๋กœ ์ƒ์„ฑํ•˜์ง€ ์•Š๋Š” flash attention๋‚˜ SDPA๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰์ด ํฌ๊ฒŒ ์ค„์ง€๋งŒ, ๋ฉ”๋ชจ๋ฆฌ ์š”๊ตฌ๋Ÿ‰์˜ ์ฆ๊ฐ€ ํญ์€ ์—ฌ์ „ํžˆ ์ƒ๋‹นํ•ฉ๋‹ˆ๋‹ค.

Ulysses Sequence parallelism์€ attention ๊ณ„์‚ฐ์˜ ์ž…๋ ฅ์„ ์‹œํ€€์Šค ์ฐจ์›๋ณ„๋กœ ์ƒค๋”ฉํ•˜๊ณ , ๊ฐ GPU์—์„œ attention head์˜ ์ผ๋ถ€๋งŒ ์‚ฌ์šฉํ•˜์—ฌ attention์„ ์ •์ƒ์ ์œผ๋กœ ๊ณ„์‚ฐํ•  ์ˆ˜ ์žˆ๊ฒŒ ํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฅผ ํ†ตํ•ด ๋ช‡ ๊ฐ€์ง€ ์ถ”๊ฐ€ ๋„๊ตฌ๋ฅผ ํ™œ์šฉํ•˜์—ฌ 15M+ ์‹œํ€€์Šค ๊ธธ์ด์— ์Šค์ผ€์ผ๋งํ•˜์—ฌ ๊ธด ์‹œํ€€์Šค๋ฅผ ๊ฐ€์ง„ ๋ชจ๋ธ์„ ํ•™์Šตํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. Ulysses SP์— TiledMLP, Liger-Kernel, Activation checkpoint offload to cpu ๋ฐ ๊ธฐํƒ€ ๋ช‡ ๊ฐ€์ง€ ํŠธ๋ฆญ์„ ๊ฒฐํ•ฉํ•˜๋Š” ๋ฐฉ๋ฒ•์— ๋Œ€ํ•ด์„œ๋Š” ๋‹ค์Œ ๋…ผ๋ฌธ์„ ์ฐธ๊ณ ํ•˜์„ธ์š”: Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences.

Ulysses SP์™€ FSDP CP์˜ ์ฐจ์ด์ ์€ ๋ฌด์—‡์ธ๊ฐ€์š”?

Context Parallelism ๋ฌธ์„œ์—์„œ๋Š” ์‹œํ€€์Šค ์ฐจ์›์„ ์Šฌ๋ผ์ด์Šคํ•˜๋˜ ํ—ค๋“œ ์ฐจ์› ์Šฌ๋ผ์ด์‹ฑ ๋Œ€์‹  Ring Attention์„ ์‚ฌ์šฉํ•˜๋Š” ๋˜ ๋‹ค๋ฅธ ๊ธฐ์ˆ ์ธ Context Parallelism์˜ ๋ฐฐํฌ์— ๋Œ€ํ•ด ๋ฐฐ์šธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋‹ค์Œ ๊ธ€์—์„œ๋Š” ๋‘ ๊ธฐ์ˆ ์˜ ์ฐจ์ด๋ฅผ ๋งค์šฐ ์ƒ์„ธํ•˜๊ฒŒ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค:

ํ•œ ๊ธฐ์‚ฌ์˜ ๋‚ด์šฉ์„ ์ฐธ๊ณ ํ•˜์—ฌ ๊ฐ„๋‹จํžˆ ์š”์•ฝํ•˜๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค:

  • Ulysses SP๋Š” ํ†ต์‹  ์˜ค๋ฒ„ํ—ค๋“œ๊ฐ€ ์ƒ๋Œ€์ ์œผ๋กœ ๋‚ฎ์ง€๋งŒ, Attention Heads์˜ ์ˆ˜์— ์˜ํ•ด ์ œํ•œ๋˜๋ฏ€๋กœ ๋„คํŠธ์›Œํฌ ํ† ํด๋กœ์ง€(๋‹จ์ผ ๋ฆฌํ”Œ๋ฆฌ์นด์— ์ฐธ์—ฌํ•˜๋Š” GPU ์ˆ˜๋กœ attention heads์˜ ์ˆ˜๊ฐ€ ๋‚˜๋ˆ„์–ด์ ธ์•ผ ํ•จ)์— ๋Œ€ํ•ด ํŠน์ • ์š”๊ตฌ ์‚ฌํ•ญ์ด ์žˆ์Šต๋‹ˆ๋‹ค. All-to-all ํ†ต์‹ ์€ ์ง€์—ฐ ์‹œ๊ฐ„์— ๋ฏผ๊ฐํ•˜๋ฉฐ Deepspeed๊ฐ€ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • FSDP CP Ring-Attention์˜ P2P ๋ง ํ†ต์‹ ์€ aforementioned ๊ฐ€๋ถ„์„ฑ ์š”๊ตฌ์‚ฌํ•ญ์ด ์—†์ง€๋งŒ, ํ†ต์‹ ๋Ÿ‰์ด ๋” ๋†’์Šต๋‹ˆ๋‹ค.

๋งˆ์ง€๋ง‰์œผ๋กœ, USP: A Unified Sequence Parallelism Approach for Long Context Generative AI ๋…ผ๋ฌธ์—์„œ ์„ค๋ช…ํ•œ ๋Œ€๋กœ SP + CP๋ฅผ ๊ฒฐํ•ฉํ•˜์—ฌ ๋” ๊ธด ์‹œํ€€์Šค ๊ธธ์ด๋ฅผ ์ง€์›ํ•˜๋Š” ๊ฒƒ์ด ๊ฐ€๋Šฅํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๋‹ค๋งŒ, ์ด ๊ธฐ๋Šฅ์€ ์•„์ง ๐Ÿค—accelerate์— ํ†ตํ•ฉ๋˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค.

์ง€์›๋˜๋Š” ์‹œํ€€์Šค ๋ณ‘๋ ฌ์„ฑ ๋ฐฑ์—”๋“œ

ํ˜„์žฌ ์œ ์ผํ•œ ์‹œํ€€์Šค ๋ณ‘๋ ฌ์„ฑ ๋ฐฑ์—”๋“œ๋Š” deepspeed์ž…๋‹ˆ๋‹ค. ์ด๋Š” Arctic Long Sequence Training ๊ธฐ์ˆ ์˜ ์ผ๋ถ€์ธ ํ˜„๋Œ€ํ™”๋œ Ulysses SP์—์„œ ํŒŒ์ƒ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ์ง์ ‘ ์ž์ฒด ์ฝ”๋“œ์— ํ†ตํ•ฉํ•˜๋ ค๋Š” ๊ฒฝ์šฐ๋ฅผ ์œ„ํ•ด ํŠœํ† ๋ฆฌ์–ผ๋„ ์ œ๊ณต๋ฉ๋‹ˆ๋‹ค.

์‹œํ€€์Šค ๋ณ‘๋ ฌ์„ฑ(sequence parallelism)์„ ์‚ฌ์šฉํ•˜๋Š” ๋ฐฉ๋ฒ•

from accelerate.utils import ParallelismConfig, DeepSpeedSequenceParallelConfig

+# Example: 4 GPUs with sp_size=4, dp_shard_size=1
+# Ensure: dp_replicate_size ร— dp_shard_size ร— sp_size = 1 ร— 1 ร— 4 = 4 GPUs
parallelism_config = ParallelismConfig(
+     sp_backend="deepspeed",
+     sp_size=4,
+     dp_shard_size=1,  # Explicit: no data parallelism
+     sp_handler=DeepSpeedSequenceParallelConfig(
+         sp_seq_length_is_variable: true,
+         sp_attn_implementation="sdpa",
+     ),
+ )

accelerator = Accelerator(
    ...,
    parallelism_config=parallelism_config,
)

๐Ÿค—accelerate์˜ ๋‹ค๋ฅธ ๋ชจ๋“  ๊ธฐ๋Šฅ๊ณผ ๋งˆ์ฐฌ๊ฐ€์ง€๋กœ, accelerate launch์— ํ•ด๋‹น ํ”Œ๋ž˜๊ทธ๋ฅผ ์ „๋‹ฌํ•˜์—ฌ ์‹œํ€€์Šค ๋ณ‘๋ ฌ์„ฑ๋„ ํ™œ์„ฑํ™”ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๊ฒฝ์šฐ์—๋„ ์ฐจ์ด๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค:

accelerate launch --parallelism-config-sp-size 8  ...

sp_size ๋ฐ ๊ธฐํƒ€ ๊ตฌ์„ฑ์„ accelerate config ๋ช…๋ น์—์„œ ์„ค์ •ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋ ‡๊ฒŒ ํ•˜๋ฉด accelerate ๊ตฌ์„ฑ ํŒŒ์ผ์— ์ €์žฅ๋˜์–ด ์Šคํฌ๋ฆฝํŠธ๋ฅผ ์‹คํ–‰ํ•  ๋•Œ๋งˆ๋‹ค ๋งค๋ฒˆ ์ „๋‹ฌํ•  ํ•„์š”๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค.

sequence parallelism์€ data parallelism๊ณผ ๊ฒฐํ•ฉ๋ฉ๋‹ˆ๋‹ค. ์ถ”๊ฐ€ GPU๊ฐ€ ํ•„์š”ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ 8๊ฐœ์˜ GPU๊ฐ€ ์žˆ๋Š” ๊ฒฝ์šฐ --parallelism-config-dp-shard-size 8 --parallelism-config-sp-size 8์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋˜๋Š” ParallelismConfig ํด๋ž˜์Šค๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ํ”„๋กœ๊ทธ๋ž˜๋ฐ ๋ฐฉ์‹์œผ๋กœ ์„ค์ •ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

์ค‘์š”: dp_replicate_size ร— dp_shard_size ร— sp_size = num_processes๋ฅผ ๋ฐ˜๋“œ์‹œ ํ™•์ธํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด, GPU 8๊ฐœ์™€ sp_size=8๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ dp_shard_size=1์ด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค(1 ร— 1 ร— 8 = 8). GPU 4๊ฐœ์™€ sp_size=2๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ 2D ๋ณ‘๋ ฌํ™”๋ฅผ ์œ„ํ•ด dp_shard_size=2๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค(1 ร— 2 ร— 2 = 4).

ALST/Ulysses SP ๋ฐฑ์—”๋“œ ๊ตฌ์„ฑ

ALST/UlyssesSP์€ ์ด ๋…ผ๋ฌธ์—์„œ ์„ค๋ช…๋œ ๋Œ€๋กœ ์–ดํ…์…˜ ํ—ค๋“œ ๋ณ‘๋ ฌ์„ฑ์„ ์‚ฌ์šฉํ•˜์—ฌ ์‹œํ€€์Šค ๋ณ‘๋ ฌ์„ฑ์„ ๊ตฌํ˜„ํ•ฉ๋‹ˆ๋‹ค. ๋‹จ์ˆœํ™”๋ฅผ ์œ„ํ•ด ์‹œํ€€์Šค ๋ณ‘๋ ฌ์„ฑ์˜ ๊ฐœ๋…๊ณผ ์„ค์ •์„ ์žฌ์‚ฌ์šฉํ•˜๋ฉฐ, ์‚ฌ์šฉ์ž ๊ด€์ ์—์„œ๋Š” ๋™์ผํ•ฉ๋‹ˆ๋‹ค. ์—ฌ๋Ÿฌ GPU๊ฐ€ ๋‹จ์ผ ๋ฐฐ์น˜ ์ฒ˜๋ฆฌ์— ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค.

ALST๊ฐ€ ์–ด๋–ค ๊ฐ€๋Šฅ์„ฑ์„ ์—ด์–ด์ฃผ์—ˆ๋Š”์ง€ ๊ฐ์„ ์žก๊ธฐ ์œ„ํ•ด, ์ด ๊ธฐ์ˆ  ๋•๋ถ„์— ๋‹จ์ผ H100 GPU์—์„œ 500K ํ† ํฐ, ๋‹จ์ผ ๋…ธ๋“œ์—์„œ 3.7M ํ† ํฐ, ๊ทธ๋ฆฌ๊ณ  Llama-8B ๋ชจ๋ธ์˜ ๊ฒฝ์šฐ ๋…ธ๋“œ 4๊ฐœ๋งŒ์œผ๋กœ 15M ํ† ํฐ๊นŒ์ง€ bf16์œผ๋กœ ํ•™์Šตํ•  ์ˆ˜ ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค. HF Accelerate์˜ ์ด ๊ธฐ๋Šฅ์€ ALST์˜ 3๊ฐ€์ง€ ๊ตฌ์„ฑ ์š”์†Œ ์ค‘ 1๊ฐœ๋งŒ ํ™œ์„ฑํ™”ํ•˜๋ฏ€๋กœ, ๋‹ฌ์„ฑ ๊ฐ€๋Šฅํ•œ ์‹œํ€€์Šค ๊ธธ์ด๋Š” ๋” ์งง์•„์ง‘๋‹ˆ๋‹ค. ALST์˜ ์ „์ฒด ์„ฑ๋Šฅ์„ ๋ฐœํœ˜ํ•˜๋ ค๋ฉด TiledMLP, CPU๋กœ์˜ Activation checkpoint offload ๋ฐ ๊ธฐํƒ€ ๋ช‡ ๊ฐ€์ง€ ์„ค์ •์„ ํ™œ์„ฑํ™”ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ž์„ธํ•œ ๋‚ด์šฉ์€ ์ด ํŠœํ† ๋ฆฌ์–ผ์„ ์ฐธ๊ณ ํ•˜์„ธ์š”.

deepspeed ๋ฐฑ์—”๋“œ๋ฅผ ๊ตฌ์„ฑํ•˜๋ ค๋ฉด:

# Example: 4 GPUs with sp_size=4, dp_shard_size=1
# Ensure: dp_replicate_size ร— dp_shard_size ร— sp_size = 1 ร— 1 ร— 4 = 4 GPUs
parallelism_config = ParallelismConfig(
    sp_backend="deepspeed",
    sp_size=4,
    dp_shard_size=1,  # Explicit: no data parallelism
    sp_handler=DeepSpeedSequenceParallelConfig(
        sp_seq_length=256,
        sp_seq_length_is_variable=True,
        sp_attn_implementation="sdpa",
    ),
)
accelerator = Accelerator(
    ...,
    parallelism_config=parallelism_config,
)
  • sp_backend: ์—ฌ๊ธฐ์„œ๋Š” deepspeed๋กœ ์„ค์ •
  • sp_size๋Š” ์‹œํ€€์Šค ๋ณ‘๋ ฌ๋„์˜ ํฌ๊ธฐ๋ฅผ ๋‚˜ํƒ€๋‚ด๋ฉฐ, ์œ„์˜ ์˜ˆ์‹œ์—์„œ๋Š” 4์ด๋ฏ€๋กœ ๋‹จ์ผ ๋ฐฐ์น˜ ์ฒ˜๋ฆฌ์— 4๊ฐœ์˜ GPU๊ฐ€ ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค(๋™์ผํ•œ GPU์—์„œ DP=4๋ฅผ ์ˆ˜ํ–‰ํ•˜๋Š” ๋™์•ˆ).
  • sp_seq_length์™€ sp_seq_length_is_variable์€ ์‹œํ€€์Šค ๊ธธ์ด๋ฅผ ์ฒ˜๋ฆฌํ•˜๋Š” ๋ฐ ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค. sp_seq_length_is_variable=True์ธ ๊ฒฝ์šฐ ๋ฐฑ์—”๋“œ๋Š” ๋ฐฐ์น˜ ๊ฐ„์— ๋ณ€๊ฒฝ๋  ์ˆ˜ ์žˆ๋Š” ์‹œํ€€์Šค ๊ธธ์ด๋ฅผ ์ฒ˜๋ฆฌํ•˜๋ฉฐ, ์ด๋•Œ sp_seq_length ๊ฐ’์€ ์‹œํ€€์Šค ๋ณ‘๋ ฌ๋„(sequence parallel degree)๋กœ ๋‚˜๋ˆ„์–ด ๋–จ์–ด์ง€๋Š” ๊ฐ’์œผ๋กœ ์„ค์ •ํ•˜๊ฑฐ๋‚˜ ์•„์˜ˆ ์„ค์ •ํ•˜์ง€ ์•Š์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๊ฒฝ์šฐ ๋ชจ๋“  forward์—์„œ ์‹œํ€€์Šค ๋ณ€์ˆ˜๊ฐ€ ์ž…๋ ฅ์œผ๋กœ๋ถ€ํ„ฐ ์œ ๋„๋ฉ๋‹ˆ๋‹ค. False์ธ ๊ฒฝ์šฐ seq_length๋Š” ๋ฐฐ์น˜์˜ ์‹œํ€€์Šค ๊ธธ์ด ์ฐจ์›๊ณผ ์ผ์น˜ํ•ด์•ผ ํ•˜๋ฉฐ, ์ด๋Š” ํ•ญ์ƒ ๋™์ผํ•˜๋„๋ก ํŒจ๋”ฉ๋˜์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ True์ž…๋‹ˆ๋‹ค.
  • sp_attn_implementation์€ sdpa, flash_attention_2, flash_attention_3 ์ค‘ ํ•˜๋‚˜์ž…๋‹ˆ๋‹ค. ์ด ์‹œํ€€์Šค ๋ณ‘๋ ฌ ๊ตฌํ˜„์€ attention_mask ๋Œ€์‹  position_ids๋ฅผ ์‚ฌ์šฉํ•˜๋ฏ€๋กœ, eager๋Š” position_ids๋ฅผ ์ง€์›ํ•˜๋Š” ๊ธฐ๋Šฅ์ด ์ถ”๊ฐ€๋  ๋•Œ๊นŒ์ง€ ์—ฌ๊ธฐ์„œ ์ž‘๋™ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋˜ํ•œ, sdpa๋Š” ์—ฌ๋Ÿฌ ์ƒ˜ํ”Œ์ด ํ•˜๋‚˜๋กœ ๊ฒฐํ•ฉ๋œ ๊ฒฝ์šฐ๋ฅผ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ์ฒ˜๋ฆฌํ•˜์ง€ ๋ชปํ•˜๋ฉฐ, ์ „์ฒด ์ƒ˜ํ”Œ์„ ํ•˜๋‚˜๋กœ ์ทจ๊ธ‰ํ•˜์—ฌ ์–ดํ…์…˜์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค. ์ƒ˜ํ”Œ์ด ๊ฒฐํ•ฉ๋˜์ง€ ์•Š์€ ๊ฒฝ์šฐ sdpa๋Š” ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ํ•ญ์ƒ ์ž‘๋™ํ•˜๋Š” Flash Attention์ด ์ด์ƒ์ ์ธ ์„ ํƒ์ž…๋‹ˆ๋‹ค.
  • sp_allow_recurrent_layers: ์„ ํ˜• ์–ดํ…์…˜ ๋ ˆ์ด์–ด(์˜ˆ: Gated DeltaNet)๋ฅผ ํฌํ•จํ•˜๋Š” ๋ชจ๋ธ์€ ๊ธฐ๋ณธ์ ์œผ๋กœ ๊ฑฐ๋ถ€๋ฉ๋‹ˆ๋‹ค. ์ด๋Ÿฌํ•œ ๋ ˆ์ด์–ด๋Š” ์‹œํ€€์Šค ๋ฐฉํ–ฅ์œผ๋กœ ์žฌ๊ท€ ์ƒํƒœ๋ฅผ ์œ ์ง€ํ•˜๋ฉฐ, ๊ฐ ๋žญํฌ๊ฐ€ ์ด๋ฅผ 0์—์„œ ๋‹ค์‹œ ์‹œ์ž‘ํ•˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค. ๋ชจ๋ธ์ด ํ•ด๋‹น ์ƒํƒœ๋ฅผ ์‹œํ€€์Šค ๋ณ‘๋ ฌ ๋žญํฌ ๊ฐ„์— ์ž์ฒด์ ์œผ๋กœ ๊ตํ™˜ํ•˜๋Š” ๊ฒฝ์šฐ์—๋งŒ True๋กœ ์„ค์ •ํ•˜์‹ญ์‹œ์˜ค. ๊ธฐ๋ณธ๊ฐ’์€ False์ž…๋‹ˆ๋‹ค.
  • sp_disable_in_eval: True์ธ ๊ฒฝ์šฐ, eval ๋ชจ๋“œ์—์„œ sequence parallelism์ด ๋น„ํ™œ์„ฑํ™”๋˜๊ณ  ๋ชจ๋“  rank๊ฐ€ ์ „์ฒด sequence๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค. ๊ธด ํ‰๊ฐ€ sequence์˜ ๊ฒฝ์šฐ ์ด๋ฅผ ์œ ์ง€ํ•˜๋ ค๋ฉด False๋กœ ์„ค์ •ํ•ด์•ผ ํ•˜๋ฉฐ, ์ด๋•Œ eval dataloader๋Š” train dataloader์™€ ๋™์ผํ•˜๊ฒŒ sharding๋˜์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ True์ž…๋‹ˆ๋‹ค.

DeepSpeedSequenceParallelConfig ๊ฐ์ฒด์— ์ด๋Ÿฌํ•œ ๊ฐ’์„ ์„ค์ •ํ•˜๋Š” ๋Œ€์‹ , ๋™์ผํ•œ ๊ฒฐ๊ณผ๋ฅผ ์–ป๊ธฐ ์œ„ํ•ด ํ™˜๊ฒฝ ๋ณ€์ˆ˜๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์•„๋ž˜ ๋ชฉ๋ก์˜ ๋๊ณผ ๋Œ€์‘ํ•˜๋Š” ํ™˜๊ฒฝ ๋ณ€์ˆ˜๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

  • PARALLELISM_CONFIG_SP_BACKEND
  • PARALLELISM_CONFIG_SP_SEQ_LENGTH
  • PARALLELISM_CONFIG_SP_SEQ_LENGTH_IS_VARIABLE
  • PARALLELISM_CONFIG_SP_ATTN_IMPLEMENTATION
  • PARALLELISM_CONFIG_SP_ALLOW_RECURRENT_LAYERS
  • PARALLELISM_CONFIG_SP_DISABLE_IN_EVAL

์ฝ”๋“œ์—์„œ ์ „๋‹ฌํ•˜์ง€ ์•Š์€ ๊ฒฝ์šฐ, sp_size๋Š” --parallelism_config_sp_size CLI ์ธ์ž๋ฅผ ํ†ตํ•ด ์„ค์ •ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค๋ฅธ ์ธ์ž๋“ค๋„ ๋งˆ์ฐฌ๊ฐ€์ง€์ž…๋‹ˆ๋‹ค. ๋˜ํ•œ, accelerate config ํŒŒ์ผ ์Šคํƒ€์ผ์˜ ์„ค์ •๋„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด, GPU 2๊ฐœ์˜ ๊ฒฝ์šฐ:

distributed_type: DEEPSPEED
deepspeed_config:
  deepspeed_config_file: path/to/ds_config.json
machine_rank: 0
num_machines: 1
num_processes: 2
parallelism_config:
  parallelism_config_dp_replicate_size: 1
  parallelism_config_dp_shard_size: 1  # Must satisfy: 1 ร— 1 ร— 2 = 2 num_processes
  parallelism_config_sp_size: 2
  parallelism_config_sp_backend: deepspeed
  parallelism_config_sp_seq_length_is_variable: true
  parallelism_config_sp_attn_implementation: sdpa

์•ž์„œ ์–ธ๊ธ‰ํ–ˆ๋“ฏ์ด Ulysses sequence parallelism์€ ์ผ๋ฐ˜์ ์œผ๋กœ data parallelism๊ณผ ํ•จ๊ป˜ ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค. ๋™์ผํ•œ rank๊ฐ€ ๊ณ ์œ ํ•œ ๋ฐ์ดํ„ฐ ์ŠคํŠธ๋ฆผ์„ ๊ณต๊ธ‰ํ•˜๋Š” ๋ฐ ์‚ฌ์šฉ๋˜๊ณ , ๋™์‹œ์— Ulysses Sequence Parallelism์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ๋ฆฌํ”Œ๋ฆฌ์นด๋ฅผ ์ƒ์„ฑํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค:

# Example: 4 GPUs with 2D parallelism (SP=2, DP=2)
# Ensure: dp_replicate_size ร— dp_shard_size ร— sp_size = 2 ร— 1 ร— 2 = 4 GPUs
parallelism_config = ParallelismConfig(
    dp_replicate_size=2,
    dp_shard_size=1,  # Explicit: no sharding within replicas
    sp_size=2,
    sp_backend="deepspeed",
    sp_handler=DeepSpeedSequenceParallelConfig(...),
)

์—ฌ๊ธฐ์„œ๋Š” 4๊ฐœ์˜ GPU๋ฅผ 2๊ฐœ์˜ sequence parallelism ๋ ˆํ”Œ๋ฆฌ์นด๋กœ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๋ฐ์ดํ„ฐ ๋ณ‘๋ ฌ์„ฑ์€ Deepspeed-ZeRO๊ฐ€ ๋‹ด๋‹นํ•ฉ๋‹ˆ๋‹ค.

UlyssesSPDataLoaderAdapter ๋‚ด๋ถ€์—๋Š” ๋งŽ์€ ๋งˆ๋ฒ•์ด ์ˆจ๊ฒจ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ํด๋ž˜์Šค๋Š” ๋ฐฑ๊ทธ๋ผ์šด๋“œ์—์„œ ๋™์ž‘ํ•˜๋ฉฐ ์›๋ž˜์˜ DataLoader ๊ฐ์ฒด๋ฅผ ๋ž˜ํ•‘ํ•˜์ง€๋งŒ, ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•  ๊ฒฝ์šฐ ์ด๋ฅผ ์ธ์ง€ํ•˜๊ณ  ์žˆ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๋˜ํ•œ, ๋ฐฐ์น˜์— ์ฐธ์—ฌํ•˜๋Š” ๊ฐ rank๋กœ ์ƒค๋”ฉ์ด ์ˆ˜ํ–‰๋˜๊ธฐ ์ „์— ๋ฐฐ์น˜ ์‚ฌ์ „์— ์˜ฌ๋ฐ”๋ฅธ shift_labels๋ฅผ ์ž๋™์œผ๋กœ ์ฃผ์ž…ํ•ฉ๋‹ˆ๋‹ค.

์ด์ œ ALST/UlyssesSP๋ฅผ ์‚ฌ์šฉํ•˜๊ธฐ ์œ„ํ•ด ๋‚จ์€ ๋งˆ์ง€๋ง‰ ๋‹จ๊ณ„๋Š”, ๊ทธ๋ž˜๋””์–ธํŠธ๋ฅผ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ณ„์‚ฐํ•˜๊ธฐ ์œ„ํ•ด ๋‹ค๋ฅธ ๊ฐ€๋ณ€ all_gather๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋žญํฌ ๊ฐ„ ์†์‹ค์„ ์ง‘๊ณ„ํ•˜๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค. ๋‹ค์Œ ์ฝ”๋“œ๋Š” ์ด๋ฅผ ์ˆ˜ํ–‰ํ•˜๋ฉฐ, -100์œผ๋กœ ๋งˆ์Šคํ‚น๋œ ํ† ํฐ์„ ์ œ์™ธํ•˜์—ฌ ์˜ฌ๋ฐ”๋ฅธ ํ‰๊ท ์„ ์–ป์Šต๋‹ˆ๋‹ค:

sp_size = parallelism_config.sp_size if parallelism_config is not None else 1
if sp_size > 1:
    sp_group = accelerator.torch_device_mesh["sp"].get_group()
    sp_world_size = parallelism_config.sp_size

# Normal training loop
for iter, batch in enumerate(dl):
    optimizer.zero_grad()

    batch = move_to_device(batch, model.device)

    # The model automatically receives shift_labels via **kwargs and uses it for loss computation.
    # Both standard transformers models and Liger-patched models handle this correctly.
    outputs = model(**batch)
    loss = outputs.loss
    shift_labels = batch["shift_labels"]

    if sp_size > 1:
        # differentiable weighted per-shard-loss aggregation across ranks
        losses_per_rank = torch.distributed.nn.functional.all_gather(loss, group=sp_group)
        # special dealing with SFT that has prompt tokens that aren't used in loss computation
        good_tokens = (shift_labels != -100).view(-1).sum()
        good_tokens_per_rank = torch.distributed.nn.functional.all_gather(
            good_tokens, group=sp_group
        )
        # Skip ranks with zero valid tokens to avoid NaN contamination (NaN * 0 = NaN)
        total_loss = sum(
            losses_per_rank[rank] * good_tokens_per_rank[rank]
            for rank in range(sp_world_size)
            if good_tokens_per_rank[rank] > 0
        )
        total_good_tokens = sum(good_tokens_per_rank)
        loss = total_loss / max(total_good_tokens, 1)

    if rank == 0: accelerator.print(f"{iter}: {loss=}")
    accelerator.log(dict(train_loss=loss, step=iter))

    accelerator.backward(loss)
    optimizer.step()

shift_labels๊ฐ€ ๋ฐฐ์น˜์— ์กด์žฌํ•˜๋Š” ๊ฒฝ์šฐ ๋ชจ๋ธ์ด ์ด๋ฅผ ์ž๋™์œผ๋กœ ์ฒ˜๋ฆฌํ•œ๋‹ค๋Š” ์ ์— ์œ ์˜ํ•˜์„ธ์š”. ๋ชจ๋ธ์˜ forward pass๋Š” **kwargs๋ฅผ ํ†ตํ•ด shift_labels๋ฅผ ์ˆ˜์‹ ํ•˜์—ฌ ์†์‹ค ํ•จ์ˆ˜๋กœ ์ „๋‹ฌํ•˜๊ณ , ์†์‹ค ํ•จ์ˆ˜๋Š” ์‹œํ€€์Šค ๋ณ‘๋ ฌ์„ฑ์— ๋Œ€ํ•ด ์†์‹ค์„ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค. Liger Kernel์„ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ์—๋„ shift_labels๋ฅผ ์›ํ™œํ•˜๊ฒŒ ์ฒ˜๋ฆฌํ•˜๋ฉฐ, ๋งค์šฐ ๋ฉ”๋ชจ๋ฆฌ ํšจ์œจ์ ์ธ ๋ฐฉ์‹์œผ๋กœ ์†์‹ค์„ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค. Liger๋Š” ์œตํ•ฉ ์—ฐ์‚ฐ(์˜ˆ: ๋ฉ”๋ชจ๋ฆฌ์— ์ „์ฒด logits ํ…์„œ๋ฅผ ์ ˆ๋Œ€ ์ƒ์„ฑํ•˜์ง€ ์•Š๋Š” ์œตํ•ฉ logit-loss ๊ณ„์‚ฐ)์„ ์‚ฌ์šฉํ•˜์—ฌ GPU ๋ฉ”๋ชจ๋ฆฌ๋ฅผ ํ™•๋ณดํ•˜๋ฏ€๋กœ ๊ธด ์‹œํ€€์Šค ๊ธธ์ด์— ๋Œ€ํ•ด ๊ฐ•๋ ฅํžˆ ๊ถŒ์žฅ๋ฉ๋‹ˆ๋‹ค.

HF Accelerate๊ฐ€ ๋ฐฑ๊ทธ๋ผ์šด๋“œ์—์„œ ์ˆ˜ํ–‰ํ•œ ์ž‘์—…์„ ํ™•์ธํ•˜๋ ค๋ฉด ์ด ์ „์ฒด ํ†ตํ•ฉ ํŠœํ† ๋ฆฌ์–ผ์„ ์ฝ์–ด๋ณด์„ธ์š”.

ALST/UlyssesSP๊ฐ€ ํ™œ์„ฑํ™”๋œ Accelerate ํ•™์Šต ๋ฃจํ”„์˜ ์˜ˆ์ œ๋Š” examples/alst_ulysses_sequence_parallelism๋ฅผ ์ฐธ๊ณ ํ•˜์„ธ์š”.

[!๊ฒฝ๊ณ ]

์ด API๋Š” ๋น„๊ต์  ์‹ ๊ทœ์ด๋ฉฐ ์•„์ง ์‹คํ—˜ ๋‹จ๊ณ„์— ์žˆ์Šต๋‹ˆ๋‹ค. ์•ˆ์ •์ ์ธ API๋ฅผ ์ œ๊ณตํ•˜๊ธฐ ์œ„ํ•ด ๋…ธ๋ ฅํ•˜๊ณ  ์žˆ์ง€๋งŒ, ๊ณต๊ฐœ API์˜ ์ผ๋ถ€ ์„ธ๋ถ€ ์‚ฌํ•ญ์€ ํ–ฅํ›„ ๋ณ€๊ฒฝ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์ด๊ฒƒ์€ Deepspeed ๋ฐฑ์—”๋“œ์ด๋ฏ€๋กœ ์ผ๋ฐ˜์ ์ธ Deepspeed ๊ตฌ์„ฑ์ด ์ ์šฉ๋ฉ๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ์‹œํ€€์Šค ๋ณ‘๋ ฌ์„ฑ์„ ์˜ตํ‹ฐ๋งˆ์ด์ € ์ƒํƒœ ๋ฐ/๋˜๋Š” ๊ฐ€์ค‘์น˜ ์˜คํ”„๋กœ๋”ฉ๊ณผ ๊ฒฐํ•ฉํ•˜์—ฌ ๋” ๋งŽ์€ GPU ๋ฉ”๋ชจ๋ฆฌ๋ฅผ ํ™•๋ณดํ•˜๊ณ  ๋” ๊ธด ์‹œํ€€์Šค ๊ธธ์ด๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๊ธฐ์ˆ ์€ DeepSpeed ZeRO stage 2 ๋ฐ 3๊ณผ ํ•จ๊ป˜ ์ž‘๋™ํ•˜๋„๋ก ํ…Œ์ŠคํŠธ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.

Update on GitHub