+ $LjB^RIt^RIt^RIHt^RIHtHtHtHt^RI H t H t H t H t ^RIHt]'d^RIHt]!RR44tR#) N) dataclass) TYPE_CHECKINGLiteralOptionalUnion)DeepSpeedSequenceParallelConfigDistributedTypeTorchContextParallelConfigTorchTensorParallelConfig)is_torch_version) Acceleratorca]tRt^!toRtRtRtRtRtRt Rt Rt Rt Rt RtRtRtRt]R4t]R4t]R4t]R4t]R 4t]R 4t]R 4t]R 4t]R 4t]R4t]R4t]R4t]R4t]R4t V3RlRlt!R V3RlRllt"V3RlRlt#Rt$V3RlRlt%V3RlRlt&V3Rlt'Rt(Vt)R#)!ParallelismConfigaM A dataclass to configure parallelisms applied to the model. Inspired by torchtitan's `ParallelDims` https://github.com/pytorch/torchtitan/blob/main/torchtitan/distributed/parallel_dims.py Args: dp_replicate_size (`int`, defaults to `1`): The size of the data parallel group. If `dp_replicate_size` is set to 1, the data parallel replication group will not be used. dp_shard_size (`int`, defaults to `1`): The size of the model shard group. If `dp_replicate_size > 1` and `tp_size > 1`, `dp_shard_size` must also be greater than 1, as composing DDP + TP is currently not supported. tp_size (`int`, defaults to `1`): The size of the tensor parallel group. If `tp_size` is set to `1`, the tensor parallel group will not be used. tp_handler (`~utils.TorchTensorParallelConfig`, defaults to `None`): The handler for the tensor parallel group. cp_size (`int`, defaults to `1`): The size of the context parallel group. Currently not supported, but reserved for future use and enabled for downstream libraries. cp_backend (`str`, defaults to `torch`): Which CP backend to use: `torch` (FSDP2) sp_size (`int`, defaults to `1`): The size of the sequence parallel group. sp_backend (`str`, defaults to `deepspeed`): Which SP backend to use:`deepspeed` (ALST/Ulysses) You may obtain different distributed data parallel paradigms by configuring `dp_replicate_size` and `dp_shard_size` together: - `dp_replicate_size == 1` and `dp_shard_size > 1`, we obtain Fully Sharded Data Parallel (FSDP). - `dp_replicate_size > 1` and `dp_shard_size > 1`, we obtain Hybrid Sharded Data Parallel (HSDP). - `dp_replicate_size > 1` and `dp_shard_size == 1` is an invalid configuration, to use pure DP, use `DistributedDataParallelKwargs` instead. Nc RVP RVP RVP RVP RVP RVP RVP RVP RVP R VP R 2#) z'ParallelismConfig( dp_replicate_size=z, dp_shard_size=z , tp_size=z , cp_size=z, cp_backend=z , sp_size=z, sp_backend=z, total_size=z tp_handler=z, cp_handler=z) ) dp_replicate_size dp_shard_sizetp_sizecp_size cp_backendsp_size sp_backend total_size tp_handler cp_handlerselfs&I/app/.local/lib/python3.14/site-packages/accelerate/parallelism_config.py__repr__ParallelismConfig.__repr__Us ##'#9#9":;#1123'' OO,-' OO,- OO,- OO,- OO,C 1 c ^RIpR.pTPVPP4UUu/uF=wr4W29gK T\ VR4'dVPVP4MTbK? upp4R#uuppi)rN device_mesh__dict__)copydeepcopyr#itemshasattr)rr$_non_serializable_fieldskvs& rto_jsonParallelismConfig.to_jsondss$1?  !MM//1 1DA4N:0F0F4==,AM1   s A= 4A= c z.pVP'd VR., pVP'd VR., pV#)zENames of enabled dimensions across which data parallelism is applied. dp_replicatedp_shard)dp_replicate_enableddp_shard_enabledrdimss& r dp_dim_namesParallelismConfig.dp_dim_namesqs?  $ $ $ ^$ $D  ZL D r c .pVP'd VR., pVP'd VR., pVP'd VR., pV#)z]Names of enabled dimensions which will receive the same batch (non-data parallel dimensions).tpcpsp) tp_enabled cp_enabled sp_enabledr2s& rnon_dp_dim_names"ParallelismConfig.non_dp_dim_names{sJ ??? TFND ??? TFND ??? TFND r c z.pVP'd VR., pVP'd VR., pV#)zlNames of enabled dimensions which will be flattened into a joint mesh across which is model sharded in FSDP.r/r8)r1r;r2s& rdp_shard_cp_dim_names'ParallelismConfig.dp_shard_cp_dim_namess:  ZL D ??? TFND r c .pVP'd VR., pVP'd VR., pVP'd VR., pV#)z@Names of enabled dimensions across which loss should be averagedr.r/r8)r0r1r;r2s& rdp_cp_dim_names!ParallelismConfig.dp_cp_dim_namessS  $ $ $ ^$ $D  ZL D ??? TFND r c V.pVP'd VR., pVR., pV#)z^Names of enabled dimensions across which FSDP is applied, including data parallel replication.r. dp_shard_cp)r0r2s& rfsdp_dim_names ParallelismConfig.fsdp_dim_namess3  $ $ $ ^$ $D  r c VPVP,VP,VP,VP,#)zSThe total size of the parallelism configuration, which is the product of all sizes.)rrrrrrs&rrParallelismConfig.total_sizes9%%(:(::T\\IDLLX[_[g[gggr c ^VPVP,VP,#)zhThe size of the non-data parallel dimensions, which is the product of tensor and context parallel sizes.)rrrrs&rnon_data_parallel_size(ParallelismConfig.non_data_parallel_sizes ||dll*T\\99r c <VPVP,#)z_The size of the data parallel dimensions, which is the product of data parallel replication and)rrrs&rdata_parallel_size$ParallelismConfig.data_parallel_sizes%%(:(:::r c VP^8#)zKTrue if data parallel replication is enabled, i.e. `dp_replicate_size > 1`.)rrs&rr0&ParallelismConfig.dp_replicate_enableds%%))r c VP^8#)zDTrue if data parallel sharding is enabled, i.e. `dp_shard_size > 1`.)rrs&rr1"ParallelismConfig.dp_shard_enableds!!A%%r c VP^8#)z:True if tensor parallelism is enabled, i.e. `tp_size > 1`.)rrs&rr:ParallelismConfig.tp_enabled||ar c VP^8#)z;True if context parallelism is enabled, i.e. `cp_size > 1`.)rrs&rr;ParallelismConfig.cp_enabledrWr c VP^8#)z;True if context parallelism is enabled, i.e. `sp_size > 1`.)rrs&rr<ParallelismConfig.sp_enabledrWr c <VPVP,#)z$Names of all active mesh dimensions.)r4r=rs&ractive_mesh_dims"ParallelismConfig.active_mesh_dimss  4#8#888r c <V^8dQhRS[/# device_type)str)format __classdict__s"r __annotate__ParallelismConfig.__annotate__s!!S!r c *VPR8XdVP^8dR#\RR4'd^RIHpM \ R4hVP 4p\V4^8XdR#VwrEV!VVVR7pVP'd"W`P,PR4VP'd"W`P,PR 4VP'd"W`P,PR 4V#) aBuilds a device mesh for the given device type based on the parallelism configuration. This method will also create required joint meshes (e.g. `dp_shard_cp`, `dp_cp`, `dp`). Args: device_type (`str`): The type of device for which to build the mesh, e deepspeedNz>=z2.2.0)init_device_meshz4Building a device_mesh requires to have torch>=2.2.0)mesh_dim_namesdprFdp_cp) rrr torch.distributed.device_meshrj RuntimeError _get_meshlenr4_flattenr@rC)rrbrjmeshrk mesh_shaper"s&& rbuild_device_mesh#ParallelismConfig.build_device_meshs ??k )dllQ.> D' * * FUV V~~ t9>%)"&  )     )) * 3 3D 9  % % % 22 3 < <] K     ,, - 6 6w ?r c0<V^8dQhRS[S[,/#r`)rrc)rdres"rrfrgs   8C=  r c"VPf2Ve#VPV4VnVP#\R4hVeBVPPV8wd'\RVPP RV R24hVP#)Nz@You need to pass a device_type e.g cuda to build the device meshz4The device_mesh is already created with device type z@. However, you are trying to get a device mesh with device_type z<. Please check if you correctly initialized your device_mesh)r"ru ValueErrorrb)rrbs&&rget_device_mesh!ParallelismConfig.get_device_meshs    #&#'#9#9+#F !!cdd&##//;>$NtO_O_OkOkNlmmnymzzvwr c^<V^8dQhRS[S[S[R3,S[S[R3,3,/#)rareturn.)tupleintrc)rdres"rrfrgs. ) )5sCx%S/!AB )r c aVPUu/uFqVPV,bK pp.ROo\VP4V3RlR7p\ \ V!4#uupi)zQGenerate mesh shape and dimension names for torch.distributed.init_device_mesh().c4<SPV^,4#)r)index)x mesh_orders&r-ParallelismConfig._get_mesh..s:++AaD1r )key)r.r/r8r9r7)r]_sizessortedr&r~zip)r parallelism mesh_dims sorted_itemsrs& @rrpParallelismConfig._get_meshsfOSNcNcdNc{$++k"::Nc dD  OO 2 S,'((esA#c D VPf/\\PP RR44VnVP f/\\PP RR44VnVP f/\\PP RR44VnVPf/\\PP RR44VnVPf&\PP RR4VnVPf/\\PP RR44Vn VPf&\PP R R 4Vn VP ^8dVPf\4Vn VP^8dVPf\4Vn M}\\R 7p\!VPWP,4'gB\#R VP R WP, R\%VP4 24hVP^8dVP&f\)4VnVP^8d\#RVP 24hVP ^8d\#RVP 24hVP ^8d\#RVP 24hVP^8d\#RVP 24hR.pVPV9d\#RV RVP 24hVP^8d\#RVP 24hR .pVPV9d\#RV RVP 24hVP^8d8VP^8d'\#RVP RVP R24hVP ^8gVP^8d.VP^8dVP ^8Xd \#R4hRVPRVP RVP RVPRVP/VnR#) N$PARALLELISM_CONFIG_DP_REPLICATE_SIZE1 PARALLELISM_CONFIG_DP_SHARD_SIZEPARALLELISM_CONFIG_TP_SIZEPARALLELISM_CONFIG_CP_SIZEPARALLELISM_CONFIG_CP_BACKENDtorchPARALLELISM_CONFIG_SP_SIZEPARALLELISM_CONFIG_SP_BACKENDri)rzParallelismConfig's cp_backend=z requires z, but cp_handler was set to z.dp_replicate_size must be at least 1, but got z*dp_shard_size must be at least 1, but got z$tp_size must be at least 1, but got z$cp_size must be at least 1, but got zcp_backend must be one of z , but got z$sp_size must be at least 1, but got zsp_backend must be one of z[Context Parallelism (CP) and Sequence Parallelism (SP) are mutually exclusive. Got cp_size=z and sp_size=z+. Please set either cp_size=1 or sp_size=1.aCTensor/Context parallelism (tp/cp_size > 1) cannot be used with pure data parallelism (dp_replicate_size > 1 and dp_shard_size == 1). Please set dp_shard_size > 1 and dp_replicate_size == 1 to compose FSDP + TP/CP for 2D parallel, or set dp_replicate_size == 1 and dp_shard_size > 1 to compose HSDP + TP/CP for 3D parallel.r.r/r7r8r9)rrosenvirongetrrrrrrrr rr dict isinstancerytype sp_handlerrr)rcp_backends_config_mapvalid_cp_backendsvalid_sp_backendss& r __post_init__ParallelismConfig.__post_init__s  ! ! )%(8^`c)d%eD "    %!$RZZ^^4VX[%\!]D  << rzz~~.JCPQDL << rzz~~.JCPQDL ?? " jjnn-LgVDO << rzz~~.JCPQDL ?? " jjnn-LkZDO <)-4*&"$//3I//3Z[[$9$//9J*Ukl{l{U|T}~Z[_`d`o`o[pZqr <RS S <RS S$I ??"3 39:K9LJW[WfWfVghi i <RS S(M ??"3 39:K9LJW[WfWfVghi i <;+D+D E ??k )dllQ.>  __ 9 9 900AB""-";";!<=9:  ??Q   '''++/H/HHbcnccb@@AB "&!2!2!4 KqyWT](+CTJV ( 5J;-XHI"5 444 MMADIIiDXX 59r c:<V^8dQh/S[S[,;R&S[S[,;R&S[S[,;R&S[S[,;R&S[R,;R&S[S[,;R&S[R,;R &S[R S[3,;R &S[R S[3,;R &S[R S[3,;R &#)rarrrrrrrrirNrrr)rrrrr r r)rdres"rrfrg!sJ }+KLC='MNc]!OPc]!QR 'STc]!UV $+W\d556=]^d667>_`d;;<Car ) rrrrr"rrrrrrr)N)*__name__ __module__ __qualname____firstlineno____doc__rrrrrrrrrrr"rr+propertyr4r=r@rCrGrrLrOr0r1r:r;r<r]rurzrprrr__annotate_func____static_attributes____classdictcell__)res@rrr!s!F(,#'M!G!G#'J!G'+J:>J:>J?CJK        hh::;;**&&      99!!F    ) )J X33 ++E r r)rr dataclassesrtypingrrrraccelerate.utils.dataclassesrr r r accelerate.utils.versionsr accelerater rrr rrsI !:: 7& ll lr