+ $LjT@^RIt^RIt^RIt^RIt^RIHt^RIHt^RIHtH t ^RI H t ^RI H t ^RIt^RIt^RIHt^RIHt^R IHt^R IHt^R IHt^R IHt^R IHt^RI H!t!H"t"H#t#H$t$H%t%^RI&H't'^RI(H)t)^RI*H+t+]!],4t-]$!4'd ^RI.H/uH0t1RRlt2RRlt3RRlt4RRlt5RRlt6RRlt7RRlt8R R!lt9R8R"R#llt:R$t;R%R&lt%is_repeated_blocks..Xs%CFq 1fQi1122Fs*-FT)r)rr ModuleListlenallr,sfr#is_repeated_blocksr@Osq 6588../ D D K!O D D CCFCCC  CFC Cr%cXV^8dQhR\PPR\/#rr)r"s"r#r$r$\s"   D r%cVP'd,VP4Fp\V4'gKR# R#)z Check whether the module has repeated blocks, i.e. `torch.nn.ModuleList` with all children of the same class, at any level of the module hierarchy. This is useful to determine whether we should apply regional compilation to the module. TF)r0r1r@r2s& r#has_repeated_blocksrC\s1 )I!),,* r%cV^8dQhR\PPR\PP/#rrrr )r"s"r#r$r$js.EEEHHOOE%((//Er%c jaRV3RlloS!V3/VBpRVP9dWPR&V#)a Performs regional compilation where we target repeated blocks of the same class and compile them sequentially to hit the compiler's cache. For example, in `GPT2LMHeadModel`, the repeated block/class is `GPT2Block`, and can be accessed as `model.transformer.h[0]`. The rest of the model (e.g. model.lm_head) is compiled separately. This allows us to speed up the compilation overhead / cold start of models like LLMs and Transformers in general. See https://pytorch.org/tutorials/recipes/regional_compilation.html for more details. Args: module (`torch.nn.Module`): The model to compile. **compile_kwargs: Additional keyword arguments to pass to `torch.compile()`. Returns: `torch.nn.Module`: A new instance of the model with some compiled regions. Example: ```python >>> from accelerate.utils import compile_regions >>> from transformers import AutoModelForCausalLM >>> model = AutoModelForCausalLM.from_pretrained("gpt2") >>> compiled_model = compile_regions(model, mode="reduce-overhead") >>> compiled_model.transformer.h[0] OptimizedModule( (_orig_mod): GPT2Block( (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D(nf=2304, nx=768) (c_proj): Conv1D(nf=768, nx=768) (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D(nf=3072, nx=768) (c_proj): Conv1D(nf=768, nx=3072) (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) ) ``` cV^8dQhR\PPR\PP/#rrE)r"s"r#r$%compile_regions..__annotate__s*uxxr%c "<\V4'dP\PP4pVF)pVP \P !V3/VB4K+ V#\ V4'dVPPVP4pVPPVP4/Vn VP4FwrCVPVS!V3/VB4K V#\P !V3/VBpV#N)r@rrr=appendcompilerCr8__new____dict__updater0named_children add_module)rcompile_kwargs new_moduler3name_compile_regionss&, r#rU)compile_regions.._compile_regionss f % %,,.J# !!%-- "L^"LM$! ( ())11&2B2BCJ    & &v 7"$J #)#8#8#:%%d,~ >' ((r%cV^8dQhR\PPR\PP/#rrE)r"s"r#r$r$s*%((//r%c \V4'dVFpVP!R/VBK V#\V4'd&VP4Fp\ V3/VBK V#VP!R/VBV#)a Like `compile_regions`, but uses the in-place `module.compile()` instead of `torch.compile(module)`. Needed for the FSDP2 prepare path: `torch.compile(module)` returns an `OptimizedModule` whose `__call__` bypasses `nn.Module._call_impl`, so forward/pre hooks added later by `fully_shard` never fire and per-layer all-gather/reshard is lost. The in-place `module.compile()` keeps `_call_impl` (and its runtime hook check) on the call path, so FSDP hooks installed afterwards still run. Args: module (`torch.nn.Module`): The model to compile. **compile_kwargs: Additional keyword arguments to pass to `module.compile()`. r[)r@rLrCr\compile_regions_fsdp2r^s&, r#rbrbsy&!!I    / /  M V $ $__&E !% :> :' M (( Mr%cXV^8dQhR\PPR\/#)rmodelrr)r"s"r#r$r$s&CCUXX__CCr%ca\RR4'd^RIHoM^RIHo\;QJd-V3RlVP 44F 'gK R# R#!V3RlVP 444#)z Check if the model has DTensor parameters. Args: model (`torch.nn.Module`): The model to check. Returns: `bool`: Whether the model has DTensor parameters. >=z2.5.0)DTensorc3<<"TFp\VS4xK R#5irJ)r))r9prgs& r#r;$model_has_dtensor..sB/A!z!W%%/AsTF)rtorch.distributed.tensorrgtorch.distributed._tensorany parameters)rdrgs&@r#model_has_dtensorrosUg&&4 6 3Bu/?/?/AB33B3B3Bu/?/?/AB BBr%c<V^8dQhR\R\R\/#)rkeep_fp32_wrapperkeep_torch_compile recursiver!)r"s"r#r$r$s)UU"U?CUW[Ur%ca \PPP\PP3p\ V4p\ V4pRpV'dTpVPpM.V'd'RVP9dTpVPR,p\4'd^RI H pWH3, p\R\4'd \4'd^RIHp WI3, p\#W4'dVP$pKV'dV 3Rlo S !V4pV'gVP&p VPP)RR4p V e8\+V R4'dV P,p W8XgK&\/W4Vn\1VR R 4'd\3VR R 7V'd/Ve+V'd WnTpV#V'dWPR&TpV#) a_ Extract a model from its distributed containers. Args: model (`torch.nn.Module`): The model to extract. keep_fp32_wrapper (`bool`, *optional*): Whether to remove mixed precision hooks from the model. keep_torch_compile (`bool`, *optional*): Whether to unwrap compiled model. recursive (`bool`, *optional*, defaults to `False`): Whether to recursively extract all cases of `module.module` from `model` as well as unwrap child sublayers recursively, not just the top-level distributed containers. Returns: `torch.nn.Module`: The extracted model. NrW)DeepSpeedEnginerf)FullyShardedDataParallelc<\VR4'dS!VP4pMTpVP4Fwr#\WS!V44K V#r,)r(rrPsetattr)runwrapped_modulerTr__recursive_unwraps& r#r{6extract_model_from_parallel.._recursive_unwrap*sTvx((#4V]]#C #) />>@ (0A%0HI A# #r%_original_forward __wrapped__ _converted_to_transformer_engineF)to_transformer_engine)rrparallelDistributedDataParallel DataParallelr-r4rWrNr deepspeedrvrrr2torch.distributed.fsdp.fully_sharded_data_parallelrwr)rforwardpopr(r~rgetattrr) rdrqrrrsoptions is_compiled has_compiledcompiled_modelrvFSDProriginal_forwardr{s &&&& @r#extract_model_from_parallelrs(xx  88%((:O:OPG$U+K'.LN  %.. ("NNN;/E-%%2338V8X8Xg7 U $ $  $"%( -- >>--.A4H  ''=11!--.&w6EM 5 Make sure all processes will reach this instruction otherwise one of your processes will hang forever. N)r wait_for_everyoner[r%r#rrPsN$$&r%c$V^8dQhR\/#)r state_dict)dict)r"s"r#r$r$]s   r%c ,\P!\4pVP4F>wr#\ V\ 4'dKV\ V4,PV4K@ VP4UUu/uFwrE\V4^8gKWEbK ppp\4pVP4FFpVUu.uF q"V9gK VNK ppVPVR,4VR,FpWK KH \V4^8d\PRV R24VP4U U u/uF8wrT \ V \P4'dV P!4MT bK: pp p V#uuppiuupiuup p i)z Cleans the state dictionary from a model and removes tensor aliasing if present. Args: state_dict (`dict`): The state dictionary from a model :NNzRemoved shared tensor zk while saving. This should be OK, but check by verifying that you don't receive any warning while reloading) collections defaultdictlistitemsr)strrrKr>setvaluesrOloggerwarningrTensor contiguous) rptrsrTtensorptrnames shared_ptrs warn_names found_nameskvs & r# clean_state_dict_for_safetensorsr]sa  " "4 (D"((* &#&& "6* + 2 24 8+ 15 O *#E Q:3: KOJ##% ).D1Ctt D+b/*OOD $& :$ZL1\ ] WaVfVfVhiVhda!z!U\\'B'BQ\\^IVhJi #PEjsFFF F >Fc0V^8dQhR\R\/#)rsave_on_each_nodesafe_serializationrt)r"s"r#r$r$sDdr%c\4P\P8Xd\P !V4pV'd6\ \RR/R7p\V\4'd \V4pM\Pp\4P'dV'g V!W4R#\4P'dV'd V!W4R#R#R#)a Save the data to disk. Use in place of `torch.save()`. Args: obj: The data to save f: The file (or file-like object) to use to save the data save_on_each_node (`bool`, *optional*, defaults to `False`): Whether to only save on the global main process safe_serialization (`bool`, *optional*, defaults to `False`): Whether to save `obj` using `safetensors` or the traditional PyTorch way (that uses `pickle`). r"pt)metadataN)r distributed_typerXLAxm_maybe_convert_to_cpursafe_save_filer)rrrsaveis_main_processis_local_main_process)objfrr save_funcs&&&& r#rrs$~&&/*=*==&&s+Nh5EF c; ' '237CJJ ~%%%.?#  - - -2C#3D -r%z2.0.0z1.25.0c |\4'dO\PP4pRV9dRVR&\PP \ 4MVP RR4\P!V3RV/VBp\4'dF\PP4X'd \PP V4V# \4'dH\PP4X'd!\PP T4iii;i)a Compatible drop-in replacement of `torch.load()` which allows for `weights_only` to be used if `torch` version is 2.4.0 or higher. Otherwise will ignore the kwarg. Will also add (and then remove) an exception for numpy arrays Args: f: The file (or file-like object) to use to load the data map_location: a function, `torch.device`, string or a dict specifying how to remap storage locations **kwargs: Additional keyword arguments to pass to `torch.load()`. weights_onlyTN map_location) rr serializationget_safe_globalsadd_safe_globalsTORCH_SAFE_GLOBALSrloadclear_safe_globals)rrkwargsold_safe_globals loaded_objs&&, r#rrs G $ & &$22CCE V+)-~&    0 01C D JJ~t ,ZZG GG $ & &    2 2 4##445EF  % & &    2 2 4##445EF  'sB C""6D;"D;c\VR4'g \VR4'g\VRV4p\VR4'd VP#\VR4'd VP#\ V4#)z Gets a pretty name from `obj`. __qualname____name__r8)r(rrrr)rs&r#get_pretty_namersd 3 ' 'Z0H0Hc;,sN##sJ|| s8Or%cVP4F>wr#\V\4'd VPV/4p\ W44K:W1V&K@ V#)z Recursively merges two dictionaries. Args: source (`dict`): The dictionary to merge into `destination`. destination (`dict`): The dictionary to merge `source` into. )rr)r setdefault merge_dicts)source destinationkeyvaluenodes&& r#rrsMlln  eT " "))#r2D  $$  % r%cFV^8dQhR\\,R\/#)rportr)rintr!)r"s"r#r$r$s66#6$6r%cVfRp\P!\P\P4;_uu_4pVPRV34^8HuuRRR4# +'giR#;i)z Checks if a port is in use on `localhost`. Useful for checking if multiple `accelerate launch` commands have been run and need to see if the port is already in use. Ni> D [[F XX0$,,?NAKw'+..&wi/[\g[his s  sd3 /r%c$V^8dQhR\/#)rattr)r)r"s"r#r$r$%s555r%cLRp\W .VPR4,4#)z Recursive `getattr`. Args: obj: A class instance holding the attribute. attr (`str`): The attribute that is to be retrieved, e.g. 'attribute1.attribute2'. c\W4#rJ)r)rrs&&r#_getattr#recursive_getattr.._getattr0s s!!r%.)rr)rrrs&& r#recursive_getattrr%s#" (EDJJsO3 44r%cV^8dQhR\PPR\R\\PP,/#)rrd return_fqnsr)rrr r!r)r"s"r#r$r$6s<!!!t!X\]b]e]e]l]lXm!r%cV'gTMRV3pV.p.pV'dVP4pV'dVwreVP4Fxwrx\V\PP 4'gK1V'd/X'dVR,V,MTp VP W34KgVP V4Kz V'dVP XV34KVP V4KVRRR1,#)a)Traverse the model in bottom-up order and return the children modules in that order. Args: model (`torch.nn.Module`): the model to get the children of Returns: `list[torch.nn.Module]`: a list of children modules of `model` in bottom-up order. The last element is the `model` itself. rrN)rrPr)rrr rK) rdr topstackordered_modulescurrent_modulecurrent_module_namerTr child_names && r#get_module_children_bottom_upr6s#%U C EEO  2@ / (779JD$00EX!4s!:T!A^bJLL*!34LL& :   " "$7#H I  " "> 2 4R4  r%)TTF)FFrJ)F)Qrrrrcodecsrr functoolsrrtypesrtypingrnumpynprpackaging.versionr safetensors.torchr rcommands.config.defaultr loggingr stater constantsr dataclassesrimportsrrrrrmodelingrtransformer_enginerversionsrrrtorch_xla.core.xla_modelcore xla_modelrr-r4r@rCrXr]rbrorrrr_corenp_core multiarray _reconstructndarraydtyperrKdtypes UInt32DTyperrrrrrrrrr[r%r#r/sP #% %98 +((-& H ))H   EP)04C(Up ' FH)11"((rww ##JJ HH hbii334@ $6 ""4$5"!!r%