+ Lj^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RIHt^RIH t ^RI H t H t ^RI t^RIt^RIHuHt]P(P+4'd^RIHtHt^RIHt^RIHtMRtRtRtRt^RIHt^R IH t ^R I!H"t"^R I#H$t$H%t%H&t&H't'H(t(H)t)H*t*H+t+]+!4'd3^RI,t,],PZP\P_4'd^RI.t.]'!4'd^R I0H1t1](!4'd^R I2H3t3]*!4'd^RI4H5t5])!4'd^RI6t6RRlt7Rt8RRRRRRRR9/RRllt9RRlt:R:RRllt;RRlt<]Pz3R R!llt>R"R#lt?R$R%lt@R;R&R'lltARN)contextmanager)partial)AnyIterable) CPUOffloadShardingStrategy)FullyShardedDataParallel)transformer_auto_wrap_policy)UNet2DConditionModel)DiffusionPipeline)SchedulerMixin)convert_state_dict_to_diffusersconvert_state_dict_to_peft deprecateis_accelerate_availableis_peft_availableis_torch_npu_availableis_torchvision_availableis_transformers_available) get_logger)set_peft_model_state_dict) transformsc$V^8dQhR\/#)seed)int)formats"D/app/.local/lib/python3.14/site-packages/diffusers/training_utils.py __annotate__r=s))3)c<\P!V4\PPV4\P!V4\ 4'd"\P PV4R#\PPV4R#)z Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch`. Args: seed (`int`): The seed to set. Returns: `None` N) randomrnptorch manual_seedrnpumanual_seed_allcuda)rs&rset_seedr)=s[ KKIINN4 d !!$' ""4(r c~VPpVR,pRV, R,pVPVPR7V,P4p\ VP 4\ VP 48d VR,pK8VP VP 4pVPVPR7V,P4p\ VP 4\ VP 48d VR,pK8VP VP 4pWV, ^,pV#)a_ Computes SNR as per https://github.com/TiankaiHang/Min-SNR-Diffusion-Training/blob/521b624bd70c67cee4bdf49225915f5945a872e3/guided_diffusion/gaussian_diffusion.py#L847-L849 for the given timesteps using the provided noise scheduler. Args: noise_scheduler (`NoiseScheduler`): An object containing the noise schedule parameters, specifically `alphas_cumprod`, which is used to compute the SNR values. timesteps (`torch.Tensor`): A tensor of timesteps for which the SNR is computed. Returns: `torch.Tensor`: A tensor containing the computed SNR values for each timestep. ??device).N)alphas_cumprodtor.floatlenshapeexpand)noise_scheduler timestepsr/sqrt_alphas_cumprodsqrt_one_minus_alphas_cumprodalphasigmasnrs&& r compute_snrr<Qs %33N(#-%(>%9c$A!.00 8H8H0I)TZZ\ !'' (3y+? ?1)<  & &y 7E$A$D$DIL\L\$D$]^g$h$n$n$p! +11 2S5I I(Ei(P% ) 0 0 AE =Q C Jr lambda_conf temperaturer,per_token_weights ignore_indexcV^8dQhR\PR\PR\R\R\PR,R\R\\P\P\P3,/#) rlogitslabelsr=r?r@NrAreturn)r$Tensorr1rtuple)rs"rrrvsS%S% LLS% LLS% S%  S% ||d* S%S% 5<<u|| 34S%r c*VP^8d#\R\VP4 R24hVPVPRR8wd9\R\VP4 R\VP4 R24hV^8:d\RV R24hVP V4pVf#\ P !WPR7pMoVPVP8wd9\R\VP4 R \VP4 R24hVPVPR7pVPR,p\P!VPRV4VPR4R VR 7PV4p WvPVP4,P4P^4p W,VPV P4,P4V , p VR 8Xd2\ P!RVP V PR 7p WV 3#\ P"!4;_uu_4VP%RR7p WmP'V4,pRRR4VP)4pVR8wdV\)V4, p\ P*!VRR7p\ P,!VP4P.p\ P0!VPV44pVV,PRR7PVPR7)pVXPVP4,P4P^4pVV,VPVP4,P4V, p V \)V4V ,,pVW3# +'giELo;i)a Computes a confidence-aware training loss for token classification-style heads. This loss combines: - `loss_sft`: standard supervised cross-entropy on all non-ignored labels. - `loss_conf`: an entropy penalty applied only on tokens that are already predicted correctly. Args: logits (`torch.Tensor`): Logits of shape `(..., vocab_size)`. labels (`torch.Tensor`): Labels of shape `(...)`, matching `logits.shape[:-1]`. Values set to `ignore_index` are excluded from both losses. lambda_conf (`float`, *optional*, defaults to `0.0`): Weight for the confidence term. temperature (`float`, *optional*, defaults to `1.0`): Temperature used for the entropy term only. Lower values sharpen the distribution and change the strength of the confidence gradients. per_token_weights (`torch.Tensor`, *optional*): Optional weights of shape `(...)` to reweight both losses per token (e.g. schedule-aware weights). Tokens with weight `0` contribute nothing. ignore_index (`int`, *optional*, defaults to `-100`): Ignore index for labels. Returns: `Tuple[torch.Tensor, torch.Tensor, torch.Tensor]`: `(loss, loss_sft, loss_conf)`. z.`logits` must have at least 2 dims, got shape .Nz:`labels` shape must match `logits.shape[:-1]`, got labels=z logits=z`temperature` must be > 0, got )dtypez9`per_token_weights` shape must match `labels` shape, got z != none) reductionrAr>)r.rJ)dimr,)ndim ValueErrorrGr3ner$ ones_likerJr0F cross_entropyreshape reshape_assum clamp_minzerosr.no_gradargmaxeqr1softmaxfinfotinylog)rCrDr=r?r@rAvalidweights vocab_size per_token_nll denom_sftloss_sft loss_confpredcorrect scaled_logitsprobseps log_probsentropy denom_conflosss&&$$$$ rcompute_confidence_aware_lossrrvsU<{{QI%PVP\P\J]I^^_`aa ||v||CR((Hv||I\H]]efklrlxlxfyezz{ |  a:;-qIJJ IIl #E //& =  " "fll 2KERcRiRiLjKkkopuv|wCwCqDpEEFG $&&V\\&:b!JOOr:&r!  j 88GMM22779CCAFI'%((=3F3F*GGLLNQZZHcKK6==O 9,, }}}$''&/) LLNMc%k(:: MM-R 0E ++ekk " ' 'C %//#./I !&&2&.11 1EEGGJJw}}55::<FFqIJ7"WZZ %>>CCE RI eK(94 4D  $$# s 1*P P c$V^8dQhR\/#)rinterpolation_typestr)rs"rrrs))3)r c@\4'g \R4hVR8Xd\PPpV#VR8Xd\PP pV#VR8Xd\PP pV#VR8Xd\PPpV#VR8Xd\PPpV#VR8Xd\PPpV#VR8Xd\PPpV#\R V R 24h) a Maps a string describing an interpolation function to the corresponding torchvision `InterpolationMode` enum. The full list of supported enums is documented at https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.functional.InterpolationMode. Args: interpolation_type (`str`): A string describing an interpolation method. Currently, `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos` are supported, corresponding to the supported interpolation modes in torchvision. Returns: `torchvision.transforms.InterpolationMode`: an `InterpolationMode` enum used by torchvision's `resize` transform. zhPlease make sure to install `torchvision` to be able to use the `resolve_interpolation_mode()` function.bilinearbicubicboxnearest nearest_exacthamminglanczoszThe given interpolation mode z is not supported. Currently supported interpolation modes are `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos`.) r ImportErrorrInterpolationModeBILINEARBICUBICBOXNEAREST NEAREST_EXACTHAMMINGLANCZOSrQ)rtinterpolation_modes& rresolve_interpolation_moders> $ % % v  Z''99BB& % y ('99AA" ! u $'99==  y ('99AA   .'99GG  y ('99AA  y ('99AA  +,>+?@m n  r c2V^8dQhR\R\R\PR\PR\PR\PR\PR\R \ \P\P3,/ #) runetr5r6noise noisy_latentstargetencoder_hidden_statesdream_detail_preservationrE)r r r$rFr1rG)rs"rrrs3#3# 3##3#||3# << 3# << 3# LL 3#!<<3# %3# 5<< %&3#r cVPPVP4VRRR3,pRV, R,p W,p Rp \P!4;_uu_4V!WBV4P p RRR4RRrVP PR8XdUT pW>, P4pVPV 4VPW,4p VPV4p W3#VP PR8Xd \R4h\RVP P 24h +'giL;i)a Implements "DREAM (Diffusion Rectification and Estimation-Adaptive Models)" from https://huggingface.co/papers/2312.00210. DREAM helps align training with sampling to help training be more efficient and accurate at the cost of an extra forward step without gradients. Args: `unet`: The state unet to use to make a prediction. `noise_scheduler`: The noise scheduler used to add noise for the given timestep. `timesteps`: The timesteps for the noise_scheduler to user. `noise`: A tensor of noise in the shape of noisy_latents. `noisy_latents`: Previously noise latents from the training loop. `target`: The ground-truth tensor to predict after eps is removed. `encoder_hidden_states`: Text embeddings from the text model. `dream_detail_preservation`: A float value that indicates detail preservation level. See reference. Returns: `tuple[torch.Tensor, torch.Tensor]`: Adjusted noisy_latents and target. Nr,r+epsilon v_predictionz/DREAM has not been implemented for v-predictionzUnknown prediction type ) r/r0r.r$r[sampleconfigprediction_typedetachmul_addNotImplementedErrorrQ)rr5r6rrrrrr/r8 dream_lambdari_noisy_latents_targetpredicted_noise delta_noises&&&&&&&& r compute_dream_and_update_latentsrs+:%3366y7G7GHTXZ^`dIdeN%(>%9c$A!1KL D M.CDKK  $TG--:.668 &&**+H+VW**[)  ""    / /> A!"STT3O4J4J4Z4Z3[\]] s (D>> E cfV^8dQhR\R\\\P3,/#)rrrE)r dictrvr$rF)rs"rrr.s)3S%,,=N8Or c/pVP4F]wr#\VR4'gK\VR4pVfK+VP4pVP 4FwrgWqV RV 2&K K_ V#)z@ Returns: A state dict containing just the LoRA parameters. set_lora_layer lora_layerz.lora.) named_moduleshasattrgetattr state_dictitems)rlora_state_dictnamemodulercurrent_lora_layer_sdlora_layer_matrix_name lora_params& runet_lora_state_dictr.s O**,  6+ , , 6J%(2(=(=(?%:O:U:U:W6*OYtfF3I2J$KL;X - r cV^8dQhR\PP\\PP,,/#)rmodel)r$nnModulelist)rs"rrrAs---$uxx2G G-r c\V\4'gV.pVFDpVP4F-pVP'gKVP V4VnK/ KF R#)z Casts the training parameters of the model to the specified data type. Args: model: The PyTorch model whose parameters will be cast. dtype: The data type to which the model parameters will be cast. N) isinstancer parameters requires_gradr0data)rrJmparams&& rcast_training_paramsrAsM eT " " \\^E""""XXe_ $r cV^8dQhR\\\P3,R\R\PP /#)rrprefix text_encoder)rrvr$rFrr)rs"rrrRs?]]#u||+,]69]IN]r cVP4UUu/uF2wr4VPV4'gKVPVR4 VbK4 ppp\\ V44p\ W%RR7R#uuppi)a, Sets the `lora_state_dict` into `text_encoder` coming from `transformers`. Args: lora_state_dict: The state dictionary to be set. prefix: String identifier to retrieve the portion of the state dict that belongs to `text_encoder`. text_encoder: Where the `lora_state_dict` is to be set. default) adapter_nameN)r startswithreplacerrr)rrrkvtext_encoder_state_dicts&&& r!_set_state_dict_into_text_encoderrRss3B2G2G2I2I$!Q\\Z`Ma%199VR !A%2I99XYp9qrlR[\ s A2A2cV^8dQhR\\\PP3,R\\\ 3,/#)rmodules_to_saverE)rrvr$rrr)rs"rrres7Dehhoo1E,F4PSUXPX>r c/pVP4F1wr#VfK VPR,P4W R2&K3 V#)Nr_lora_adapter_metadata)r peft_configto_dict)r metadatas module_namers& r_collate_lora_metadataresPI.446  @F@R@RS\@]@e@e@gI %;< = 7 r cV^8dQhR\R\R\R\R\R\P\,R\P R,/#) rweighting_scheme batch_size logit_mean logit_std mode_scaler. generatorN)rvrr1r$r. Generator)rs"rrrms`         LL3   % r cVR8XdF\P!W#V3WVR7p\PPP V4pV#VR8Xdx\P !V3WVR7p^V, V\P !\PV,^, 4^,^, V,,, pV#\P !V3WVR7pV#)z Compute the density for sampling the timesteps when doing SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. logit_normal)meanstdsizer.rmode)rr.r) r$normalr functionalsigmoidrandcosmathpi)rrrrrr.rus&&&&&&& r%compute_density_for_timestep_samplingrms >) LLjzmTZ p HH   ' ' * H V # JJZM& N EJ%))DGGaK!O"<"AA"E"IJ J H JJZM& N Hr c$V^8dQhR\/#)rrru)rs"rrrsSr c VR8XdVR,P4pV#VR8XdF^^V,, ^V^,,,p^\PV,, pV#\P!V4pV#)z Computes loss weighting scheme for SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. sigma_sqrtcosmapg)r1rrr$rS)rsigmas weightingbots&& rcompute_loss_weighting_for_sd3rsz<'T\((*  X %!f*nq619},3'  OOF+ r cl\P!4\PP 4'd!\PP 4R#\P PP 4'd!\PP 4R#\4'd!\PP 4R#\\R4'dG\PP 4'd!\PP 4R#R#R#)zN Runs garbage collection. Then clears the cache of the available accelerator. xpuN) gccollectr$r( is_available empty_cachebackendsmpsr torch_npur&rrrOr r free_memoryrsJJL zz       ( ( * *   ! ! !!#   599#9#9#;#; $< r offloadTcV^8dQhR\PP\,R\\P ,R\ /#)rmodulesr.r)r$rrr rvr.bool)rs"rrrs<UXX__/@@#PUP\P\J\gkr c'b"V'd\;QJdRV4F 'gK RM RM !RV44'*pV'd4VUu.uF&p\VP44PNK( ppM&\ V4^8XgQhV^,P.pVFpVP V4K RxV'd)\ VX4FwrFVP V4K R#R#uupi T'd(\ TX4FwrFTP T4K ii;i5i)a Context manager that, if offload=True, moves each module to `device` on enter, then moves it back to its original device on exit. Args: device (`str` or `torch.Device`): Device to move the `modules` to. offload (`bool`): Flag to enable offloading. c3B"TFp\V\4xK R#5iN)rr ).0rs& r !offload_models..sMW:a):;;WsTFN)anynextrr.r2r0zip)r.rris_modelroriginal_devicesorig_devs$$* roffload_modelsrssMWMsssMWMMM ELMWQ\\^ 4 ; ;W M w<1$ $$ ' 1 12 A DDL "7,<= X >  N "7,<= X > s:D/D/!D/ D/,C6> !ij j    $lm m**4+<+<=__& -   .:. U%%'( d#r c&<V^8dQhRS[RS[/#)rrprE)rr1)rrds"rrres35r c j\^WP, ^, 4pV^8:dR#VP'd4^^W P, ,VP),, pM^V,^ V,, p\ W0P 4p\W0P4pV#)z> Compute the decay factor for the exponential moving average. r>)maxr\r]r^r_minrZr[)rrrpsteprqs&& r get_decayEMAModel.get_decays1'*@*@@1DE 19    1tnn'<#<$**"LLO 4xBI6Oozz:o~~>r cX<V^8dQhRS[S[PP,/#)rrrr$rrc)rrds"rrres#7-7-x(:(:;7-r c \V\PP4'd"Rp\ RRVRR7VP 4p\ V4pV;P^, unVPVP4pW0n ^V, p\P!4pVP'Ed\4'dP\PP P#4'd"\ P$P'VRR7pT;_uu_4VUu.uFqfP('gKVNK pp\+VP,V4UUu.uFwrVP('gKVNK p pp\/V4\/V48dy\P0!\+VP,V4UUu.uFwrVP('dKVNK uppVUu.uFqfP('dKVNK upRR7\P2!V \P4!W4VR 7RRR4R#\+VP,V4Fwr\4'dP\PP P#4'd"\ P$P'VRR7pT;_uu_4VP('d VP7WHV, ,4MVP9V4RRR4K R#uupiuuppiuuppiuupi +'giR#;i +'giK;i) zPassing a `torch.nn.Module` to `ExponentialMovingAverage.step` is deprecated. Please pass the parameters of the module instead.z>passing a `torch.nn.Module` to `ExponentialMovingAverage.step`rgFrhN) modifier_rankT) non_blocking)r9)rr$rrrrrrprrq contextlib nullcontextr`r transformers integrations deepspeedis_deepspeed_zero3_enabledzeroGatheredParametersrr rnr2_foreach_copy_ _foreach_sub_ _foreach_subsub_copy_) rrrrsrZone_minus_decaycontext_managerr params_grads_param s_params_grads && rr EMAModel.steps j%((// 2 2D  P##   $..0J*%  !#t556$e)$002 <<<(**|/H/H/R/R/m/m/o/o"+.."C"CJ^b"C"c 2<T*@S@Suu* T25d6H6H*2U!2UY^YlYlGG2U!{#c*o5((7:4;M;Mz7Zv7Z^Wbgbubu7Zv,6RJ5>Q>QJR%) ##!5#5#5m#QYh!$#&d&8&8*"E,..<3L3L3V3V3q3q3s3s&/nn&G&G]a&G&bO$_*** _%%HI e, %_ #F#U! wR!,%__sh+M0L*L*M)L/ L/ AML5*L50M7L;L;6MAM*M M  M% c\<V^8dQhRS[S[PP,RR/#rrrENr)rrds"rrres.@@(588+=+=">@4@r c  \V4pVP'd~\P!VUu.uFq"PNK up\ VP V4UUu.uF*wr2VPVP4PNK, upp4R#\ VP V4FCwr2VPPVPVP4P4KE R#uupiuuppi)a1 Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored moving averages. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. N) rr`r$rrr rnr0r.r)rrrrrs&& rrEMAModel.copy_tos*% <<<  )344EHI[I[]gEhiEh>7ELL)..Ehi  #&d&8&8*"E   ELL!9!>!>?#F 5is C:!0C? c<V^8dQhRR/#rrENrO)rrds"rrresJJDJr c lVPUu.uFqP4NK upVnR#uupi)z Move internal buffers of the ExponentialMovingAverage to pinned memory. Useful for non-blocking transfers for offloading EMA params to the host. N)rn pin_memory)rrrts& rrEMAModel.pin_memorys* 7;6H6HI6Hlln6HIIs1c<V^8dQhRR/#rrO)rrds"rrres     r c  VPUu.uF>pVP4'dVPWVR7MVPWR7NK@ upVnR#uupi)z Move internal buffers of the ExponentialMovingAverage to `device`. Args: device: like `device` argument to `torch.Tensor.to` )r.rJr)r.rN)rnis_floating_pointr0)rrr.rJrrts&&&& rr0 EMAModel.tose''  (""$$ DD,D GV? @(   s A)Ac <V^8dQhRS[/#r1r2)rrds"rrre)s  D r c RVPRVPRVPRVPRVPRVP RVP RVP/#) z Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during checkpointing to save the ema state dict. rZr[rpr\r]r^r_rn)rZr[rpr\r]r^r_rn)rrs&rrEMAModel.state_dict)s^ TZZ  !7!7 !7!7 d11  TZZ T//  r c\<V^8dQhRS[S[PP,RR/#rr)rrds"rrre<s.YY);); <YYr c VUu.uF.q"P4P4P4NK0 upVnR#uupi)z Saves the current parameters for restoring later. Args: parameters: Iterable of `torch.nn.Parameter`. The parameters to be temporarily stored. N)rr4rmro)rrrrs&& rstoreEMAModel.store<s6NX"XZE<<>#5#5#7#=#=#?Z"X"Xs4Ac\<V^8dQhRS[S[PP,RR/#rr)rrds"rrreEs*''(588+=+=">'4'r c VPf \R4hVP'dU\P!VUu.uFq"P NK upVPUu.uFq3P NK up4MD\ VPV4F*wr2VP PVP 4K, RVnR#uupiuupi)a Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without: affecting the original optimization process. Store the parameters before the `copy_to()` method. After validation (or model saving), use this to restore the former parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`)ro RuntimeErrorr`r$rrr r)rrrrc_params&& rrestoreEMAModel.restoreEs  " " *hi i <<<  )344SWSjSj6kSj||Sj6k #&d&=&=z"J   .#K#'56ks C $C c$<V^8dQhRS[RR/#)rrrENr2)rrds"rrre^s.F.F$.F4.Fr c \P!V4pVPRVP4VnVPR8gVPR8d \ R4hVPRVP 4Vn\ VP \4'g \ R4hVPRVP4Vn\ VP\4'g \ R4hVPR VP4Vn \ VP\4'g \ R 4hVPR VP4Vn \ VP\4'g \ R 4hVPR VP4Vn \ VP\\34'g \ R4hVPRVP4Vn\ VP\\34'g \ R4hVPRR4pVeW n\ VP\ 4'g \ R4h\";QJd&RVP4F 'dK RM RM!RVP44'g \ R4hR#R#)z Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the ema state dict. Args: state_dict (dict): EMA state. Should be an object returned from a call to :meth:`state_dict`. rZr>r,zDecay must be between 0 and 1r[zInvalid min_decayrpzInvalid optimization_stepr\zInvalid update_after_stepr]zInvalid use_ema_warmupr^zInvalid inv_gammar_z Invalid powerrnNzshadow_params must be a listc3V"TFp\V\P4xK! R#5ir)rr$rF)rrts& rr+EMAModel.load_state_dict..s OrFr,gUUUUUU?FNN)F)NNF)rO __module__ __qualname____firstlineno____doc__ru classmethodr{rrr$r[rrrr0rrrr|__static_attributes____classdictcell__)rds@rrXrX@sR)R)h $& ]]_7-7-r@@&JJ     &YY''2.F.Fr rXi)r,)NNNr4Nr)TTTNN)Krrrrr"rrr functoolsrtypingrrnumpyr#r$torch.nn.functionalrrrT distributedrtorch.distributed.fsdprrr rItorch.distributed.fsdp.wrapr modelsr pipelinesr schedulersr utilsrrrrrrrrrrrraccelerate.loggingrpeftr torchvisionrrr)r<rrrrrfloat32rrrrrrrr!r/r7rArVrXrOr rrs %   !!##CGHJ D#' ((&     **EEGG-.&)("JS% S%  S% .2 S%S%l)X3#l&PU}}-"]& 6$  nr>8 t./fLFLFr