+ Lj^RIHtHt^RIt^RIHt^RIHuHt^RI H t H t ^RI H t HtHt^RIHt^RIHtHtHtHtHtHt^RIHtHtHt^RIHt^R IH t H!t!H"t"H#t#H$t$]!4'd^RI%t&MRt&] PN!](4t)!R R 4t*!R R 4t+RRlt,]!RR]PZ44t.]!RR]PZ44t/]!RR]PZ44t0!RR]PZ4t1]!RR]PZ44t2!RR]PZ4t3]!RR]PZ44t4!RR]PZ4t5R#) )AnyCallableN) deprecatelogging)is_torch_npu_availableis_torch_xla_availableis_xformers_available)maybe_allow_in_graph)GEGLUGELUApproximateGELUFP32SiLULinearActivationSwiGLU) AttentionAttentionProcessorJointAttnProcessor2_0)SinusoidalPositionalEmbedding) AdaLayerNormAdaLayerNormContinuousAdaLayerNormZeroRMSNormSD35AdaLayerNormZeroXcZa]tRt^'to]V3RlRl4tV3RlRltRtRtRt Vt R#) AttentionMixinc6<V^8dQhRS[S[S[3,/#)return)dictstrr)format __classdict__s"F/app/.local/lib/python3.14/site-packages/diffusers/models/attention.py __annotate__AttentionMixin.__annotate__)sc+=&=!>c ba/pRV3RlloVP4Fwr#S!W#V4K V#)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. cV^8dQhR\R\PPR\\\ 3,/#)rnamemodule processors)r torchnnModulerr)r!s"r#r$4AttentionMixin.attn_processors..__annotate__2s6  c 588?? X\]`bt]tXu r&c<\VR4'dVP4W R2&VP4Fwr4S!V RV 2WB4K V#) get_processor .processor.)hasattrr1named_children)r)r*r+sub_namechildfn_recursive_add_processorss&&& r#r8CAttentionMixin.attn_processors..fn_recursive_add_processors2sZv//282F2F2H V:./#)#8#8#:+tfAhZ,@%T$; r&)r5)selfr+r)r*r8s& @r#attn_processorsAttentionMixin.attn_processors(s=   !//1LD 'j A2r&cF<V^8dQhRS[S[S[S[3,,/#)r processor)rrr )r!r"s"r#r$r%@s( A A,>cK]F]A^,^ Ar&c  ,a\VPP44p\V\4'd/\V4V8wd\ R\V4 RV RV R24hRV3RlloVP 4Fwr4S!W4V4K R#)a Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.cXV^8dQhR\R\PP/#)rr)r*)r r,r-r.)r!s"r#r$7AttentionMixin.set_attn_processor..__annotate__Us& T Tc T588?? Tr&c<\VR4'dL\V\4'gVPV4M#VPVP V R244VP 4Fwr4S!V RV 2WB4K R#) set_processorr2r3N)r4 isinstancerrCpopr5)r)r*r>r6r7fn_recursive_attn_processors&&& r#rFFAttentionMixin.set_attn_processor..fn_recursive_attn_processorUssv//!)T22((3(($z7J)KL#)#8#8#:+tfAhZ,@%S$;r&N)lenr;keysrDr ValueErrorr5)r:r>countr)r*rFs&& @r#set_attn_processor!AttentionMixin.set_attn_processor@sD((--/0 i & &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   T T!//1LD 'i @2r&c PVPP4F4wrR\VPP49gK+\ R4h VP 4F?p\V\4'gKVP'gK/VP4KA R#)z Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. AddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.N) r;itemsr __class____name__rJmodulesrDAttentionModuleMixin_supports_qkv_fusionfuse_projections)r:_attn_processorr*s& r#fuse_qkv_projections#AttentionMixin.fuse_qkv_projectionsbs| "&!5!5!;!;!= A#n66??@@ !tuu">llnF&"677F [!WARNING] > This API is 🧪 experimental. N)rSrDrTrUunfuse_projections)r:r*s& r#unfuse_qkv_projections%AttentionMixin.unfuse_qkv_projectionsos9 llnF&"677FrN)r)r!r"s"r#r$!AttentionModuleMixin.__annotate__s##'9#d#r&c r\VR4'd\VP\PP 4'dk\V\PP 4'gA\ PRVP RV 24VPPR4WnR#)zu Set the attention processor to use. Args: processor (`AttnProcessor`): The attention processor to use. r>z-You are removing possibly trained weights of z with N) r4rDr>r,r-r.loggerinfo_modulesrE)r:r>s&&r#rC"AttentionModuleMixin.set_processorsx D+ & &4>>588??;;y%((//:: KKGGWW]^g]hi j MM  k *"r&c$<V^8dQhRS[RR/#)rreturn_deprecated_lorarrbool)r!r"s"r#r$ris " "D "EY "r&c .V'g VP#R#)z Get the attention processor in use. Args: return_deprecated_lora (`bool`, *optional*, defaults to `False`): Set to `True` to return the deprecated LoRA attention processor. Returns: "AttentionProcessor": The attention processor in use. N)r>)r:rps&&r#r1"AttentionModuleMixin.get_processors&>> !&r&c <V^8dQhRS[/#)rbackend)r )r!r"s"r#r$ris44S4r&c$^RIHpVPP4Uu0uFq3PkK ppW9d'\ RV: R2RP V4,4hV!VP44pWPn R#uupi))AttentionBackendNamez `backend=z ` must be one of the following: z, N) attention_dispatchry __members__valuesvaluerJjoinlowerr>_attention_backend)r:rvryxavailable_backendss&& r#set_attention_backend*AttentionModuleMixin.set_attention_backendsw?? ""=1r&cZ<V^8dQhRS[RS[S[R,R3,R,RR/#)ruse_xla_flash_attentionpartition_specN.r)rrtupler )r!r"s"r#r$ris;22!%2cDj#o.52  2r&c nV'd\4'g \R4hVPR4R#)a Set whether to use XLA flash attention from `torch_xla` or not. Args: use_xla_flash_attention (`bool`): Whether to use pallas flash attention kernel from `torch_xla` or not. partition_spec (`tuple[]`, *optional*): Specify the partition specification if using SPMD. Otherwise None. is_flux (`bool`, *optional*, defaults to `False`): Whether the model is a Flux model. ztorch_xla is not available _native_xlaN)rrr)r:rris_fluxs&&&&r#set_use_xla_flash_attention0AttentionModuleMixin.set_use_xla_flash_attentions*" #)++!">?? ""=1r&c8<V^8dQhRS[RS[R,RR/#)r'use_memory_efficient_attention_xformers attention_opNr)rrr)r!r"s"r#r$ris*%7%77;%7KSVZ?%7 %7r&c V'd\4'g\RRR7h\PP 4'g \ R4h\4'dQRpVeVwrEVP vr6\P!RRVR7p\PPWwV4pTPR4R#R# \dpThRp?ii;i) ax Set whether to use memory efficient attention from `xformers` or not. Args: use_memory_efficient_attention_xformers (`bool`): Whether to use memory efficient attention from `xformers` or not. attention_op (`Callable`, *optional*): The attention operation to use. Defaults to `None` which uses the default attention operation from `xformers`. zeRefer to https://github.com/facebookresearch/xformers for more information on how to install xformersxformers)r)zvtorch.cuda.is_available() should be True but is False. xformers' memory efficient attention is only available for GPU Ncudadevicedtype)rxr() r ModuleNotFoundErrorr,r is_availablerJSUPPORTED_DTYPESrandnxopsopsmemory_efficient_attention Exceptionr) r:rrrop_fwop_bwrWqes &&& r#+set_use_memory_efficient_attention_xformers@AttentionModuleMixin.set_use_memory_efficient_attention_xformerss 3(**){#ZZ,,.. / ,.. $'3+7LE(-(>(>IE!KK 6O HH??aH**:61 3*!GsA C CCCc B VP'g/\PVPP R24R#\ VRR4'dR#VP PPPpVP PPPp\VR4'EdzVP'Edg\P!VPPPVP PP.4pVP"^,pVP"^,p\$P&!WEVP(WR7VnVP*PP-V4\VR4'dVP('dz\P!VPP.PVP P.P.4pVP*P.P-V4EM\P!VP PPVPPPVP PP.4pVP"^,pVP"^,p\$P&!WEVP(WR7VnVP0PP-V4\VR4'dVP('d\P!VP P.PVPP.PVP P.P.4pVP0P.P-V4\ VRR4Ee\ VR R4Ee\ VR R4Ee\P!VP2PPVP4PPVP6PP.4pVP"^,pVP"^,p\$P&!WEVP8WR7VnVP:PP-V4VP8'd\P!VP2P.PVP4P.PVP6P.P.4pVP:P.P-V4R VnR#) zU Fuse the query, key, and value projections into a single projection for efficiency. zK does not support fusing QKV projections, so `fuse_projections` will no-op.Nfused_projectionsFis_cross_attention)biasrruse_bias add_q_proj add_k_proj add_v_projT)rUrkdebugrQrRgetattrto_qweightdatarrr4rr,catto_kto_vshaper-Linearrto_kvcopy_rto_qkvrrradded_proj_bias to_added_qkvr)r:rrconcatenated_weights in_features out_featuresconcatenated_biass& r#rV%AttentionModuleMixin.fuse_projectionss((( LL>>**++vw   4,e 4 4 !!&&--   %%++ 4- . .43J3J3J#(99dii.>.>.C.CTYYEUEUEZEZ-[#\ .44Q7K/55a8L;4==Y_mDJ JJ   # #$8 9tZ((T]]]$)IItyy~~/B/BDIINNDWDW.X$Y! %%&78$)99dii.>.>.C.CTYYEUEUEZEZ\`\e\e\l\l\q\q-r#s .44Q7K/55a8L))KDMMZ`nDK KK   $ $%9 :tZ((T]]]$)IItyy~~/B/BDIINNDWDWY]YbYbYgYgYlYl.m$n!   &&'89 D, - 9lD1=lD1=#(99'',,doo.D.D.I.I4??KaKaKfKfg$ /44Q7K/55a8L " 0D0DV!D     $ $ * *+? @###$)II__))..0D0D0I0I4??K_K_KdKde%!!!&&,,->?!%r&c VP'gR#\VRR4'gR#\VR4'd \VR4\VR4'd \VR4\VR4'd \VR4RVnR#)zL Unfuse the query, key, and value projections back to separate projections. NrFrrr)rUrr4delattrr)r:s&r#r\'AttentionModuleMixin.unfuse_projections:sw((( t0%88  4 " " D( # 4 ! ! D' " 4 ( ( D. )!&r&c$<V^8dQhRS[RR/#)r slice_sizerNint)r!r"s"r#r$riTs&&c&d&r&c \VR4'd1Ve-WP8d\RV RVP R24hRpVeVPR4pVfVP 4pVP V4R#)z Set the slice size for attention computation. Args: slice_size (`int`): The slice size for attention computation. sliceable_head_dimNz slice_size z has to be smaller or equal to r3sliced)r4rrJ_get_compatible_processordefault_processor_clsrC)r:rr>s&& r#set_attention_slice(AttentionModuleMixin.set_attention_sliceTs 4- . .:3Ij[r[rNr{:,6UVZVmVmUnnopq q   !66x@I  224I 9%r&cN<V^8dQhRS[PRS[P/#)rtensorrr,Tensor)r!r"s"r#r$riks# r&c VPpVPwr4pVPW2,W$V4pVP^^^^4PW2,WEV,4pV#)z Reshape the tensor from `[batch_size, seq_len, dim]` to `[batch_size // heads, seq_len, dim * heads]`. Args: tensor (`torch.Tensor`): The tensor to reshape. Returns: `torch.Tensor`: The reshaped tensor. )headsrreshapepermute)r:r head_size batch_sizeseq_lendims&& r#batch_to_head_dim&AttentionModuleMixin.batch_to_head_dimks_JJ #)<< S 7SQ1a+33J4KW\eVef r&cT<V^8dQhRS[PRS[RS[P/#)rrout_dimrr,rr)r!r"s"r#r$ri{s* s5<<r&c BVPpVP^8XdVPwrEp^pMVPwrGrVVPWEV,W6V,4pVP ^^^^4pV^8Xd&VPWC,WW,Wc,4pV#)z Reshape the tensor for multi-head attention processing. Args: tensor (`torch.Tensor`): The tensor to reshape. out_dim (`int`, *optional*, defaults to `3`): The output dimension of the tensor. Returns: `torch.Tensor`: The reshaped tensor. )rndimrrr)r:rrrrrr extra_dims&&& r#head_to_batch_dim&AttentionModuleMixin.head_to_batch_dim{sJJ ;;! '-|| $JI28,, /J7 i,?S\L\]1a+ a<^^J$:G)LO)Z]) )r&c lVPpVfV#VPR,pWb8wdVPPR8XdnVP^,VP^,V3p\P !WqP VPR7p\P!W.^R7pM\P!V^V3RR7pV^8Xd4VP^,W5,8dVPV^R7pV#V^8Xd%VP^4pVPV^R7pV#)a Prepare the attention mask for the attention computation. Args: attention_mask (`torch.Tensor`): The attention mask to prepare. target_length (`int`): The target length of the attention mask. batch_size (`int`): The batch size for repeating the attention mask. out_dim (`int`, *optional*, defaults to `3`): Output dimension. Returns: `torch.Tensor`: The prepared attention mask. mpsrr)r}r) rrrtyper,zerosrrFpadrepeat_interleave unsqueeze) r:rrrrrcurrent_length padding_shapepaddings &&&&& r#prepare_attention_mask+AttentionModuleMixin.prepare_attention_masks!JJ  !! !,2226  *$$))U2"0!5!5a!8.:N:Nq:QS` a ++m;O;OXfXmXmn!&N+D!!L "#~=7IQT!U a<##A&)??!/!A!A)QR!A!S  \+55a8N+==iQ=ONr&cN<V^8dQhRS[PRS[P/#)rencoder_hidden_statesrr)r!r"s"r#r$ris&%% %QVQ]Q]%r&c lVPfQR4h\VP\P4'dVPV4pV#\VP\P4'd8VP ^^4pVPV4pVP ^^4pV#Qh)z Normalize the encoder hidden states. Args: encoder_hidden_states (`torch.Tensor`): Hidden states of the encoder. Returns: `torch.Tensor`: The normalized encoder hidden states. zGself.norm_cross must be defined to call self.norm_encoder_hidden_states) norm_crossrDr- LayerNorm GroupNormr)r:rs&&r#norm_encoder_hidden_states/AttentionModuleMixin.norm_encoder_hidden_statess*u,uu* door|| 4 4$(OO4I$J !%$ 6 6 %:$C$CAq$I !$(OO4I$J !$9$C$CAq$I !%$ 5r&)rr>rrr)F)NFN))rRr`rarb_default_processor_cls_available_processorsrUrrCr1rrrrr,no_gradrVr\rrrrrrrdrerfs@r#rTrTys!##( " "44 2 222.%7%7N ]]_@&@&D ]]_''2&&. 2--^))V%%r&rTcpV^8dQhR\PR\PR\R\/#)rff hidden_states chunk_dim chunk_size)r-r.r,rr)r!s"r#r$r$ s2  bii  QT be r&cBVPV,V,^8wd$\RVPV, RV R24hVPV,V,p\P!VP WBR7Uu.uF qP!V4NK upVR7pV#uupi)rz)`hidden_states` dimension to be chunked: z$ has to be divisible by chunk size: z[. Make sure to set an appropriate `chunk_size` when calling `unet.enable_forward_chunking`.r)rrJr,rchunk)r$r%r&r' num_chunks hid_slice ff_outputs&&&& r#_chunked_feed_forwardr- s9% 2a77 8K8KI8V7WW{}G|HHc d  $$Y/:=J (5(;(;J(;(VW(V9I(VW I  XsBcTaa]tRtRtoRtV3RlV3RlltV3RlRltRtVtV;t #)GatedSelfAttentionDenseiaX A gated self-attention dense layer that combines visual features and object features. Parameters: query_dim (`int`): The number of channels in the query. context_dim (`int`): The number of channels in the context. n_heads (`int`): The number of heads to use for attention. d_head (`int`): The number of channels in each head. c2<V^8dQhRS[RS[RS[RS[/#)r query_dim context_dimn_headsd_headr)r!r"s"r#r$$GatedSelfAttentionDense.__annotate__%s)#C#sr&c<\SV`4\P!W!4Vn\ WVR7Vn\VRR7Vn\P!V4Vn \P!V4Vn VPR\P!\P!R444VPR\P!\P!R444RVnR#))r1rdim_headgeglu activation_fn alpha_attnr  alpha_denseTN)super__init__r-rlinearrattn FeedForwardr$rnorm1norm2register_parameter Parameterr,renabled)r:r1r2r3r4rQs&&&&&r#r> GatedSelfAttentionDense.__init__%s ii 7  6R iw?\\), \\),   bll5<<;L.MN  r||ELLY3Z)[\]_h`h_hjk\k)ll l   %%'$''$**Q-*@@ @r&)r@rFr$r?rBrC rRr`rarb__doc__r>rNrdre __classcell__rQr"s@@r#r/r/s#"   r&r/craa]tRtRtoRtR V3RlV3RllltR V3RlRlltR V3RlRlltR tVt V;t #) JointTransformerBlockiCa A Transformer block following the MMDiT architecture, introduced in Stable Diffusion 3. Reference: https://huggingface.co/papers/2403.03206 Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. context_pre_only (`bool`): Boolean to determine if we should add some blocks associated with the processing of `context` conditions. c L<V^8dQhRS[RS[RS[RS[RS[R,RS[/#)rrnum_attention_headsattention_head_dimcontext_pre_onlyqk_normNuse_dual_attention)rrrr )r!r"s"r#r$"JointTransformerBlock.__annotate__RsSOO O!O O  O t O!Or&c<\S V`4W`nW@nV'dRMRpV'd\ V4VnM\ V4VnVR8Xd\WRRRRR7VnM'VR8Xd\ V4VnM\RV R24h\\R 4'd \4pM \R 4h\VR VVVVVRVVRR 7 VnV'd\VR VVVRVVRR 7 VnMR Vn\ P"!VRRR7Vn\'WRR7VnV'g2\ P"!VRRR7Vn\'WRR7VnMR VnR VnR Vn^VnR #)ada_norm_continous ada_norm_zeroFư>T layer_norm)elementwise_affineepsr norm_typezUnknown context_norm_type: z>, currently only support `ada_norm_continous`, `ada_norm_zero`scaled_dot_product_attentionzYThe current PyTorch version does not support the `scaled_dot_product_attention` function.N) r1cross_attention_dimadded_kv_proj_dimr7rrrYrr>rZrc) r1rfr7rrrr>rZrcrbrcgelu-approximate)rdim_outr:)r=r>r[rYrrBrr norm1_contextrJr4r rrr@attn2r-rrCrAr$ norm2_context ff_context _chunk_size _chunk_dim) r:rrWrXrYrZr[context_norm_typer>rQs &&&&&&& r#r>JointTransformerBlock.__init__Rs "4 04D0/ .s3DJ)#.DJ  4 4!7U4S_"D / 1!1#!6D -.?-@@~  14 5 5-/Ik  $!'%-    "$(+)# DJDJ\\#%TJ #BTU!#ceQU!VD )cN`aDO!%D "DO r&c4<V^8dQhRS[R,RS[/#rr'Nrr)r!r"s"r#r$r\t#r&cWnW nR#rrorpr:r'rs&&&r#set_chunk_feed_forward,JointTransformerBlock.set_chunk_feed_forward %r&c <V^8dQhRS[PRS[PRS[PRS[S[S[3,R,RS[S[P S[P 3,/#)rr%rtembjoint_attention_kwargsNr)r, FloatTensorrr rrr)r!r"s"r#r$r\suC4C4((C4 %00C4 C4 !%S#X 5 C4 u||U\\) * C4r&c T;'g/pVP'dVPWR7wrVrxrp MVPWR7wrVrxp VP'dVPW#4p MVPW#R7wrrpVP!RRVRV /VBwppVP ^4V,pVV,pVP'd6VP !RRX /VBpX P ^4V,pVV,pVPV4pV^VR,,,VR,,pVPe-\VPWPPVP4pMVPV4pV P ^4V,pVV,pVP'dRpW!3#X P ^4V,pVV,pVPV4p V ^XR,,,XR,,p VPe-\VPWPVP4pMVPV 4pVXP ^4V,,pW!3#))embr%rNr_rKN)r[rBrYrkr@rrlrCror-r$rprmrn)r:r%rr}r~norm_hidden_statesgate_msa shift_mlp scale_mlpgate_mlpnorm_hidden_states2 gate_msa2r c_gate_msa c_shift_mlp c_scale_mlp c_gate_mlp attn_outputcontext_attn_output attn_output2r,context_ff_outputs&&&&& r#rNJointTransformerBlock.forwards"8!=!=2  " " "kokukulvl h )_hLP::Vc:Kn H )  )-););ryrNrdrerRrSs@@r#rUrUCs3 OOd C4C4C4r&rUcraa]tRtRtoRtR V3RlV3RllltR V3RlRlltR V3RlRlltR tVt V;t #) BasicTransformerBlockiah A basic Transformer block. Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. num_embeds_ada_norm (: obj: `int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`. attention_bias (: obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter. only_cross_attention (`bool`, *optional*): Whether to use only cross-attention layers. In this case two cross attention layers are used. double_self_attention (`bool`, *optional*): Whether to use two self-attention layers. In this case no cross attention layers are used. upcast_attention (`bool`, *optional*): Whether to upcast the attention computation to float32. This is useful for mixed precision training. norm_elementwise_affine (`bool`, *optional*, defaults to `True`): Whether to use learnable elementwise affine parameters for normalization. norm_type (`str`, *optional*, defaults to `"layer_norm"`): The normalization layer to use. Can be `"layer_norm"`, `"ada_norm"` or `"ada_norm_zero"`. final_dropout (`bool` *optional*, defaults to False): Whether to apply a final dropout after the last feed-forward layer. attention_type (`str`, *optional*, defaults to `"default"`): The type of attention to use. Can be `"default"` or `"gated"` or `"gated-text-image"`. positional_embeddings (`str`, *optional*, defaults to `None`): The type of positional embeddings to apply to. num_positional_embeddings (`int`, *optional*, defaults to `None`): The maximum number of positional embeddings to apply. c,<V^8dQhRS[RS[RS[RS[R,RS[RS[R,RS[R S[R S[R S[R S[R S[RS[RS[RS[RS[R,RS[R,RS[R,RS[R,RS[R,RS[RS[/#)rrrWrXrfNr:num_embeds_ada_normattention_biasonly_cross_attentiondouble_self_attentionrnorm_elementwise_affinerdnorm_eps final_dropoutattention_typepositional_embeddingsnum_positional_embeddings-ada_norm_continous_conditioning_embedding_dim ada_norm_bias ff_inner_dimff_biasattention_out_bias)rr rrr)r!r"s"r#r$"BasicTransformerBlock.__annotate__sff f!f f !4Z ff!4Zff#f $ff"&fff !f"#f$ #Tz%f&$':'f(8;Tz)f*Tz+f,Dj-f./f0!1fr&c 8<\SV`4WnW nW0nW@nWPnW`nWnWn Wn VVn VVn Wn VRJ;'dV R8HVnVRJ;'dV R8HVnV R8HVnV R8HVnV R8HVnV R9dVf\'RV RV R24hWnWpnV'dVf \'R 4hVR 8Xd\-VVR 7VnMRVnV R8Xd\1W4VnMRV R8Xd\5W4VnM:V R8Xd\7VVV VVR 4VnM\8P:!WVR 7Vn\=TTTTTV 'dTMRV VR7VnVf V 'duV R8Xd\1W4Vn M9V R8Xd\7VVV VVR 4Vn M\8P:!WV 4Vn \=TV 'gTMRVVVVV VR7Vn!M2V R8Xd\8P:!WV 4Vn MRVn RVn!V R8Xd\7VVV VVR4Vn"M2V R9d\8P:!WV 4Vn"MV R8XdRVn"\GVVVVVVR7Vn$VR8XgVR8Xd\KWW#4Vn&V R8Xd?\8PN!\PPR!^V4VR,, 4Vn*RVn+^Vn,R#)Nr_ada_normada_norm_singleraada_norm_continuous`norm_type` is set to w, but `num_embeds_ada_norm` is not defined. Please make sure to define `num_embeds_ada_norm` if setting `norm_type` to r3\If `positional_embedding` type is defined, `num_positition_embeddings` must also be defined. sinusoidalmax_seq_lengthrms_normrhr1rr7dropoutrrfrout_biasr1rfrr7rrrrlayer_norm_i2vgenrr:r inner_dimrgatedzgated-text-imageg?rr_)r_rra)-r=r>rrWrXrrfr:rrrrrruse_ada_layer_norm_zerouse_ada_layer_normuse_ada_layer_norm_singleuse_layer_normuse_ada_layer_norm_continuousrJrdrr pos_embedrrBrrr-rrattn1rCrlnorm3rAr$r/fuserrEr,rscale_shift_tablerorp)r:rrWrXrrfr:rrrrrrrdrrrrrrrrrrrQs&&&&&&&&&&&&&&&&&&&&&&&&r#r>BasicTransformerBlock.__init__sR4 #6 "4 #6 *,%:"'>$%:")B&$8!)<4(G'i'iYZiMi$#6d#B"_"_ U_H_)26G)G&'<7-6:O-O* 5 5:M:U( 4KKT+UVX  ##6 &?&Gn  !L 0:3OhiDN!DN  "%c?DJ / ))#CDJ / //=' DJc[cdDJ%'7K 3QU-'     *.CJ&)#C 333A+!   \\#9PQ "?T$7Z^)+#!1+ DJ--\\#9PQ ! DJ - -/=' DJE Ec5LMDJ - -DJ ''"   W $:L(L0K^sDJ ) )%'\\%++a2ES2P%QD " r&c4<V^8dQhRS[R,RS[/#rtr)r!r"s"r#r$rrur&cWnW nR#rrwrxs&&&r#ry,BasicTransformerBlock.set_chunk_feed_forwardr{r&cp<V^8dQhRS[PRS[PR,RS[PR,RS[PR,RS[PR,RS[S[S[3,RS[PR,R S[S[S[P3,R,R S[P/ #) rr%rNrencoder_attention_masktimestepcross_attention_kwargs class_labelsadded_cond_kwargsr)r,r LongTensorrr r)r!r"s"r#r$rsxx||x t+x %||d2 x !& t 3 x ""T) x!%S#Xx&&-x U\\ 12T9x xr&c  Ve*VPRR4e\PR4VP^,p VPR8XdVP W4p EMVPR8Xd#VP WWqP R7wrrpMVPR9dVP V4p MVPR8XdVP WR,4p MVPR8XdkVPR,VPV ^R4,P^^R 7wpprrVP V4p V ^V,,V,p M \R 4hVPeVPV 4p VeVP4M/pVPR R4pVP!V 3R VP'dTMRR V/VBpVPR8XdX P!^4V,pMVPR8Xd X V,pVV,pVP"^8XdVP%^4pVeVP'VVR,4pVP(eVPR8XdVP+W4p MlVPR9dVP+V4p MIVPR8XdTp M5VPR8XdVP+WR,4p M \R4hVPe#VPR8wdVPV 4p VP(!V 3R VR V/VBpVV,pVPR8XdVP-WR,4p M"VPR8XgVP-V4p VPR8Xd&V ^X R,,,X R,,p VPR8Xd)VP+V4p V ^X ,,X ,p VP.e-\1VP2WP4VP.4pMVP3V 4pVPR8XdXP!^4V,pMVPR8Xd XV,pVV,pVP"^8XdVP%^4pV#)NrSPassing `scale` to `cross_attention_kwargs` is deprecated. `scale` will be ignored.rr_) hidden_dtyperpooled_text_embrrzIncorrect norm usedgligenrrrIzIncorrect norm)rarr)r_rarr)getrkwarningrrdrBrrrr)rJrcopyrErrrrsqueezerrlrCrror-r$rp)r:r%rrrrrrrrrrrrr shift_msa scale_msa gligen_kwargsrr,s&&&&&&&&& r#rNBasicTransformerBlock.forwardsk " -%))'48Dtu#((+ >>Z '!%M!D  ^^ .KO::DWDWLVL H )^^B B!%M!:  ^^4 4!%MM^;_!`  ^^0 0&&t,x/?/? Ar/RReA1eo KIy(y"&M!: !3q9}!E !Q 23 3 >> %!%0B!C CYBd!7!jl.228TB jj  ;?;T;T;T"7Z^ * %   >>_ ,",,Q/+=K ^^0 0"[0K#m3    ")11!4M  $ JJ}mF6KLM :: !~~+%)ZZ %H"#WW%)ZZ %>"#44&3"#88%)ZZ Qb?c%d" !122~~)dnn@Q.Q%)^^4F%G"**"&; 6) K (-7M >>2 2!%MM^;_!` #44!%M!:  >>_ ,!3q9W;M7M!NQZ[bQc!c  >>. .!%M!: !3q9}!E !Q     '-dgg7I??\`\l\lmI 23I >>_ , **1- 9I ^^0 0 9,I!M1    ")11!4Mr&)rpror:rrXrrlrfrrrr$rrBrCrrrdrWrrrrrrrrrrr)r Nr8NFFFFTrah㈵>FdefaultNNNNNTTr)NNNNNNNrrSs@@r#rrs4 DffP xxxr&rcLaa]tRtRtoRtRV3RlV3RllltRtRtVtV;t #)LuminaFeedForwardi;a A feed-forward layer. Parameters: hidden_size (`int`): The dimensionality of the hidden layers in the model. This parameter determines the width of the model's hidden representations. intermediate_size (`int`): The intermediate dimension of the feedforward layer. multiple_of (`int`, *optional*): Value to ensure hidden dimension is a multiple of this value. ffn_dim_multiplier (float, *optional*): Custom multiplier for hidden dimension. Defaults to None. c N<V^8dQhRS[RS[RS[R,RS[R,/#)rrr multiple_ofNffn_dim_multiplier)rr)r!r"s"r#r$LuminaFeedForward.__annotate__Js; 4Z  "DL r&cZ<\SV`4Ve\WB,4pW2V,^, V,,p\P!VVRR7Vn\P!VVRR7Vn\P!VVRR7Vn\4Vn R#)NFr) r=r>rr-rlinear_1linear_2linear_3rsilu)r:rrrrrQs&&&&&r#r>LuminaFeedForward.__init__Js   ).:;I $;a$?K#OP             J r&cVPVPVPV44VPV4,4#r)rrrr)r:rs&&r#rNLuminaFeedForward.forwardhs1}}TYYt}}Q'784==;KKLLr&)rrrr)NrPrSs@@r#rr;s  <MMr&rcnaa]tRtRtoRtR V3RlV3RllltV3RlRltR V3RlRlltR tVt V;t #) TemporalBasicTransformerBlockila A basic Transformer block for video like data. Parameters: dim (`int`): The number of channels in the input and output. time_mix_inner_dim (`int`): The number of channels for temporal attention. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention. c F<V^8dQhRS[RS[RS[RS[RS[R,/#)rrtime_mix_inner_dimrWrXrfNr)r!r"s"r#r$*TemporalBasicTransformerBlock.__annotate__ysA33 3 3! 3  3 !4Z 3r&c<\SV`4W8HVn\P!V4Vn\ VVRR7Vn\P!V4Vn\VVVRR7Vn Ve1\P!V4Vn \VVVVR7Vn MRVn RVn \P!V4Vn \ VRR7VnRVnRVnR#)r8)rjr:N)r1rr7rf)r1rfrr7r9)r=r>is_resr-rnorm_inrAff_inrBrrrCrlrr$rorp)r:rrrWrXrfrQs&&&&&&r#r>&TemporalBasicTransformerBlock.__init__ys / ||C( ! &!  \\"45 (%' $     *&89DJ",$7)+ DJDJDJ\\"45 0H r&c.<V^8dQhRS[R,/#)rr'Nr)r!r"s"r#r$rstr&c  Wn^VnR#)rxNrw)r:r'kwargss&&,r#ry4TemporalBasicTransformerBlock.set_chunk_feed_forwards%r&c|<V^8dQhRS[PRS[RS[PR,RS[P/#)rr% num_framesrNrr)r!r"s"r#r$rsD77||77 %||d2 7  7r&cVP^,pVPwrVpWR,pVR,PWBWg4pVP^^^^4pVPWF,W'4pTpVPV4pVPe-\ VP WPVP4pMVP V4pVP'd W,pVPV4p VPV RR7p W,pVPe,VPV4p VPWR7p W,pVPV4p VPe-\ VPWPVP4p MVPV 4p VP'd W,pMT pVR,PWFW'4pVP^^^^4pVPWB,Wg4pV#)rN)r)NrK)rrrrror-rrprrBrrlrCrr$) r:r%rrr batch_frames seq_lengthchannelsresidualrrr,s &&&& r#rN%TemporalBasicTransformerBlock.forwards#((+ -:-@-@* (!/ %g.66zzd %--aAq9 %--j.Ez\   ]3    '1$**m__^b^n^noM JJ}5M ;;;)4M!ZZ 6jj!34jP #3  :: !!%M!: **%7*eK'7M"ZZ 6    '-dgg7I??\`\l\lmI 23I ;;;%5M%M%g.66zzd %--aAq9 %--j.Ez\ r&) rprorrlr$rrrBrCrrrrrSs@@r#rrls. 33j 777r&rcHaa]tRtRtoRV3RlV3RllltRtRtVtV;t#)SkipFFTransformerBlockicX<V^8dQhRS[RS[RS[RS[RS[RS[R,RS[R S[/#) rrrWrX kv_input_dimkv_input_dim_proj_use_biasrfNrr)rrr)r!r"s"r#r$#SkipFFTransformerBlock.__annotate__s_( ( ( !(  (  ( %) ( !4Z( ( !( r&c <\S V`4WA8wd\P!WAV4VnMRVn\ VR4Vn\VVVVVVV R7Vn\ VR4Vn \VVVVVVV R7Vn R#)Nr`)r1rr7rrrfr)r1rfrr7rrr) r=r>r-r kv_mapperrrBrrrCrl) r:rrWrXr r rrfrrrQs &&&&&&&&&&r#r>SkipFFTransformerBlock.__init__s   YY|:TUDN!DNS%( %' 3'  S%(  3%''  r&cPVeVP4M/pVPe&VP\P!V44pVP V4pVP !V3RV/VBpWQ,pVP V4pVP!V3RV/VBpWQ,pV#)Nr)rr r rrBrrCrl)r:r%rrrrs&&&& r#rNSkipFFTransformerBlock.forwardsBXBd!7!jl >> %$(NN166:O3P$Q !!ZZ 6jj  "7 % $3 !ZZ 6jj  "7 % $3 r&)rrlr rBrC)r NFT) rRr`rarbr>rNrdrerRrSs@@r#rrs( ( Tr&rcaa]tRtRtoRtRV3RlV3RllltV3RlRltRV3RlRlltRV3R lR lltRV3R lR llt RV3R lRllt Rt Vt V;t #)FreeNoiseTransformerBlocki6a A FreeNoise Transformer block. Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. num_embeds_ada_norm (`int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`. attention_bias (`bool`, defaults to `False`): Configure if the attentions should contain a bias parameter. only_cross_attention (`bool`, defaults to `False`): Whether to use only cross-attention layers. In this case two cross attention layers are used. double_self_attention (`bool`, defaults to `False`): Whether to use two self-attention layers. In this case no cross attention layers are used. upcast_attention (`bool`, defaults to `False`): Whether to upcast the attention computation to float32. This is useful for mixed precision training. norm_elementwise_affine (`bool`, defaults to `True`): Whether to use learnable elementwise affine parameters for normalization. norm_type (`str`, defaults to `"layer_norm"`): The normalization layer to use. Can be `"layer_norm"`, `"ada_norm"` or `"ada_norm_zero"`. final_dropout (`bool` defaults to `False`): Whether to apply a final dropout after the last feed-forward layer. attention_type (`str`, defaults to `"default"`): The type of attention to use. Can be `"default"` or `"gated"` or `"gated-text-image"`. positional_embeddings (`str`, *optional*): The type of positional embeddings to apply to. num_positional_embeddings (`int`, *optional*, defaults to `None`): The maximum number of positional embeddings to apply. ff_inner_dim (`int`, *optional*): Hidden dimension of feed-forward MLP. ff_bias (`bool`, defaults to `True`): Whether or not to use bias in feed-forward MLP. attention_out_bias (`bool`, defaults to `True`): Whether or not to use bias in attention output project layer. context_length (`int`, defaults to `16`): The maximum number of frames that the FreeNoise block processes at once. context_stride (`int`, defaults to `4`): The number of frames to be skipped before starting to process a new batch of `context_length` frames. weighting_scheme (`str`, defaults to `"pyramid"`): The weighting scheme to use for weighting averaging of processed latent frames. As described in the Equation 9. of the [FreeNoise](https://huggingface.co/papers/2310.15169) paper, "pyramid" is the default setting used. c.<V^8dQhRS[RS[RS[RS[RS[R,RS[RS[R,R S[R S[R S[R S[R S[RS[RS[RS[RS[R,RS[R,RS[R,RS[RS[RS[RS[RS[/#)rrrWrXrrfNr:rrrrrrrdrrrrrrrcontext_lengthcontext_strideweighting_schemerrr rr)r!r"s"r#r$&FreeNoiseTransformerBlock.__annotate__nspp p!p p  p !4Z pp!4Zpp#p $pp"&ppp !p" #Tz#p$$':%p&Dj'p()p*!+p,-p./p01pr&c <\SV`4WnW nW0nW@nWPnW`nWnWn Wn VVn VVn Wn VPVVV4VRJ;'dV R8HVnVRJ;'dV R8HVnV R8HVnV R8HVnV R8HVnV R9dVf\)RV RV R24hWnWpnV'dVf \)R 4hVR 8Xd\/VVR 7VnMRVn\2P4!WVR 7Vn\9TTTTTV 'dTMRV VR 7VnVf V 'd?\2P4!WV 4Vn\9TV 'gTMRVVVVV VR7Vn\AVVVVVVR7Vn!\2P4!WV 4Vn"RVn#^Vn$R#)Nr_rrrarrrr3rrrrhrrrr)%r=r>rrWrXrrfr:rrrrrrset_free_noise_propertiesrrrrrrJrdrrrr-rrBrrrCrlrAr$rrorp)r:rrWrXrrfr:rrrrrrrdrrrrrrrrrrrQs&&&&&&&&&&&&&&&&&&&&&&&&r#r>"FreeNoiseTransformerBlock.__init__ns4 #6 "4 #6 *,%:"'>$%:")B&$8! &&~~GWX)<4(G'i'iYZiMi$#6d#B"_"_ U_H_)26G)G&'<7-6:O-O* 5 5:M:U( 4KKT+UVX  ##6 &?&Gn  !L 0:3OhiDN!DN\\#W_` %'7K 3QU-'     *.Cc5LMDJ"?T$7Z^)+#!1+ DJ ''"  \\#1HI  r&cL<V^8dQhRS[RS[S[S[S[3,,/#)rrr)rlistr)r!r"s"r#r$rs(ST%S/5Jr&c.p\^WP, ^,VP4F3pTp\WVP,4pVP WE34K5 V#r)rangerrminappend)r:r frame_indicesi window_start window_ends&& r#_get_frame_indices,FreeNoiseTransformerBlock._get_frame_indicessb q*':'::Q>@S@STALZT-@-@)@AJ  ,!; <Ur&c<<V^8dQhRS[RS[RS[S[,/#)rrrr)rr rr)r!r"s"r#r$rs)SCX\]bXcr&cVR8Xd R.V,pV#VR8XdV^,^8Xd:V^,p\\^V^,44pW3RRR1,,pV#V^,^,p\\^V44pW4.,VRRR1,,pV#VR8XdV^,^8XdBV^,pR.V^, ,V.,pV\\V^R44,pV#V^,^,pR.V,pV\\V^R44,pV#\RV 24h)flatg?pyramidNdelayed_reverse_sawtoothg{Gz?z'Unsupported value for weighting_scheme=r)rrrJ)r:rrweightsmids&&& r#_get_frame_weights,FreeNoiseTransformerBlock._get_frame_weightssL v %ej(G85 *A~" AouQa01!DbDM1*%"A~!+uQ}-!E/GDbDM9 !; ;A~" Ao&C!G,u4!DsAr):$;; "A~!+&3,!DsAr):$;;FGWFXYZ Zr&c0<V^8dQhRS[RS[RS[RR/#)rrrrrN)rr )r!r"s"r#r$rs-11!1361JM1 1r&c*WnW nW0nR#r)rrr)r:rrrs&&&&r#r3FreeNoiseTransformerBlock.set_free_noise_propertiess-, 0r&c8<V^8dQhRS[R,RS[RR/#)rr'Nrrr)r!r"s"r#r$rs&t#dr&cWnW nR#rrwrxs&&&r#ry0FreeNoiseTransformerBlock.set_chunk_feed_forwardr{r&c <V^8dQhRS[PRS[PR,RS[PR,RS[PR,RS[S[S[3,RS[P/#)rr%rNrrrr)r,rrr r)r!r"s"r#r$rsz{{||{ t+{ %||d2 { !& t 3 { !%S#X { {r&c l Ve*VPRR4e\PR4VeVP4M/pVPpVP p VP ^4p VPV 4p VPVPVP4p \P!WV R7P^4PR 4p V R ,^,V 8Hp V 'ggWP8d\RV : RVP: 24hWR ,^,, pV PWP, V 34\P !^V ^3VR7p\P"!V4p\%V 4EFwpwpp\P&!VRVV13,4pVV ,pVRVV13,pVP)V4pVP*eVP+V4pVP,!V3RVP.'dTMRR V/VBpVV,pVP0^8XdVP3^4pVP4ebVP7V4pVP*e#VP8R 8wdVP+V4pVP4!V3RVR V/VBpVV,pV\;V 4^, 8XdiV 'gaVRX)R13;;,VRV)R13,VRV)R13,,, uu&VRV)R13;;,VRV)3,, uu&EKVRVV13;;,VV,, uu&VRVV13;;,V, uu&EK \P<!\?VPAVP^R 7VPAVP^R 74UUu.uF(wpp\PB!V^8VV, V4NK* upp^R 7PEV 4pVPGV4pVPHe.\KVPLVVPNVPH4pMVPMV4pVV,pVP0^8XdVP3^4pV#uuppi) NrrrzExpected num_frames=z1 to be greater or equal than self.context_length=)rrKrrrrr)(rrkrrrrsizer&r/rrr,rrrJr!r  zeros_like enumerate ones_likerBrrrrrrlrCrdrHrzipsplitwhererrror-r$rp)r:r%rrrrargsrrrrr" frame_weightsis_last_frame_batch_completelast_frame_batch_lengthnum_times_accumulatedaccumulated_valuesr# frame_start frame_endr-hidden_states_chunkrraccumulated_splitnum_times_splitr,s&&&&&&*, r#rN!FreeNoiseTransformerBlock.forwards " -%))'48DtuBXBd!7!jl%%##"''* // ; //0C0CTEZEZ[  ]OYYZ[\ffgij '4R'8';z'I$ ,/// #8ZM9kW[WjWjVl!mnn&03DQ3G&G #  */B/B"BJ!O P % Q A,>v N"--m<+4]+C 'A' Yoo&;A{9?T>-$..DU2U)-8J)K&"jj&*?$:-  '24G&G#C &**3O"1'>&>&?#?@',C+C+D(DEPQTkSkSlPlHmm@&a*A)A)B&BCwqSjRjOjGkkC"1k)&;#;<@SV]@]]<%aY)>&>?7J?c,D| ;>&,,T-@-@a,H)//0C0C/K; ;6% Oa/1B_1TVgh;    "U) "ZZ 6    '-dgg7I4??\`\l\lmI 23I!M1    ")11!4M- s.R0 )rpror:rrXrrlrrrfrrrr$rBrCrrrdrWrrrrrrrrrrr)r Nr8NFFFFTrarFNNNTTr+)r+r)NNNN)rRr`rarbrQr>r&r/rryrNrdrerRrSs@@r#rr6sS4lppd@11 {{{r&rcXaa]tRtRtoRtRV3RlV3RllltV3RlRltRtVtV;t #) rAia A feed-forward layer. Parameters: dim (`int`): The number of channels in the input. dim_out (`int`, *optional*): The number of channels in the output. If not given, defaults to `dim`. mult (`int`, *optional*, defaults to 4): The multiplier to use for the hidden dimension. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. final_dropout (`bool` *optional*, defaults to False): Apply a final dropout. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. cR<V^8dQhRS[RS[R,RS[RS[RS[RS[RS[/#) rrrjNmultrr:rrr)r!r"s"r#r$FeedForward.__annotate__sU&1&1 &1t&1 &1  &1  &1&1&1r&c <\S V`4Vf\W,4pVeTMTpVR8Xd\WVR7p VR8Xd\WRVR7p MTVR8Xd\ WVR7p M?VR8Xd\ WVR7p M*VR8Xd\ WVR7p MVR 8Xd\WVR R 7p \P!.4Vn VPPX 4VPP\P!V44VPP\P!WrVR74V'd2VPP\P!V44R#R#) NgelurrirL) approximaterr8zgeglu-approximateswigluz linear-silur)r activation)r=r>rr r r rrr- ModuleListnetr!Dropoutr) r:rrjrPrr:rrract_fnrQs &&&&&&&&& r#r>FeedForward.__init__s    CJI$0'c F "#t4F . .#f4HF g %35F 1 1$S$?F h &C6F m +%c4FSF==$   7+,  )4@A  HHOOBJJw/ 0 r&cN<V^8dQhRS[PRS[P/#)rr%rr)r!r"s"r#r$rQs#U\\u||r&c\V4^8gVPRR4eRp\RRV4VPF pV!V4pK V#)rrNzThe `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.z1.0.0)rHrrrX)r:r%r@rdeprecation_messager*s&&*, r#rNFeedForward.forwardsQ t9q=FJJw5A#U  gw(; <hhF"=1Mr&)rX)NrMr r8FNTrPrSs@@r#rArAs$ &1&1Pr&rA)6typingrrr,torch.nnr-torch.nn.functional functionalr utilsrrutils.import_utilsrrr utils.torch_utilsr activationsr r r rrrattention_processorrrr embeddingsr normalizationrrrrrrr get_loggerrRrkrrTr-r.r/rUrrrrrrAr_r&r#rls]! &ff4YYUU5qq D   H %O,O,dN%N%b &bii&&Rh4BIIh4h4VHBIIHHV .M .Mb~BII~~BERYYEPX XXv <"))<r&