# Add additional conditioning info
**URL:**
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [January 23, 2023, 2:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F1 "2023-01-23T02:25:38Z")
Hi All,
Does anybody have any guidance as to how/where to add further conditioning info to the HF stable diffusion training/inference pipelines? Everything Iāve read about stable diffusion seems to suggest that multiple different types of conditioning should be possible, but Iām not sure how to integrate it. Since the text embeddings are integrated using self-attention I feel like it should probably be added there, but how? Would I concatenate it to the text embeddings, for example?
Any thoughts appreciated.
---
**Author:**  [@pcuenq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpcuenq)%5C
**Post date:** [January 24, 2023, 11:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F2 "2023-01-24T11:12:21Z")
Hi @jbmaxwell! Thatās an excellent question.
The easiest way, I think, would be to leverage the `UNet2DConditionModel` and indicate [here](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/unet_2d_condition.py#L123) that youāll be using custom class embeddings. Similar to what you suspected, these embeddings are simply [added to the timestep embeddings](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/unet_2d_condition.py#L88-L89). If you use the `"timestep"` `class_embed_type`, for example, then you need to [pass your custom class labels](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/unet_2d_condition.py#L398) during the `forward` pass and then those values are [passed through an embedding layer and added to the timestep embeddings](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/unet_2d_condition.py#L464-L472).
I hope thatās enough to get you started! Please, do share if it works as well as what you are trying to achieve (if you can make it public).
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [January 24, 2023, 4:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F3 "2023-01-24T16:22:31Z")
Excellent, thanks so much @pcuenq!
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [February 1, 2023, 1:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F4 "2023-02-01T01:29:01Z")
Okay, Iāve got a bit furtherā¦
Iāve trained a VQ-VAE to generate my conditioning embeddings, but Iām wondering whether I can/should pass the (integer) latent code straight in as my ācustom class labelsā, or if I should/must normalize them first? If I normalize them, is it (0,1), or (-1, 1), orā¦ ? š
Any help appreciated.
āOh!.. Also, this tensor contains duplicates. Should I remove duplicates? (My concern here is that it will change the shapeā¦)
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [February 1, 2023, 4:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F5 "2023-02-01T16:08:40Z")
Hi @pcuenq, Iāve just come back to this to work on today and I think your links above have changed/movedāi.e., the code was maybe updated so they no longer point to the right lines. Just an fyi since the answer might be a bit confusing for future readers (I went through it the other day, so not a huge deal right away). Not sure if thereās a way to avoid this in futureā¦ ?
---
**Author:**  [@pcuenq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpcuenq)%5C
**Post date:** [February 2, 2023, 7:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F6 "2023-02-02T07:57:34Z")
Hi @jbmaxwell!
You are right, I should have used a tag instead of `main`. Sorry about that.
Since we last talked weāve added optional class conditioning to `UNet2DModel`, in addition to what was available in `UNet2DConditionModel`. The difference is that `UNet2DModel` is simpler because it doesnāt use text conditioning (for text to image generation). So if you donāt need to train your model for text to image tasks, you can use `UNet2DModel` instead and training should be faster. [This is the revision where that feature was added](https://github.com/huggingface/diffusers/pull/2080/files) ā and itās from the PR so it should outlive future changes in `main` :). Youād use it the same way we discussed:
- You select a class-conditioning embedding type when you create the UNet.
- You pass your custom class labels in the forward pass.
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [February 2, 2023, 11:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F7 "2023-02-02T23:14:47Z")
This is great, thanks. I will be using both text and this new conditioning info (which Iāll pass via the class-conditioning mechanism), so Iāll stick with `UNet2DConditionModel`ā¦ But itās cool that `UNet2DModel` has the option for class-conditioning now, so thanks for the heads-up!
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [February 9, 2023, 8:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F8 "2023-02-09T20:54:05Z")
Hi again, @pcuenq.
I think I managed to run some training with my additional conditioning info, and now Iām trying to test inference. Is there a straightforward way to use the āclass labelsā during inferenceāi.e., in one of the pipelines? I didnāt see anything obvious, so Iāve been working on an adaptation of StableDiffusionPipeline to do itā¦ But It thought Iād ask, in case thereās something simpler I can make use of.
Thanks!
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [February 10, 2023, 1:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F9 "2023-02-10T01:11:17Z")
Unfortunately, it seems like thereās a significant missing piece here.
I thought I had trained on my data, with the class embeddings, but I donāt think I did. Stepping through the code, it looks like the class embeddings will be silently skipped if `class_embed_type` isnāt set (yes, you did mention this), but trying to set it manually I crash with the following error:
```auto
File "/home/james/anaconda3/envs/riffusion/lib/python3.9/site-packages/torch/nn/modules/module.py", line 987, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!
```
I tried by both setting the class embedding type in the `config.json` and adding it when I instantiate the unet, as an argument to `from_pretrained()`, but Iām guessing maybe it fails because there are no weights in the `diffusion_pytorch_model.bin` for the class embeddings, so it canāt instantiate it.
So perhaps Iām forced to train from scratchā¦ which is actually fine, but how do I do that???
* * *
Okay, I think I worked out a way to get started:
```auto
unet = UNet2DConditionModel(class_embed_type='timestep')
```
And I have a feeling this works, because I run out of CUDA memory when trying to process it with my embedding! š¤£
(Fortunately I now have access to a bigger GPU, so Iāll give it a try on thatā¦)
But please let me know if thereās another (or a better) way!
* * *
Another update. I had mistakenly assumed the unet was using the default values; adding the non-default values (from `config.json`) to the init got me further:
```auto
unet = UNet2DConditionModel(sample_size=64, cross_attention_dim=768, class_embed_type='timestep')
```
However, Iām running into problems with shapes when using the `timestep` type. Iāve been able to at least get the model training by using `identity`, then adding a block in the unetās `forward` to adjust the shape of my custom conditioning embedding, like so:
```auto
class_emb = self.class_embedding(class_labels).to(dtype=self.dtype)
if not class_emb.shape == emb.shape:
emb_len = emb.nelement()
cl_emb_len = class_emb.nelement()
if cl_emb_len > emb_len:
# here we can only truncate
class_emb = class_emb[:emb_len]
else:
# here we can repeat, pad, and reshape to match emb
cl_emb_repeat = emb_len // cl_emb_len
cl_em_pad_len = emb_len - (cl_emb_repeat * cl_emb_len)
cl_em_pad = torch.zeros(cl_em_pad_len).to(emb.device)
class_emb = class_emb.repeat(cl_emb_repeat)
class_emb = torch.cat((class_emb, cl_em_pad), 0)
class_emb = class_emb.reshape(emb.shape)
emb = emb + class_emb
```
This at least allows me to use the `class_labels` argument to pass in my (non-class) custom conditioning embedding. If this is clearly a bad idea, any help would be greatly appreciated.
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [February 12, 2023, 6:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F10 "2023-02-12T18:46:16Z")
Okay, some real progress!
I trained a model with this type of conditioning and it does seem to be working. However, although itās difficult to say for certain, I seem to be getting less influence from my custom conditioning that I would like. Basically, the text seems to have much more impact than my conditioning, and Iām wondering how to balance things out.
One thing Iād thought of was to move my conditioning from being added to the time embedding, `emb`, to being added to the text embedding, `encoder_hidden_states`, perhaps adding a parameter to adjust the āmixā of the two. I may try this anyway, but if anybody has any thoughts, please share.
On that note, @pcuenq, I realize Iām not really clear on the roles/functions of the time embedding and the text embedding. Intuitively, it seems to me that the time embedding is related to the basic task of generating _anything_, and impacts directly on the denoising process, whereas the text embedding is an additional feature used to kind of āfocusā the generation in the latent space. Is that roughly correct?
---
**Author:**  [@pcuenq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpcuenq)%5C
**Post date:** [February 13, 2023, 11:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F11 "2023-02-13T11:05:15Z")
Hi @jbmaxwell! Congrats on making progress on this task!
I think your intuition is correct. The time embeddings provide a hint to the model about the step in the (de)noising process we are. Because timesteps are semantically related to one another (they follow a progression, so `4` is a time instance larger than `3` but smaller than `5`), they are encoded using a fancy method that tries to preserve that relationship - those are the sinusoidal embeddings that youād probably have seen in the code.
Depending on the nature of your additional conditioning, you may not need to capture a similar relationship on your data, and thatās probably why you didnāt see great results when using the `timestep` conditioning type, which applies the same sinusoidal method to your custom conditioning data.
For example, if you were training a model to generate 5 different classes of objects, the numerical representations of those 5 categories do not bear any relationship with one another. In this case, you might want to explore the `None` `class_embed_type`, but indicate that your `num_class_embeds` is `5`. (`None` is probably not a good choice for this use-case, as it appears that only `timestep` or `identity` are supported, but itās actually a _third_ choice you can use). If you use this method, your model will learn to differentiate about those 5 categories, and then you can request to generate one of your desired subjects by supplying the class information at inference time.
Let us know if thatās something that sounds useful for your project! š
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [February 13, 2023, 10:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F12 "2023-02-13T22:11:56Z")
Thanks for the info. Very helpful!
---
**Author:**  [@linpang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flinpang)%5C
**Post date:** [April 25, 2023, 11:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F13 "2023-04-25T23:55:48Z")
Hi, have you successfully made adding conditional embedding working ? if it works, do you mind to share the script? thank you.
---
**Author:**  [@linpang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flinpang)%5C
**Post date:** [April 26, 2023, 12:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F14 "2023-04-26T00:09:43Z")
Hi, thanks for all of these discussions. I have one question: for the conditional text embedding, can I replace it as image embedding ( for instance, I would like to replace image A to the part of image B which is already generated without text input. ) Hope my question is clear.
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [April 26, 2023, 12:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F15 "2023-04-26T00:28:33Z")
I did get a version of this to āworkā, but the effect was pretty subtle. It did seem to do _something_, but not what I was after, and the result was overwhelmingly dominated by the text promptā¦ I donāt think I have the code for that anymore, as I re-wrote that script with a version that added to the text embeddingāwhich was spectacularly bad, so I abandoned the effort. š
You should have a look into ControlNet for what it sounds like youāre trying to do. I think thereās a ton of room for experimenting with different types of conditioning using that approach.
---
**Author:**  [@linpang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flinpang)%5C
**Post date:** [April 26, 2023, 7:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F16 "2023-04-26T19:39:19Z")
Thank, I will read more and ask again if I have any more questions.
---
**Author:**  [@barry556652](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbarry556652)%5C
**Post date:** [May 16, 2023, 1:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F17 "2023-05-16T13:30:19Z")
Hello, I also have four different classes that I want to train. Here, my `num_class_embedds` is set to 4 and `class_embed_type` is set to None. However, Iām having trouble writing the `class_labels` , which is causing an error in the line `hidden_states = hidden_states + temb` . Can you please tell me how to create the `class_labels` ?
This is my class\_labels code
def class\_label\_tensor(examples, is\_train=True):
```
def class_tokenizer(text):
class_names = [['C0201'], ['R0201'], ['L2016'], ['F1210']]
class_label = text
num_classes = len(class_names)
class_vector = torch.zeros(num_classes, dtype=torch.int)
class_index = class_names.index(class_label)
class_vector[class_index] = 1
class_tensor = class_vector.view(1, num_classes)
return class_tensor
captions = []
for caption in examples[caption_column]:
if isinstance(caption, str):
captions.append(caption)
elif isinstance(caption, (list, np.ndarray)):
# take a random caption if there are multiple
captions.append(random.choice(caption) if is_train else caption[0])
else:
raise ValueError(
f"Caption column `{caption_column}` should contain either strings or lists of strings."
)
label_tensor = class_tokenizer(captions)
return label_tensor
```
I always get **RuntimeError: The size of tensor a (64) must match the size of tensor b (320) at non-singleton dimension 4**in my case.
Thx!
---
**Author:**  [@Meghnad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMeghnad)%5C
**Post date:** [September 17, 2023, 3:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F18 "2023-09-17T15:29:34Z")
@pcuenq I am trying to make an EEG to Image model, my EEG encoder is a separate model and I intend to use Stable Diffusion without text conditioning, the idea is Iāll map the EEGs to their corresponding images. Would you please guide me in this regard, where and how do I attach this encoder model?
---
**Author:**  [@maliozer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaliozer)%5C
**Post date:** [February 16, 2024, 12:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F19 "2024-02-16T00:22:09Z")
how about added\_cond\_kwargs , can we pass the embeddings we have to make another condition here what do you think ?
>
@pcuenq
---
**Author:**  [@reesekneeland](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Freesekneeland)%5C
**Post date:** [June 20, 2024, 7:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195%2F20 "2024-06-20T19:29:07Z")
Hello, Iām curious if you ever made progress on this idea? I am looking to tackle a similar idea for fMRI, where I will train a new encoder (brain ā embedding) end to end with the diffusion model that I am fine tuning to reconstruct the original image with my conditioning info. Let me know if you have any insights on this front.
[Next page](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-additional-conditioning-info%2F30195.md%3Fpage%3D2)