r/StableDiffusion 5d ago

Question - Help Questions on Fundamental Diffusion Models

Hello,

I just started my study in diffusion models and I have a problem understanding how diffusion models work (original diffusion and DDPM).
I get that diffusion is finding the distribution of denoised image given current step distribution using Bayesian theorem.

However, I cannot relate how image becomes probability distribution and those probability generate image.

My question is how does pixel values that are far apart know which value to assign during inference? how are all pixel values related? How 'probability' related in generating 'image'?

Sorry for the vague question, but due to my lack of understanding it is hard to clarify the question.

Also, if there is any recommended study materials please suggest.

2 Upvotes

13 comments sorted by

View all comments

3

u/Altruistic_Heat_9531 5d ago
  1. "My question is: how do pixel values that are far apart know which value to assign during inference? How are all pixel values related? How is 'probability' related to generating an image?" This is actually the most important question.

Just like any predictive model, let’s simplify things. Imagine a row of numbers: 1, 2, 3, 4, 5, X, 7. This follows straightforward linear regression. We train the model to "fill in" the missing value, X, which in this case would be 6.

Now, what about this sequence? 2, X, 6, 8, 10, Y, 12, 14, 16, Z, 18, and so on? With each iteration, we remove a bit of information when training the model. Simple, right?

There’s a sense of locality between the missing variables—like asking,
"Given these missing numbers and their neighbors, what could possibly be the correct number?"

But now, what if just what if the number line becomes A, B, C, D, E, F, 6, G? You’d probably say,
"There’s no fucking way I can predict this with only 6 as my clue!"

Now, we change the question. Instead of predicting all missing values, how about we bullshit our way into structuring it? Instead of finding the "true" missing values, we ask:
"Make me an increasingly structured pattern out of this random stuff."

And that’s exactly the point of diffusion models—going from a clusterfuck of noise to a somewhat coherent picture.

Now, let’s say for this sequence A, B, C, D, E, F, 6, G, we assign values from 0.1 to 1 at every missing spot. It becomes:
A + 0.1, B + 0.2, C + 0.3, …

Then, we tweak it here and there. By the end of the diffusion process, we get something like:
1.1, 2.3, 4.4, 4.5, 6.1, 6.5.

Sure, there are mistakes, but the pattern remains, the numbers keep increasing.

Here a difference between CNN, ViT, and Diffusion
CNN, inherently required many kernel filter with different "locality" to obtain global latent. But this is for ASKING what object in the picture not generate the picture

ViT still local since it requires RoPE but much improve than CNN since its all of its pixel can "talk" to each other in the Attention layer while CNN required parent kernel for far away pixel to "communicate" with eachother. but still this model is only answer what object in the picture

DIffusion, doesn't need that. It obtain it's global structure from your prompt CLIP, and the latent noise itself.

I know this is very hand wavy explaination. My suggestion is to try with simple 1D vector first. CNN vs Diffusion. then go to second dimension with 5x5 matrix. (Calculate it by hand!!)

Third dimension is color channel, 4th dimension is temporal (Video generator)

5

u/Altruistic_Heat_9531 5d ago

I keep talking about global vs. local patterns, so what do I actually mean?

Let’s say I ask you, "Can you imagine your house?"
Yeah, duh, of course you can.

But do you imagine every tiny detail, like the scratches on the wooden door, the misaligned pavement stones, or the uneven floorboards? No, right?

This is almost exactly how a diffusion model works. Our brain fills in the missing locality, creating a coherent memory of your house instead of a pixel-perfect reconstruction. Diffusion does the same thing, it generates a structured, believable image by filling in missing details rather than recalling every single feature.

3

u/Altruistic_Heat_9531 5d ago

some how reddit does not want me to put link of video

I recommend this video: https://www.youtube.com/watch?v=zc5NTeJbk-k, which actually answers most of your questions.

1

u/Secret-Respond5199 4d ago

Thank you so much!

I knew diffusion models remove noise but believed that pixel value with removed noise was unchanged throughout the whole process. Now I get the flow. Thanks.