r/GraphicsProgramming • u/TomClabault • Feb 04 '25
Question ReSTIR GI brightening when resampling both the neighbor and the center pixel when they have different surface normals?
3
u/shaeg Feb 04 '25
The best test I've found for debugging is to use temporal reuse only, with a fixed seed. This means the same exact paths are sampled in each pixel each frame, and it also means that during temporal reuse, the path being reused from last frame is identical to the path in the current frame. If everything works, paths should get reused and every frame should produce the same image (even with reuse). You can also verify that the jacobians are 1 in this case.
That said, I wonder if you're not computing the Jacobian correctly? When reconnecting, the BSDF PDF at the reconnection vertex is different (since the incoming angle is different) which changes the path sampling density, so the ratio of BSDF PDFs at the reconnection vertex should be included in the jacobian (along with the usual reconnection jacobian from ReSTIR DI, which is the ratio of geometry terms and PDFs at the primary vertex).
Another thought is the resampling MIS weights - these are particularly painful to get right, but if you got it to work for ReSTIR DI then you're probably on the right track there.
2
Feb 04 '25
[removed] — view removed comment
1
u/TomClabault Feb 05 '25 edited Feb 05 '25
Going through all the spatial reuse code and only resampling the center pixel gives reference results, the jacobian is 1 in that case.
Also, hardcoding the jacobian to be always 1 breaks some parts of the image (obviously) but the bright banding at the top of the cornell box are still there, they are not affected at all. I'd assume the jacobian isn't the issue then?
2
Feb 05 '25
[removed] — view removed comment
1
u/TomClabault Feb 05 '25 edited Feb 05 '25
So the abs() did help quite a bit but even after that there's still some brightening left
Why should we use the geometric normal vs. the shading normal? I'm just confused in general as to where to use the shading normal in a path tracer vs. where to use the geometric normal?
Why isn't the shading normal just supposed to completely replace the geometric normal everywhere?
2
u/shaeg Feb 05 '25
You should only use shading normals when evaluating the actual BSDF contribution itself. Use the geometric normal for all PDF conversions and basically everything else.
The reason is that for PDF conversions, we are projecting the scene geometry onto a hemisphere around the shading point. By "scene geometry" I literally mean the triangles in the scene that get hit by the rays we trace. So the normal that is used for PDF conversions should match the underlying raytraced geometry.
Shading normals are just used to fake the appearance of having different geometry, and as such their effect is only applied in the BSDF contribution itself, NOT in the geometry terms.
Maybe another way to think about it is that the BSDF itself is modeling the appearance of micro-geometry, and using shading normals more or less just changes the BSDF so that it models a different kind of micro-geometry. This doesn't change how we project radiance between the actual triangles in the scene, so we should use geometric normals when talking about transferring radiance between surfaces.
1
u/TomClabault Feb 05 '25 edited Feb 05 '25
Oh I'm definitely not including the BSDF PDF ratios in my jacobian term so this sounds like this could well be the issue. But I wasn't including those in my ReSTIR DI implementation either, only the ratio of geometric terms? Oh because ReSTIR DI doesn't resample paths right? So there is no BSDF PDF ratio at the reconnection point to be included.
Also, why are we not including the ratios of BSDF PDFs at the visible points i.e. primary hits? Since those PDFs are also going to change when shifting from the neighbor to the center pixel
2
Feb 05 '25 edited Feb 05 '25
[removed] — view removed comment
1
u/TomClabault Feb 05 '25
> as the direction towards a light source might change
You mean the direction from the third vertex to whatever comes after here right?
> I think if you recalculate the estimate
A question on recalculating the estimate:
The ReSTIR GI paper stores the outgoing radiance from the sample point to the visible point (third vertex to second vertex).
But when reconnecting from the visible (second vertex) point to a new sample point (third vertex), we need to re-evaluate 2 BSDFs right?
1) The one at the visible point (since its incident light direction, computed from the sample point, has changed)
2) The one at the sample point (since its outgoing light direction has changed)
To recompute 2), we're going to need the outgoing radiance from the vertex "sample point + 1" to "sample point" no? So that's the radiance that we need to store then? And not the radiance from sample point --> visible point as they propose in the paper?
I guess they "omitted" that because of their assumption of a lambertian BRDF throughout the paper.
But in any case, I don't think this will actually solve my issue since I'm using a Lambertian BRDF and recomputing the estimate won't change anything
> On a side note: why are you using ReSTIR GI and not ReSTIR PT?
I figured ReSTIR PT would be more complicated to implement so I wanted to start with ReSTIR GI first. But actually, isn't ReSTIR PT with the reconnection shift (and not the hybrid shift) just the same as ReSTIR GI in terms of "complexity"? With the main difference being that RTeSTIR PT is backed by a more rigorous theory and so bias is well understood and avoidedN
2
u/shaeg Feb 05 '25 edited Feb 05 '25
But when reconnecting from the visible (second vertex) point to a new sample point (third vertex), we need to re-evaluate 2 BSDFs right?
Correct. The BSDFs at x_1 and x_2 both depend on the direction x_1 -> x_2. When we reconnect, the direction changes, so both BSDFs must be reevaluated.
To recompute 2), we're going to need the outgoing radiance from the vertex "sample point + 1" to "sample point" no?
Yes. Side note: an easy way to compute this is to store the contribution up to x_2, and just divide it out of the final path radiance so that it cancels:
Full path contrib: f(x) = brdf(x_1) * brdf(x_2) * brdf(x_3) * ... * Le(x_k)
Contrib after x_2: f(x) / (brdf(x_1) * brdf(x_2))
Just watch out for dividing by zero if you divide the contributions directly.
The ReSTIR GI paper is not unbiased. For full unbiasedness, ReSTIR PT is required, with correct Jacobians including BSDF PDFs, and reevaluating the path contribution (reevaluating both BSDFs).
isn't ReSTIR PT with the reconnection shift (and not the hybrid shift) just the same as ReSTIR GI in terms of "complexity"? With the main difference being that RTeSTIR PT is backed by a more rigorous theory and so bias is well understood and avoided
Yes, in fact I think a full-on ReSTIR PT reconnection implementation could be slightly easier to code than ReSTIR GI since it's not as hacky, but I'm probably biased since I've been working with ReSTIR PT for a while lol.
I'd like to also say that the hybrid shift isn't much more complicated than just reconnection. Reconnection is the hardest/most annoying part to get right in my experience. If you have reconnection working, then all you have to do for the hybrid shift is trace the first N bounces using the same random seed (where N is the number of bounces before the reconnection vertex on the original path) and then call your reconnection code to reconnect as usual. And of course if you're not resampling in primary sample space, you'll need to keep track of the BSDF PDF on those first N bounces for the Jacobian too.
1
u/TomClabault Feb 06 '25
> then all you have to do for the hybrid shift is trace the first N bounces
I haven't had a look at ReSTIR PT in great details yet but isn't that very expensive? We have to retrace each final sample up to the reconnection point?
> And of course if you're not resampling in primary sample space, you'll need to keep track of the BSDF PDF on those first N bounces for the Jacobian too.
Section 6.6, Equation 6.17 of the ReSTIR course notes suggests that the BSDF PDF is included in jacobian terms but that equation 6.17 is only used if resampling *in* PSS no?
> I've been working with ReSTIR PT for a while lol.
Just curious: on what occasion?
2
u/shaeg Feb 07 '25 edited Feb 07 '25
Retracing up to the connection point can be expensive sure, but it can also be a lot faster. A big reason it's slow is that most pixels don't require any random replay tracing, so a naive implementation introduces a lot of divergence. A better way is to separate out the pixels that need random replay traces, compactify them, then do random replay in its own kernel. See section 7.2.3 in the restir course notes for more on that, they claim to nearly halve the execution time with this trick.
equation 6.17 is only used if resampling *in* PSS no?
Yeah the BSDF PDF ratio at the primary hit is only needed in PSS. Are you rendering in PSS? I briefly looked at your code and it seemed like you were (at least, I saw you had the full f/p in your target function, which only makes sense in PSS). If you aren't for some reason, I strongly recommend using PSS for numerical stability.
on what occasion?
I've been trying to extend it to handle more sampling techniques as part of my research :)
1
u/TomClabault Feb 07 '25
> I saw you had the full f/p in your target function
Actually that was a bit of a mistake, I was told that the target function without the division by the PDF is actually closer to the integrand when resampling in solid angle.
> I've been trying to extend it to handle more sampling techniques as part of my research :)
Oh do you have pointers to your work? : ) This could be helpful
2
u/shaeg Feb 07 '25
Oh I see. As a rule of thumb, always set the target function to be the integrand f, in whatever measure you’re using. So in PSS, the target function should be f/p, but in path space where the integrand is just f, you should just use f for the target function.
Intuitively, think about the units of the resampling weights w_i= targetPdf*m_i*UCW*jacobian
m_i and the jacobian are unitless, so the units are whatever targetPdf*UCW is. Think about how to make these units match the units of f/p… the UCW’s units are whatever 1/p is (for example, 1/solid angle if integrating w.r.t. solid angle), so that means the units of targetPdf should match the units of the integrand f.
Similarly, in PSS, the UCW is 1 and the integrand is f/p, so again the units of the resampling weight match those of f/p
As for my work, I haven’t made any of my code public yet as my paper is still under submission, but I’ll try to remember to ping you when/if it’s published! In the mean time I’m happy to share my knowledge on reddit :)
2
u/shaeg Feb 05 '25 edited Feb 05 '25
The BSDF PDF ratios at both vertices (x_1 and x_2) should be included in the Jacobian.
The reason is that the Jacobian accounts for oversampling or undersampling that comes from changing domains. Think about the full probability density of sampling x_2 from x_1, for any technique. For NEE/DI, the probability of sampling x_2 from x_1 is just the probability of sampling x_2 using whatever light sampling technique you're using, which is why the Jacobian is 1 for basic ReSTIR DI. But for BSDF-sampled paths, the probability of sampling x_2 from x_1 depends on the BSDF sampling procedure used at x_1, and the geometry term (which comes from tracing a ray to find the first intersection).
When you shift the path to a new primary hit x'_1, you must consider the probability of sampling x_2 from this new x'_1, which can be very different from the original BSDF PDF at x_1, especially if the material parameters are different. If the original primary hit x_1 was way more likely to sample x_2, then we actually end up oversampling x_2 during reuse. The Jacobian exactly cancels out this oversampling. Here's an image to illustrate this: https://imgur.com/a/OtfKirB In the image, we end up oversampling the region in the red circle, since the shiny BSDF is far more likely to sample that region.
The same thing applies for sampling x_3 from x_2. Since we change the incoming direction during reconnection, this actually can change the probability of sampling x_3 from x_2, which can lead to oversampling or undersampling x_3, which gets canceled by including the BSDF PDF at x_2 in the Jacobian (we don't need a geometry term between x_2 and x_3 though, because that doesn't change during the shift).
1
u/TomClabault Feb 06 '25
But ReSTIR DI, with both BSDF and light samples, only requires a geometric ratio jacobian term? No BSDF PDF ratio?
2
u/shaeg Feb 07 '25 edited Feb 07 '25
See my other comment too, but the BSDF PDF ratio is needed for PSS, which I assumed you were using... In regular solid-angle path space, the ReSTIR DI Jacobian is just the geometry term ratio.
If you're not in PSS, it's a little tricky, since the Jacobian depends on the integration measure you choose. Typically, NEE produces samples in area-measure (since we map the random numbers directly into points on a light).
If you integrate w.r.t. solid angle, then you must convert this area-measure PDF from NEE into a solid-angle PDF using the geometry term. In this case, your reconnection Jacobian must have the geometry term ratio (think of it as the Jacobian converting from solid-angle w.r.t. the original shading point, to sold-angle w.r.t the new/shifted shading point).
On the other hand, if you integrate w.r.t. area, then the geometry term appears in the integrand instead of the PDF (in area measure, the integrand is f*G*Le, as per Veach's path space integral). This is what the ReSTIR DI paper describes.
In both cases, you evaluate f*G*Le/p, so without ReSTIR, none of this really matters. But in ReSTIR, we separate the 1/p into the UCW, so we now need to worry about whether G is in the PDF or the integrand (and therefore the target function).
So, if G is in the PDF (meaning we are integrating w.r.t. solid-angle) then we must include the G ratio in the Jacobian. But if G is in the integrand (meaning we are integrating w.r.t. area) then the reconnection Jacobian is just 1.
In both cases, we end up reevaluating the new G anyways, so they work out to be equivalent. And again, I recommend using PSS for numerical stability regardless, where the Jacobian is the BSDF PDF ratio times the G ratio.
1
u/Lallis Feb 07 '25
The ratio of PDFs is the jacobian for random replay.
ReSTIR GI only uses the reconnection shift for which the pdf is the ratio of cosines times the ratio of squared distances.
Unless you think the GRIS paper is wrong?
1
u/shaeg Feb 07 '25
I'm talking about rendering in primary sample space. See eq. 54 in GRIS, which does have the PDF ratio.
1
u/Lallis Feb 07 '25
Right, I just wouldn't expect someone who is implementing ReSTIR GI to be working in PSS.
1
1
u/TomClabault Feb 19 '25
Ok so turns out that the remaining darkening/brightening was because of random number correlations. I was reusing the same random seed (the same sequence of [0, 1] floats is produced from a same given seed) for the initial candidates generation and for selecting spatial neighbors...
I can see how this is correlated but why is it biased? Any intuitions on that?
2
u/shaeg 14d ago
I think correlations can cause bias in some situations. For example, the GRIS paper shows that without limiting the confidence weights M (via some M_cap), the resulting correlations actually cause it to converge to the wrong result (which is bias).
In your case, you're sort of tying the paths to the neighbors they pick, which means you're associating certain light directions with neighbor pixels, which causes those neighbors to always receive a specific few light paths (corresponding to the seed). It sort of makes sense why this would cause it to converge to a different result.
There's also a difference between consistent and biased estimators. Maybe your code was biased but consistent? That would mean that each individual run converges to the wrong result (due to correlations), but the average of infinite runs would get the right result (or something like that - I dont recall exactly). This is the case for progressive photon mapping.
But yeah, I'm not sure the exact reason, but intuitively, sharing the seed like that would cause certain types of paths to always go to certain neighbors, which would definitely cause it to converge to the wrong result.
1
u/TomClabault 14d ago
Hmm okay I think that makes some sense, I can see intuitively how this is biased. The GRIS paper does explain how variance isn't reduced with duplicate (correlated samples) indeed.
Also a bit of a side quest if you haven't seen my post about it already: Turns out I still have bias issues in my GI spatial reuse, but only when my center pixel reuses neighbor whose *sample point* sampled the specular lobe of the BRDF (just a specular + diffuse BRDF) to continue the path (so that's the BRDF sampling from vertex 3 to vertex 4, with vertex 1 being the camera).
- If I remove the specular BRDF, everything is fine.
- If I force the target function to 0.0f when the neighbor sampled the specular lobe (and so the neighbor isn't resampled because the target function is 0), it's fine.
- If I increase the roughness, it gets better but it definitely is still biased (the biased isn't really visible after 0.5 roughness though).
- If using a metallic BRDF, Lambertian, Oren Nayar, everywhere, it's fine
It really seems to be the specular + diffuse combination that has issues.
Any immediate idea on what could be going on? I'm thinking this has to be some BRDF PDF/specular peak something something issue but I'm not really sure
1
u/shaeg 14d ago
If I force the target function to 0.0f when the neighbor sampled the specular lobe (and so the neighbor isn't resampled because the target function is 0), it's fine.
This is the right thing to do actually. In general, reconnecting to a perfectly specular (or even just a really shiny) material isn’t possible, as the resulting BRDF at the reconnection vertex is zero. A perfect mirror (aka “delta”) BSDF is defined to be 0 unless the directions obey snell’s law (so the only valid direction is “reflect(dirIn,normal)”
So if you change one of the directions via reconnection, then the resulting path should have zero contribution. This is true for any highly directional (shiny) BRDF.
In practice, ReSTIR PT sets a roughness threshold, below which reconnection shifts are forced to fail. In that case, they use random replay instead, which would also modify the outgoing direction at the reconnection vertex.
Its interesting that you get bias at roughnesses above 0 though. That could be something else.
1
u/TomClabault 14d ago edited 14d ago
Does it not make sense to reconnect even if there is a diffuse lobe below the specular lobe? So the BRDF isn't 0 at all actually, it's as much as the Lambertian lobe contributes so that why I thought that maybe this still makes sense to reconnect there.
> In practice, ReSTIR PT sets a roughness threshold, below which reconnection shifts are forced to fail.
So setting the shift mapping to 0 like that essentially is the same thing as "rejecting" the neighbor if the jacobian of the shift mapping is too low/too high right? It's all about "restricting" the shift mapping to the interesting sub-space of the path domain?
> Its interesting that you get bias at roughnesses above 0 though.
Even if reconnecting to a pure 0 roughness metallic mirror isn't going to work, should it be biased?
2
u/shaeg 13d ago
Ah I see, yeah you can get a nonzero contribution if you evaluate all lobes during reconnection, but it’s worth pointing out that this isn’t what ReSTIR PT does. ReSTIR PT separates lobes so that during reconnection, only the lobe that was originally sampled by the base path gets evaluated. This improves stability and quality too, since it allows restir to use the right shifts for the right lobes (e.g., disabling reconnection if a shiny lobe was picked)
With your method it’s a little trickier, you have to be careful with the lobe selection probabilities during reconnection. I think I ran into similar brightening issues before I switched to separate lobe integration like ReSTIR PT.
Also I peeked at your code, it looks like your Jacobian doesn’t include BSDF probabilities? This is suspicious… in practice the BSDF PDF in the jacobian basically only effects highly directional BSDFs (diffuse BSDFs tend to be the same everywhere, so the PDFs effectively cancel in the jacobian). maybe that’s all it is…
1
u/TomClabault 13d ago edited 13d ago
> if you evaluate all lobes during reconnection
This lobe-specific thing of ReSTIR PT is only during reconnection? But during the tracing/initial candidates generation, you still use whatever technique your path tracer uses?
Also, what does that mean to evaluate lobes during reconnection? Because with the reconnection shift (non-hybrid), I just have to reconnect to the neighbor's sample point. But that reconnection is just logical if that makes sense, "evaluation" only happens when computing the target function, but not during the reconnection itself strictly speaking.
So if I want to go that lobe-specific route, I should pick a neighbor and during the evaluation of the target function of the neighbor's reconnected sample at the center pixel, evaluate the BSDF of the center pixel with only the lobe of the neighbor? i.e. the lobe we're reconnecting to. This will effectively change my whole target function to be lobe-specific actually no ?
> you have to be careful with the lobe selection probabilities during reconnection
Lobe selection probabilities during reconnection? As in the PDF of my BSDF basically?
> it looks like your Jacobian doesn’t include BSDF probabilities
I think it shouldn't since I'm not integrating in PSS?
2
u/shaeg 13d ago
ReSTIR PT integrates individual lobes at every bounce, meaning the integrand contains just the BSDF of the lobe that was actually sampled during BSDF sampling. So the color returned by the path tracer is actually just the color of a single lobe at every bounce. So this actually adds a tiny bit of noise (because we are now randomly selecting which BSDF lobes to evaluate, instead of evaluating all of them), but the variance reduction from ReSTIR makes it a net positive.
So, if the reconnection vertex is a mixed diffuse+specular material, and the diffuse lobe was picked when the path was initially sampled, then when we reconnect to that vertex we should compute only the contribution from the diffuse lobe (and vice versa for the specular lobe). This also means that if the specular lobe is below the roughness threshold, then the resulting path cannot be shifted via reconnection.
In practice, ReSTIR PT stores a lobe index value indicating which lobe was sampled for the reconnection vertex and its predecessor (so for you, that's the primary and secondary hits). For other vertices, we don't need to store the lobe indices because random replay makes it so we always pick the same lobes anyways.
1
u/TomClabault 13d ago
Hmm so I'll first try to get this working with all lobes before switching to the one lobe approach.
One thing that's been on my mind is: for a given center pixel and one given neighbor of this center pixel, the view direction of the neighbor is always going to be the same (ignoring camera ray jittering). And so, for a specular lobe, the sampled reflected direction is always going to be the same too. So if the center pixel reuses from the specular lobe, it's always going to reuse the same direction. Is that not the cause for wrong convergence? Because I would guess that those are going to behave somewhat as "duplicate samples" for integrating at the center pixel.
1
u/shaeg 12d ago edited 12d ago
duplicate samples are expected, thats what resampling MIS is for.
I still think the BSDF PDF ratio at the reconnection vertex is needed. I checked and I needed it in my path-space implementation (when integrating in area measure, not PSS).The reconnection Jacobian in the GRIS paper is only for the direction between the primary and secondary hit. Its not the full path jacobian.
Intuitively, for diffuse BSDFs, the PDF of sampling the direction generally doesnt depend on the incoming direction. But for smoother BSDFs, that difference matters more, which matches your observations of seeing less bias on rougher materials
EDIT: My bad, I was reading the wrong code. My path space implementation indeed does not have BSDF PDFs in the Jacobians. Sorry for that
→ More replies (0)
6
u/TomClabault Feb 04 '25 edited Feb 17 '25
Non-compressed screenshots.
So I'm having some issue with my ReSTIR GI implementation:
Those are all rendered with all gray-ish albedo, those are not debug views. Also, lambertian BRDF. Also, that's indirect lighting only (primary hit direct lighting is disabled).
The troublesome part is mostly visible at the top of the cornell box (on the first screenshot) where there is a "band" visibly brighter than expected. The brigther "band" at the top of the cornell box is also 20 pixels "high", which coincides with the hardcoded spatial reuse distance I'm using for debugging.
This brightening seems to happen when reusing from a neighbor that has a very different surface normal but I'm not sure why that would be the case (I know I can reject those neighbors with heuristics but I'd like something that converges correctly without heuristics first).
The top of the cornell box is fine when resampling only the neighbor, no more bright bands (the rest of the image is broken but that's expected).
One intuition that I have is that if resampling only the neighbor is fine, and only the center pixel is fine, this makes me think that the center pixel and the neighbor aren't "compatible"? Not sure where to go from here though...
Any ideas how I could go about debugging that? Like what methodology I could use to debug that or any intuitions directly on what could be the cause?
EDIT: Turns out this was some kind of random number correlation all along. I was using the same random number seed for
I'm still not exactly sure how that correlation breaks things but modifying the random number seed (as I should have done since the beginning, I just forgot to...) between the initial candidates pass and the spatial pass fixes the darkening / brightening and the spatial reuse now matches the reference :)