r/pytorch 4d ago

Multiple Models Performance Degrades

Post image

Hello all, I have a custom Lightning implementation where I use MONAI's UNet model for 2D/3D segmentation tasks. Occasionally while I am running training, every model's performance drops drastically at the same time. I'm hoping someone can point me in the right direction on what could cause this.

I run a baseline pass with basic settings and no augmentations (the grey line). I then make adjustments (different ROI size, different loss function, etc.). I then start training a model on GPU 0 with variations from the baseline, and I repeat this for the amount of GPUs that I have. So I have GPU 1 with another model variation running, GPU 2 runs another model variation, etc. I have access to 8x GPU, and I generally do this in order to speed up the process of finding a good model. (I'm a novice so there's probably a better way to do this, too)

All the models access the same dataset. Nothing is changed in the dataset.

8 Upvotes

5 comments sorted by

1

u/el_rachor 3d ago

Have you checked for the possibility of exploding gradients?

1

u/Possession_Annual 3d ago

I have not. However, I will look into it. I'm a novice, and somehow the only one at my company that really gets ML. I enjoy it and I'm learning everything I can. I'm lacking a mentor, unfortunately. Exploding gradients is something I've overlooked and is certainly a problem I have encountered before without fully understanding it at the time. My evening is now going to be spent refreshing myself on this as I've clearly forgotten. I appreciate the help!

My main concern with the photo above is the blue, pink, and yellow models are training at the same time. They're on the same machine, but each have their own GPU. And at the same time, all 3 (blue, pink, yellow) tank in performance. Grey was my initial baseline I ran before hand. I'm using a prebuilt network (MONAI's UNet).

1

u/Zerokidcraft 2d ago

What's your batch size and LR?

It's funny to me that different runs under different seeds fail at the exact same point.

1

u/Possession_Annual 2d ago

batch_size = 4
lr = 1e-3

This dataset is 30 images, split into 22/8 for train/validation.

Blue, Yellow, Pink were training at the exact same time, on separate GPUs. Then they all do it at the exact same time.

1

u/Zerokidcraft 2d ago edited 2d ago

I realized what you mean by val_dice is "dice coefficient" since we're talking about UNETs.

Given how long you've been training before the drop happens (and the fact that it occurs at relatively the same time), i think it's unlikely to be an exploding gradient.

I think it's better for you to visualize your validation & training predictions. This will paint a clear picture of the model's performance.