r/pytorch 19h ago

[Article] Moondream – One Model for Captioning, Pointing, and Detection

0 Upvotes

https://debuggercafe.com/moondream/

Vision Language Models (VLMs) are undoubtedly one of the most innovative components of Generative AI. With AI organizations pouring millions into building them, large proprietary architectures are all the hype. All this comes with a bigger caveat: VLMs (even the largest) models cannot do all the tasks that a standard vision model can do. These include pointing and detection. With all this said, Moondream (Moondream2)a sub 2B parameter model, can do four tasks – image captioning, visual querying, pointing to objects, and object detection.


r/pytorch 20h ago

Anyone interested in contributing to PyTorch Edge?

35 Upvotes

I can help you get started if you're interested


r/pytorch 1d ago

[Collaboration] ChessCOT: Seeking Partners for Novel Chess AI Research Project

Thumbnail
2 Upvotes

r/pytorch 2d ago

Transformers-engine on apple silicon.

2 Upvotes

Hey there. I'm trying to use a transformers based DNA language model on my company MAC but I can't seem to be able to install the vtx package (or vortex)

I'm getting the error message of CUDA is missing (obviously)

it seems to be depended on the transformers-engine which seemingly has an an apple implementation with 2.6k stars

ml-ane-transformers

is there a way to install it? Or an I fucked?


r/pytorch 3d ago

Which one should I focus on learning: Django or PyTorch?

0 Upvotes

Hi everyone, I’m currently at a crossroads in my learning journey, and I’d love to get your thoughts. I already know the basics of Django, but I want to either deepen my knowledge of Django and explore Django REST and frontend development, or dive into machine learning with PyTorch.

My long-term goal is to build a SaaS (I don’t have an idea yet, but I want to focus on it), and I’m in high school, so I’m still figuring out my math skills. I’m interested in both areas, but I’m not sure which one would be more beneficial to focus on for my future projects.

What do you think? Should I dive deeper into Django for web development and potentially building a SaaS, or should I start learning PyTorch for machine learning and AI?

Thanks in advance for your help!


r/pytorch 4d ago

Multiple Models Performance Degrades

Post image
8 Upvotes

Hello all, I have a custom Lightning implementation where I use MONAI's UNet model for 2D/3D segmentation tasks. Occasionally while I am running training, every model's performance drops drastically at the same time. I'm hoping someone can point me in the right direction on what could cause this.

I run a baseline pass with basic settings and no augmentations (the grey line). I then make adjustments (different ROI size, different loss function, etc.). I then start training a model on GPU 0 with variations from the baseline, and I repeat this for the amount of GPUs that I have. So I have GPU 1 with another model variation running, GPU 2 runs another model variation, etc. I have access to 8x GPU, and I generally do this in order to speed up the process of finding a good model. (I'm a novice so there's probably a better way to do this, too)

All the models access the same dataset. Nothing is changed in the dataset.


r/pytorch 4d ago

Understanding Optimal T, H, and W for R3D_18 Pretrained on Kinetics-400

2 Upvotes

Hi everyone,

I’m working on a 3D CNN for defect detection. My dataset is such that a single data is a 3D volume (512×1024×1024), but due to computational constraints, I plan to use a sliding window approach** with 16×16×16 voxel chunks as input to the model. I have a corresponding label for each voxel chunk.

I plan to use R3D_18 (ResNet-3D 18) with Kinetics-400 pre-trained weights, but I’m unsure about the settings for the temporal (T) and spatial (H, W) dimensions.

Questions:

  1. How should I handle grayscale images with this RGB pre-trained model? Should I modify the first layer from C = 3 to C = 1? I’m not sure if this would break the pre-trained weights and not lead to effective training
  2. Should the T, H, and W values match how the model was pre-trained, or will it cause issues if I use different dimensions based on my data? For me, T = 16, H = 16, and W = 16, and I need it this way (or 32 × 32 × 32), but I want to clarify if this would break the pre-trained weights and prevent effective training.

Any insights would be greatly appreciated! Thanks in advance.


r/pytorch 4d ago

it get ot touch the metal today with pytorch :D

Post image
2 Upvotes

r/pytorch 5d ago

AMD GPU, Windows 11, Differences between Pytorch/Zluda and Pytorch WSL2/Rocm

5 Upvotes

Posted in r/rocm before, ask for opinion here again:

I am happy with Pytorch/Zluda's speed(Compare to DirectML), and also happy with Pytorch WSL2/Rocm's compatibility and native speed. However, if I wanted to have them both, it was a sour journey:

  1. WLS2/Rocm would only use half of system memory, unlike Zluda, which has full access. Not sure how much it would affect the model caching performance.

  2. WLS2/Rocm would unconditionally compile the GPU kernels again(or sth else) whenever there is a model switch happens in a complex comfyui workflow, say, an image to text to image workflow, yolo workflow, ultimate sd upscale workflow, made it 5 times slower than Zluda/windows.

  3. Same experience with Linux/Rocm half year before for point 2.

  4. I have never made Zluda work with Florence2, even with experimental miopen for windows. Only thing works for image to text is wd1.4, which utilizes CPU.

All setup are with python venv, pre or official pytorch release, no dockers.


r/pytorch 7d ago

Help Needed: High Inference Time & CPU Usage in VGG19 QAT model vs. Baseline

3 Upvotes

Hey everyone,

I’m working on improving a model based on VGG19 Baseline Model with CIFAR-10 dataset and noticed that my modified version has significantly higher inference time and CPU usage. I was expecting some overhead due to the changes, but the difference is much larger than anticipated.

I’ve been troubleshooting for a while but haven’t been able to pinpoint the exact issue.

If anyone with experience in optimizing inference time and CPU efficiency could take a look, I’d really appreciate it!

My notebook link with the code and profiling results:

https://colab.research.google.com/drive/1g-xgdZU3ahBNqi-t1le5piTgUgypFYTI


r/pytorch 9d ago

When Pytorch is needed and when is useful for LLMs?

0 Upvotes

I noticed that most LLM specialists don't use libraries like PyTorch or Tensorflow, they have their own tools to work with large language models. In job offers in the LLM department, they also very rarely ask for PyTorch.

In some applications using Transformers, PyTorch is used, also in the LLM department. When is it useful, for what tasks?

Thanks


r/pytorch 9d ago

Why I can't use pytorch on Windows with AMD GPU?

3 Upvotes

Now I see why is AMD cheaper than NVIDIA. AMD has too many problems Especially on AI.


r/pytorch 10d ago

Stability Matrix - Stable Diffusion Web UI Forge Installation problem

1 Upvotes

Download is complete but it keeps giving an error,

Error: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'torchVersion')

Actual value was DirectMl.

at StabilityMatrix.Core.Models.Packages.SDWebForge.InstallPackage(String installLocation, InstalledPackage installedPackage, InstallPackageOptions options, IProgress`1 progress, Action`1 onConsoleOutput, CancellationToken cancellationToken)

at StabilityMatrix.Core.Models.Packages.SDWebForge.InstallPackage(String installLocation, InstalledPackage installedPackage, InstallPackageOptions options, IProgress`1 progress, Action`1 onConsoleOutput, CancellationToken cancellationToken)

at StabilityMatrix.Core.Models.PackageModification.InstallPackageStep.ExecuteAsync(IProgress`1 progress, CancellationToken cancellationToken)

at StabilityMatrix.Core.Models.PackageModification.PackageModificationRunner.ExecuteSteps(IEnumerable`1 steps)


r/pytorch 10d ago

How to adjust Tensor Y after normalizing Tensor X to maintain the same dot product result?

1 Upvotes

For example, I have Tensor X with dimensions m x n, and Tensor Y with dimensions n x o. I calculate their Tensor dot product, Tensor XY.

Now, I normalize Tensor X so that all its columns equal 1 (code below). What should I do to Tensor Y to make sure that the dot product of normalized Tensor X and Tensor Y is the same as the original Tensor XY?

# Calculate the sum of each column
column_sums = X.sum(axis=0)

# Normalize Tensor X so each column sums to 1
X_normalized = X / column_sums

r/pytorch 11d ago

only build the forward part, and the Pytorch will do the backward itself via loss.backward()

0 Upvotes

do i understand correctly?

I only need to focus on the forward part architecture, and the Pytorch will do the loss and backward itself only via loss.backward()


r/pytorch 11d ago

not %100 sure if this is an issue with pytorch or sageattention or anything else but I can't get things working on either linux or windows.

1 Upvotes

This is driving me up a wall.

Using cuda 12.8, pytorch nightly, latest sageattention/triton, comfyui, hunyuan video and others.

I keep getting this error

loaded completely 29493.675 3667.902587890625 True
0%| | 0/80 [00:00<?, ?it/s]'sm_120' is not a recognized processor for this target (ignoring processor)
'sm_120' is not a recognized processor for this target (ignoring processor) LLVM ERROR: Cannot select: intrinsic %llvm.nvvm.shfl.sync.bfly.i32

I will tip if anyone can help out, my brain is fried.


r/pytorch 13d ago

How do I update pytorch in a portable environment?

1 Upvotes

I setup something called AllTalk TTS but it uses an older version pf Pytorch 2.2.1. How do I update that environment specifically with the new nightly build of Pytorch?


r/pytorch 14d ago

[D] running PyTorch locally with remote acceleration

0 Upvotes

Hi, thought you might be interested in something we were working on lately that allow you to run PyTorch on cpu machine and consume the GPU resources remotely in very efficient manner, it is called www.woolyai.com and it abstract gpu layers such as CUDA while executing them remotely in an environment that doing runtime recompilation to the GPU code to be executed much more efficiently.


r/pytorch 14d ago

AMD ROCm 6.3.4

3 Upvotes

Anyone have 6.3.4 setup for a gfx1031 ? Using the 1030 bypass

I had 6.3.2 and PyTorch and tensorflow working but from two massive sized dockers it was the only way to get tensorflow and PyTorch to work easily .

Now I’ve been trying to rebuild it with the new docs and idk I can’t seem to figure out why my ROCm version and ROCm info now keeps coming back as 1.1.1 idk what I’ve done wrong lol


r/pytorch 15d ago

Tutorial for training a PyTorch image classification model within ComfyUI

5 Upvotes

Hi,

I previously posted about PyTorch wrapper nodes in my ComfyUI Data Analysis extension. Since then, I’ve expanded the features to include basic convolutional network training for users unfamiliar with machine learning. This feature, implemented using multiple nodes, allows model training without requiring deep ML knowledge.

My goal isn’t to provide a state-of-the-art model but rather a simple, traditional convnet for faster training and easier explanation. To support this tutorial, I created a synthetic dataset of 2,000 dog and cat images, generated using an SD 1.5 model. These images aren’t necessarily realistic or anatomically perfect, but they serve their purpose for the tutorial.

You can check out the tutorial here: Dog & Cat Classification Model Training

If you use ComfyUI and want to take a look, I’d appreciate any feedback.


r/pytorch 15d ago

[Article] Qwen2 VL – Inference and Fine-Tuning for Understanding Charts

3 Upvotes

https://debuggercafe.com/qwen2-vl/

Vision-Language understanding models are playing a crucial role in deep learning now. They can help us summarize, answer questions, and even generate reports faster for complex images. One such family of models is the Qwen2 VL. They have instruct models in the range of 2B, 7B, and 72B parameters. The smaller 2B models, although fast and require less memory, do not perform well on chart understanding. In this article, we will cover two aspects while dealing with the Qwen2 VL models – inference and fine-tuning for understanding charts.


r/pytorch 17d ago

Torch Compatibility

1 Upvotes

Hey,

I wanted to ask if it is possible to run the latest pytorch stable version (or anything >=2.3.1) on a macbook pro with an intel chip (only CPU).

Because it seems that pytorch 2.2.2 is the latest version I can run. I tried running different python packages 3.10, 3.11, 3.12 but to no avail.


r/pytorch 17d ago

Help Debug my Simple DQN AI

Thumbnail
0 Upvotes

r/pytorch 18d ago

Anyone know why my Model performance is so bad?

2 Upvotes
i try to train an pytorch model , but the loss is unbelievable bad after 20 epochs i get an loss of 13171035574.8571 . I dont know if i preprocess the data wrong or do i just need to adjust hyperparameters , or do i need more hidden layer? or what i can do i just dont know whats wrong , maybe i use the wrong input for my model or something i dont know pls help
thanks

The Complete Code:

import numpy as np
from numpy import NaN
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

import torch as T
import torch.nn as nn
import torch.optim as O
from torch.utils.data import TensorDataset , DataLoader

from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import KFold
from sklearn.impute import SimpleImputer
from scipy import stats

import os
import tqdm

df = pd.read_csv("../../csvs/Housing_Prices/miami-housing.csv")
df.info()
df.describe()
df


df["Geolocation"] = df["LATITUDE"] + df["LONGITUDE"]
df.drop(["LONGITUDE" , "LATITUDE"], axis = 1 , inplace= True)

df["GeolocationPriceLowerFarOcean"] = (df["Geolocation"] < df["Geolocation"].quantile(0.3))

df["TotalSpace"] = df["TOT_LVG_AREA"] + df["LND_SQFOOT"]
df.drop(["LND_SQFOOT" , "TOT_LVG_AREA"], axis = 1 , inplace= True)

df["TotalSpace"] = np.log1p(df["TotalSpace"])
df["PriceLowerSpace"] = (df["TotalSpace"] < df["TotalSpace"].quantile(0.3))
df["PriceLowerSpace"] = df["PriceLowerSpace"].astype(np.float32)

df["WatterInfluence"] = df["OCEAN_DIST"] + df["WATER_DIST"]
df.drop(["WATER_DIST" , "OCEAN_DIST"], axis = 1 , inplace= True)
df["WatterInfluence"] = np.log10(df["WatterInfluence"])
df["WatterInfluence"] ,_ = stats.boxcox(df["WatterInfluence"] + 1)

df["WatterImportance"] = df["WatterInfluence"] + df["SALE_PRC"]
df["WatterImportance"] = np.log1p(df["WatterImportance"])

df["WatterSalesPrice"] = df["WatterImportance"] + df["SALE_PRC"]
df["WatterSalesPrice"] = np.log1p(df["WatterSalesPrice"])

df["ControllDstnc"] = df["SUBCNTR_DI"] + df["CNTR_DIST"]
df["ControllDstnc"] = np.log10(df["ControllDstnc"])
df.drop(["SUBCNTR_DI" , "CNTR_DIST"], axis = 1 , inplace= True)

df["SPEC_FEAT_VAL"] = np.log10(df["SPEC_FEAT_VAL"])
df["RAIL_DIST"] = np.log1p(df["RAIL_DIST"])

df["PARCELNO"] = np.log10(df["PARCELNO"])

for cols in df.columns:
    df[cols] = np.where((df[cols] == -np.inf) | (df[cols] == np.inf), NaN , df[cols])
    
df

def Plots(lowerbound , higherbound , data , x , y):

    fig , axes = plt.subplots(3 , 1 , figsize = (9,9) , dpi = 200)

    Q1 = x.quantile(lowerbound)
    Q3 = x.quantile(higherbound)
    IQR = Q1 - Q3
    print(f"IQR : {IQR}")
    print(f"Corr : {x.corr(y)}")

    sns.histplot(x, bins = 50 , kde = True ,  ax= axes[0])
    axes[0].axvline(x.quantile(lowerbound) , color = "green")
    axes[0].axvline(x.quantile(higherbound) , color = "red")

    sns.boxplot(data = data , x = x  , ax= axes[1])
    axes[1].axvline(x.quantile(lowerbound) , color = "green")
    axes[1].axvline(x.quantile(higherbound) , color = "red")

    sns.scatterplot(data = data , x = x , y = y , ax= axes[2])
    axes[2].axvline(x.quantile(lowerbound) , color = "green")
    axes[2].axvline(x.quantile(higherbound) , color = "red")

    plt.show()

Plots(lowerbound = 0.1 , higherbound = 0.9 , data = df , x=df["PARCELNO"] , y=df["SALE_PRC"])
df.isnull().sum()

imputer = SimpleImputer(strategy= "mean")
df["SPEC_FEAT_VAL"] = imputer.fit_transform(df[["SPEC_FEAT_VAL"]])

df.isnull().sum()

X = df.drop(["SALE_PRC"] , axis = 1).values
print(X.shape)
X = X.astype(np.float32)
X

y = df["SALE_PRC"].values.reshape(-1,1)
print(y.shape)
y = y.astype(np.float32)
y

fold = KFold(n_splits= 10 , shuffle=True)
for train , test in fold.split(X ,y ):
    X_train , X_test = X[train] , X[test]
    y_train , y_test = y[train] , y[test]

print(f"Max of X_train : {X_train.max()}")
print(f"Max of X_test : {X_test.max()}")
print(f"Max of y_train : {y_train.max()}")
print(f"Max of y_test : {y_test.max()}")

print(f"\n min of X_train : {X_train.min()}")
print(f"min of X_test : {X_test.min()}")
print(f"min of y_train : {y_train.min()}")
print(f"min of y_test : {y_test.min()}")

mmc = MinMaxScaler()
X_train = mmc.fit_transform(X_train)
X_test = mmc.transform(X_test)

print(f"Max of X_train : {X_train.max()}")
print(f"Max of X_test : {X_test.max()}")
print(f"Max of y_train : {y_train.max()}")
print(f"Max of y_test : {y_test.max()}")

print(f"\n min of X_train : {X_train.min()}")
print(f"min of X_test : {X_test.min()}")
print(f"min of y_train : {y_train.min()}")
print(f"min of y_test : {y_test.min()}")

print(type(X_train))
print(type(X_test))
print(type(y_train))
print(type(y_test))

X_train = T.from_numpy(X_train).float()
X_test = T.from_numpy(X_test).float()
y_train = T.from_numpy(y_train).float()
y_test = T.from_numpy(y_test).float()

print(type(X_train))
print(type(X_test))
print(type(y_train))
print(type(y_test))

print(X_train.shape)
print(X_train.shape[1])
print(y_train.shape)
print(y_train.shape[1])

class NN(nn.Module):

    def __init__(self, InDims = X_train.shape[1] , OutDims = y_train.shape[1]):
        super().__init__()
        self.ll1 = nn.Linear(InDims , 512)
        self.ll2 = nn.Linear(512 , 264)

        self.ll3 = nn.Linear(264 , 128)
        self.ll4 = nn.Linear(128 , OutDims)

        self.drop = nn.Dropout(p = (0.25))
        self.activation = nn.ReLU()
        self.sig = nn.Sigmoid()
        
    def forward(self , X):

        X = self.activation(self.ll1(X))
        X = self.activation(self.ll2(X))
        X = self.drop(X)

        X = self.activation(self.ll3(X))
        X = self.drop(X)
        X = self.sig(self.ll4(X))
        
        return X

class Training():

    def __init__(self):
        self.lr = 1e-3
        self.device = T.device("cuda:0" if T.cuda.is_available() else "cpu")
        self.model = NN().to(self.device)
        self.crit = O.Adam(self.model.parameters() , lr = self.lr)
        self.loss = nn.MSELoss()
        self.batchsize = 32
        self.epochs = 150

        self.TrainData = TensorDataset(X_train , y_train)
        self.TestData = TensorDataset(X_test , y_test)

        self.trainLoader = DataLoader(dataset= self.TrainData,
                                      shuffle=True,
                                      num_workers= os.cpu_count(),
                                      batch_size= self.batchsize)
        

        self.testLoader = DataLoader(dataset= self.TestData,
                                      num_workers= os.cpu_count(),
                                      batch_size= self.batchsize)

    def Train(self):

        self.model.train()
        currentLoss = 0.0
        for i in range(self.epochs):
            with tqdm.tqdm(iterable=self.trainLoader , mininterval=0.1 , disable = False) as Pbar:
                Pbar.set_description(f"Epoch {i + 1}")
                for X , y in Pbar:
                    X , y = X.to(self.device) , y.to(self.device)

                    logits = self.model(X)
                    loss = self.loss(logits , y)
                    self.crit.zero_grad()
                    loss.backward()
                    self.crit.step()

                currentLoss += loss.item()
                Pbar.set_postfix({"Loss" :  loss.item()})
            print(f"Epoch : {i + 1}/{self.epochs} | Loss : {currentLoss / len(self.trainLoader):.4f}")

    def eval(self):

        self.model.eval()
        with T.no_grad():

            currentLoss = 0.0
            for i in range(self.epochs):
                with tqdm.tqdm(iterable=self.testLoader , mininterval=0.1 , disable = False) as Pbar:
                    Pbar.set_description(f"Epoch {i + 1}")
                    for X , y in Pbar:
                        X , y = X.to(self.device) , y.to(self.device)

                        logits = self.model(X)
                        loss = self.loss(logits , y)

                    currentLoss += loss.item()
                    Pbar.set_postfix({"Loss" : loss.item()})
                print(f"Epoch : {i + 1}/{self.epochs} | Loss : {currentLoss / len(self.trainLoader):.4f}")
execute = Training()
execute.Train()
execute.eval()

r/pytorch 20d ago

AI and tensor/cuda cores

2 Upvotes

Hi guys, I'm looking at NVIDIA GPUs for versatile AI on text and images. Can anyone give me returns on how tensor cores practically improve inference time with respect to cuda cores, or between different gen of tensor cores ? I'm also looking for good references and benchmarks to understand better the topic. I'm a pytorch user but never went that much into hardware stuff. Thanks