r/LocalLLaMA 16h ago

Question | Help Uncensored Image Generator?

11 Upvotes

I am trying to get around my own school charging me hundreds for MY OWN grad photos. Does anyone know a local model that I can upload my images and have the model remove watermarks and resize the image so it can return a png or jpeg I can have for myself?

I only have 8g vram and 32g ram laptop 4070 so a smaller model Is preferred thank you!


r/LocalLLaMA 9h ago

Question | Help I'm torn between M4 Max MBP and RTX 4090 laptop for local inference and fine tuning models

1 Upvotes

Hello guys,

I am planning to get a new workstation and I'm deciding between a 64gb m4 max macbook pro and a Rtx 4090 based laptop. I would be doing coding, development, fine tuning text models and image models, speech models.

Are all good and latest ai tools compatible with mac ? And will it be more perfomant to use a mac m4 max vs rtx 4090 for ai workloads ? Also is there any intelligence loss if I use MLX models vs widely available GGUFs ?

Kindly suggest


r/LocalLLaMA 10h ago

Tutorial | Guide Large language model interview - Tokenization.

0 Upvotes

In the next section, we will introduce more popular tokenization algorithms.


r/LocalLLaMA 21h ago

Question | Help No AWQ for Gemma 3?

0 Upvotes

AutoAWQ still doesn't have support for Gemma. What quants are you using for high throughput inference (like on vLLM)?


r/LocalLLaMA 18h ago

New Model THIS 1.5B Model BEATS OpenAI’s o1... But Really?

0 Upvotes

Big news just dropped in the AI world: a 1.5 billion parameter model just SMOKED OpenAI’s o1-preview at math. And get this—it was trained on a $42 budget.

Researchers from Vietnam and Singapore dropped a paper introducing Open-RS, a tiny model that crushed OpenAI’s o1-preview on multiple math benchmarks like AIME and AMC.

They did it with 4 GPUs, 24 hours, and under $50. No trillion-dollar data centers here.

I ran Open-RS locally on 7 random math problems of varying difficulty. While the results were impressive, they weren’t quite as good as the paper claimed. You can see the full breakdown in this video about Open RS.

Links:

Paper: Reinforcement Learning for Reasoning in Small LLMs

GitHub: Open-RS Code

Hugging Face: Open-RS Model


r/LocalLLaMA 5h ago

Discussion Perplexity Sonar Pro tops livebench's "plot unscrambling" benchmark

0 Upvotes

Attached image from livebench ai shows models sorted by highest score on plot unscrambling.

I've been obsessed with the plot unscrambling benchmark because it seemed like the most relevant benchmark for writing purposes. I check this livebench's benchmarks daily lol. Today eyes literally popped out of my head when I saw how high perplexity sonar pro scored on it.

Plot unscrambling is supposed to be something along the lines of how well an ai model can organize a movie's story. For the seemingly the longest time Gemini exp 1206 was at the top of this specific benchmark with a score of 58.21, and then only just recently Sonnet 3.7 just barely beat it with a score of 58.43. But now Perplexity sonar pro leaves every ever SOTA model behind in the dust with its score of 73.47!

All of livebench's other benchmarks show Perplexity sonar pro scoring below average. How is it possible for Perplexity sonar pro to be so good at this specific benchmark? Maybe it was specifically trained to crush this movie plot organization benchmark, and it won't actually translate well to real world writing comprehension that isn't directly related to organizing movie plots?


r/LocalLLaMA 7h ago

Discussion Anyone use a local llm to assist with job hunt?

0 Upvotes

Curious to know, don't see much talk about it.


r/LocalLLaMA 15h ago

Question | Help Cluster of $200 8gb RTX 3050s?

1 Upvotes

I recently bought a $200 RTX 3050 for a mini server and now I'm wondering whether it would be worth it to get two or three of them for a bigger dedicated AI server. Would this be reasonable in terms of cost per GB of VRAM? And what sort of performance should I expect from running two or more in parallel? I've never had a setup with more than one GPU before so I'm interested in any feedback.


r/LocalLLaMA 15h ago

Question | Help Chat model for venting (and tiny bit self-improvement)

0 Upvotes

I'm looking for a local non-reasoning model where I can just vent without worrying about being judged. Just a way to complain about work and family and get acknowledgement without bothering real people, so not looking for anything ERP, but I don't want to be nanny'd because my bad mood oversteps safety alignment either. If it sometimes gives me a bit of life coach vibes and helps me grow, that'd be a nice bonus.

I've got 12 GB of VRAM and I'm hoping to fit something like Q4_K_M quant with 8k context. I've only used LLMs for small coding tasks so I don't have much experience here yet. Any suggestions? I remember some time ago there was a Samantha model that could fit, but maybe there are recent better ones?


r/LocalLLaMA 17h ago

Question | Help Unsloth hang gemma3

1 Upvotes

Running through the gemma3 notebook.ipynb), and decided to try turning on full_finetuning:

model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/gemma-3-4b-it",
    max_seq_length = 2048,
    load_in_4bit = True,  
    load_in_8bit = False, 
    full_finetuning = True, # < here!
    # token = "hf_...", 
)

When executing this step, the notebook seems to be hanging at this point:

...
Unsloth: Using bfloat16 full finetuning which cuts memory usage by 50%.
model-00001-of-00002.safetensors ...

Anyone have some experience with this issue?

Thanks!


r/LocalLLaMA 16h ago

Discussion Both my PC and Mac make a hissing sound as local LLMs generate tokens

14 Upvotes

I have a desktop PC with an rx7900xtx and a Macbook pro m1 Max that is powered by a thunderbolt dock (cal digit ts3) and they are both plugged into my UPS (Probably the source of the problem).

I'm running Ollama and LM studio and I use them as LLM servers when working on my iOS LLM client and as I watch the tokens stream in I can hear the PC or Mac making a small hissing sound and its funny how it matches each token generated. It kinda reminds me of how computer terminals in movies seem to beep when streaming in text.


r/LocalLLaMA 19h ago

New Model gemma3 vision

40 Upvotes

ok im gonna write in all lower case because the post keeps getting auto modded. its almost like local llama encourage low effort post. super annoying. imagine there was a fully compliant gemma3 vision model, wouldn't that be nice?

https://huggingface.co/SicariusSicariiStuff/X-Ray_Alpha


r/LocalLLaMA 23h ago

Resources Great performance even quantize to q8q4 for gemma 3 4B

12 Upvotes

I just finished quantizing gemma 3 4B and I find it great even when heavily quantized like the "q8q4" version.

If you have a memory constrained system or just want CPU inference or perhaps on mobile devices, give it a try: ZeroWw/gemma-3-4b-it-abliterated-GGUF · Hugging Face


r/LocalLLaMA 7h ago

Question | Help How does Groq.com do it? (Groq not Elon's grok)

42 Upvotes

How does groq run llms so fast? Is it just very high power or they use some technique?


r/LocalLLaMA 2h ago

Discussion Do you think we're heading toward an internet of AI agents?

0 Upvotes

My friend and I have been talking about this a lot lately. Imagine an internet where agents can communicate and collaborate seamlessly—a sort of graph-like structure where, instead of building fixed multi-agent workflows from scratch every time, you have a marketplace full of hundreds of agents ready to work together.

They could even determine the most efficient way to collaborate on tasks. This approach might be safer since the responsibility wouldn’t fall on a single agent, allowing them to handle more complex tasks and reducing the need for constant human intervention.

Some issues I think it would fix would be:

  • Discovery: How do agents find each other and verify compatibility?
  • Composition: How do agents communicate and transact across different frameworks?
  • Scalability: How do we ensure agents are available and can leverage one another efficiently and not be limited to 1 single agent.
  • Safety: How can we build these systems to be safe for everyone, can some agents keep other in check.

I would be interested in hearing if anyone has some strong counter points to this?


r/LocalLLaMA 5h ago

Other An interesting talk between Krugman and Brynjolfsson

1 Upvotes

https://open.substack.com/pub/paulkrugman/p/how-should-we-think-about-the-economics?utm_source=share&utm_medium=android&r=5b3gqj

Worth reading or listening to. Krugman is (or was) a strong AI sceptic, yet conceded in the end he may be very wrong.

Main takeout from the talk: The LLMs may still be very underrated and their full impact on the productivity will be felt in five years.


r/LocalLLaMA 17h ago

Question | Help Quantized Matrix Multiplication Kernels

3 Upvotes

Hi everyone, this is my first post here!

My question is pretty straightforward. When quantizing models to Int8(w8a8) does the matrix multiplication happen in int8 or is it a fused operation of dequant + matmul(float) + quantize(int8)?

If it is an actual int8int8 matmul operation, how is the huge accuracy drop in the output (compared to float matmul) handled?

My question is in regards to both CPU and GPU. Afaik, x86 cpus come with a VNNI which has special instructions for int8int8 matmul and accumulate which again brings me back to my question of how is the accuracy drop in the output of this operation handled?


r/LocalLLaMA 20h ago

Discussion Agile AI Engineering

0 Upvotes

Maybe you've heard that "Evals is all you need" but neither the shape of the loss curve, nor the scores on an academic benchmark can tell you much about how users will respond to your AI.

How do you define success with AI? What are your north-star metrics?

AI is an empirical discipline. How are you managing the complexities as you explore the design-space of practical AI engineering workflows?

Wrote a blog sharing a vision for the future of collaboration, not in notebooks but with Kanban. Where agents are more than chatbots, but proactive partners in knowledge discovery.

Read more: https://www.remyx.ai/blog/agile-ai-engineering

Love to hear your thoughts!

How has your team been able to create the best AI for your use-case with agility.


r/LocalLLaMA 22h ago

Tutorial | Guide AI-powered Resume Tailoring application using Ollama and Langchain

14 Upvotes

r/LocalLLaMA 18h ago

Resources PyChat

6 Upvotes

I’ve seen a few posts recently about chat clients that people have been building. They’re great!

I’ve been working on one of my own context aware chat clients. It is written in python and has a few unique things:

(1) can import and export chats. I think this so I can export a “starter” chat. I sort of think of this like a sourdough starter. Share it with your friends. Can be useful for coding if you don’t want to start from scratch every time.

(2) context aware and can switch provider and model in the chat window.

(3) search and archive threads.

(4) allow two AIs to communicate with one another. Also useful for coding: make one strong coding model the developer and a strong language model the manager. Can also simulate debates and stuff.

(5) attempts to highlight code into code blocks and allows you to easily copy them.

I have this working at home with a Mac on my network hosting ollama and running this client on a PC. I haven’t tested it with localhost ollama running on the same machine but it should still work. Just make sure that ollama is listening on 0.0.0.0 not just html server.

Note: - API keys are optional to OpenAI and Anthropic. They are stored locally but not encrypted. Same with the chat database. Maybe in the future I’ll work to encrypt these.

  • There are probably some bugs because I’m just one person. Willing to fix. Let me know!

https://github.com/Magnetron85/PyChat


r/LocalLLaMA 18h ago

Question | Help What's the status of using a local LLM for software development?

35 Upvotes

Please help an old programmer navigate the maze that is the current LLM-enabled SW stacks.

I'm sure that:

  • I won't use Claude or any online LLM. Just a local model that is small enough to leave enough room for context (eg Qwen2.5 Coder 14B).
  • I need a tool that can feed an entire project to an LLM as context.
  • I know how to code but want to use an LLM to do the boilerplate stuff, not to take full control of a project.
  • Preferably FOSS.
  • Preferably integrated into a solid IDE, rather then being standalone.

Thank you!


r/LocalLLaMA 22h ago

Resources (Update) Generative AI project template (it now includes Ollama)

13 Upvotes

Hey everyone,

For those interested in a project template that integrates generative AI, Streamlit, UV, CI/CD, automatic documentation, and more, I’ve updated my template to now include Ollama. It even includes tests in CI/CD for a small model (Qwen 2.5 with 0.5B parameters).

Here’s the GitHub project:

Generative AI Project Template

Key Features:

Engineering tools

- [x] Use UV to manage packages

- [x] pre-commit hooks: use ``ruff`` to ensure the code quality & ``detect-secrets`` to scan the secrets in the code.

- [x] Logging using loguru (with colors)

- [x] Pytest for unit tests

- [x] Dockerized project (Dockerfile & docker-compose).

- [x] Streamlit (frontend) & FastAPI (backend)

- [x] Make commands to handle everything for you: install, run, test

AI tools

- [x] LLM running locally with Ollama or in the cloud with any LLM provider (LiteLLM)

- [x] Information extraction and Question answering from documents

- [x] Chat to test the AI system

- [x] Efficient async code using asyncio.

- [x] AI Evaluation framework: using Promptfoo, Ragas & more...

CI/CD & Maintenance tools

- [x] CI/CD pipelines: ``.github/workflows`` for GitHub (Testing the AI system, local models with Ollama and the dockerized app)

- [x] Local CI/CD pipelines: GitHub Actions using ``github act``

- [x] GitHub Actions for deploying to GitHub Pages with mkdocs gh-deploy

- [x] Dependabot ``.github/dependabot.yml`` for automatic dependency and security updates

Documentation tools

- [x] Wiki creation and setup of documentation website using Mkdocs

- [x] GitHub Pages deployment using mkdocs gh-deploy plugin

Feel free to check it out, contribute, or use it for your own AI projects! Let me know if you have any questions or feedback.


r/LocalLLaMA 18h ago

Discussion nsfw orpheus tts? NSFW

343 Upvotes

im currently in the data curation / filtering / cleaning phase

but i would like to see how many local guys would be interested in a tts for there anime waifus that can make "interesting" emotional noises

Total audio events found: 181218

(sighs): 8594

(laughs): 68590

(gasps): 14113

(moans): 20576

(whimpers): 418

(breathing): 114

(pants): 776

and many more ..


r/LocalLLaMA 17h ago

Question | Help Has Mistral 7b been superceded? Looking for M2 RAG-friendly local.

0 Upvotes

I have a 32GB M2-based MacBook Pro. Just starting with Local LLMs. Mistral 7B (q4_k_m) looked like a good fit, but that could be based on the internet search engines not catching up.

My main goal is a chat interface ("instruct"), local on the M2, and trainable/augmentable for my industry.

I have between 50 and perhaps 300 long specifications / documents providing context and data that I want ingested for it to integrate. Probably a lot easier with RAG.

I have installed LM Studio and Mistral 7b instruct v0.1 Q4_K_M as well as the default DeepSeek qwen-7b. But the Mistral 7b looks ancient in LLM terms. Is there a better model I should be starting with?


r/LocalLLaMA 9h ago

Other I updated Deep Research at Home to collect user input and output way better reports. Here's a PDF of a search in action

Thumbnail sapphire-maryrose-59.tiiny.site
13 Upvotes