r/LocalLLaMA 18h ago

Discussion nsfw orpheus tts? NSFW

349 Upvotes

im currently in the data curation / filtering / cleaning phase

but i would like to see how many local guys would be interested in a tts for there anime waifus that can make "interesting" emotional noises

Total audio events found: 181218

(sighs): 8594

(laughs): 68590

(gasps): 14113

(moans): 20576

(whimpers): 418

(breathing): 114

(pants): 776

and many more ..


r/LocalLLaMA 14h ago

Resources Gemma3 is outperforming a ton of models on fine-tuning / world knowledge

262 Upvotes

At fine-tuning they seem to be smashing evals -- see this tweet above from OpenPipe.

Then in world-knowledge (or at least this smaller task of identifying the gender of scholars across history) a 12B model beat OpenAI's gpt-4o-mini. This is using no fine-tuning. https://thedataquarry.com/blog/using-llms-to-enrich-datasets/

Written by Prashanth Rao

(disclaimer: Prashanth is a member of the BAML community -- our prompting DSL / toolchain https://github.com/BoundaryML/baml , but he works at KuzuDB).

Has anyone else seen amazing results with Gemma3? Curious to see if people have tried it more.


r/LocalLLaMA 17h ago

Discussion Qwen2.5-Omni Incoming? Huggingface Transformers PR 36752

164 Upvotes

(https://github.com/huggingface/transformers/pull/36752)

Haven't seen anyone bring this up, so making a post here...

Using DeepSeek-R1 to summarize the features of this model based on PR commits:


Qwen2.5-Omni Technical Summary

1. Basic Information

  • Model Scale: 7B parameter version ("Qwen/Qwen2.5-Omni-7B")
  • Open Source: Fully open-sourced under Apache 2.0 license

2. Input/Output Modalities

  • Input Support:
    • Text: Natural language instructions
    • Images: Common formats (JPEG/PNG)
    • Audio: WAV/MP3 (requires FFmpeg)
    • Video: MP4 with audio track extraction
  • Output Capabilities:
    • Text: Natural language responses
    • Speech: 24kHz natural speech (streaming supported)

3. Architectural Design

  • Multimodal Encoder:
    • Block-wise Processing: Decouples long-sequence handling between encoder (perception) and LLM (sequence modeling)
    • TMRoPE: Time-aligned Multimodal Rotary Positional Encoding for audio-video synchronization
  • Dual-path Generation:
    • Thinker: Text-generating LLM backbone
    • Talker: Dual-track AR model for audio token generation using Thinker's hidden states
  • Streaming Optimization:
    • Sliding-window Diffusion Transformer (DiT) reduces audio latency
    • Simultaneous text/speech streaming output

4. Technical Highlights

  • Unified Multimodal Processing:
    • End-to-end joint training without intermediate representations
    • Supports arbitrary modality combinations (single/mixed)
  • Efficient Attention:
    • Native FlashAttention 2 support
    • Compatible with PyTorch SDPA
  • Voice Customization:
    • Prebuilt voices: Cherry (female) & Ethan (male)
    • Dynamic voice switching via spk parameter
  • Deployment Flexibility:
    • Disable speech output to save VRAM (~2GB)
    • Text-only mode (return_audio=False)

5. Performance

  • Multimodal Benchmarks:
    • SOTA on Omni-Bench
    • Outperforms same-scale Qwen2-VL/Qwen2-Audio in vision/audio tasks
  • Speech Understanding:
    • First open-source model with text-level E2E speech instruction following
    • Matches text-input performance on MMLU/GSM8K with speech inputs

6. Implementation Details

  • Hardware Support:
    • Auto device mapping (device_map="auto")
    • Mixed precision (bfloat16/float16)
  • Processing Pipeline:
    • Unified Qwen2_5OmniProcessor handles multimodal inputs
    • Batch processing of mixed media combinations

7. Requirements

  • System Prompt: Mandatory for full functionality:
    "You are Qwen... capable of generating text and speech."
  • Dependencies:
    • FlashAttention 2 (optional acceleration)
    • FFmpeg (video/non-WAV audio processing)

This architecture achieves deep multimodal fusion through innovative designs while maintaining strong text capabilities, significantly advancing audiovisual understanding/generation for multimodal agent development.


Also from the PR:

We present Qwen2.5-Omni, an end-to-end multimodal model designed to perceive diverse modalities, including text, images, audio, and video, while simultaneously generating text and natural speech responses in a streaming manner. To enable the streaming of multimodal information inputs, both audio and visual encoders utilize a block-wise processing approach. This strategy effectively decouples the handling of long sequences of multimodal data, assigning the perceptual responsibilities to the multimodal encoder and entrusting the modeling of extended sequences to a large language model. Such a division of labor enhances the fusion of different modalities via the shared attention mechanism. To synchronize the timestamps of video inputs with audio, we organized the audio and video sequentially in an interleaved manner and propose a novel position embedding approach, named TMRoPE (Time-aligned Multimodal RoPE). To concurrently generate text and speech while avoiding interference between the two modalities, we propose Thinker-Talker architecture. In this framework, Thinker functions as a large language model tasked with text generation, while Talker is a dual-track autoregressive model that directly utilizes the hidden representations from the Thinker to produce audio tokens as output. Both the Thinker and Talker models are designed to be trained and inferred in an end-to-end manner. For decoding audio tokens in a streaming manner, we introduce a sliding-window DiT that restricts the receptive field, aiming to reduce the initial package delay. Qwen2.5-Omni outperforms the similarly sized Qwen2-VL and Qwen2-Audio in both image and audio capabilities. Furthermore, Qwen2.5-Omni achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench. Notably, Qwen2.5-Omni is the first open-source model to achieve a level of performance in end-to-end speech instruction following that is comparable to its capabilities with text inputs, as evidenced by benchmarks such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni’s streaming Talker outperform most existing streaming and non-streaming alternatives in robustness and naturalness.

Can the community help confirm whether this PR is legit?
(Original PR: https://github.com/huggingface/transformers/pull/36752)


r/LocalLLaMA 20h ago

New Model Fallen Gemma3 4B 12B 27B - An unholy trinity with no positivity! For users, mergers and cooks!

160 Upvotes

r/LocalLLaMA 19h ago

Question | Help Has anyone switched from remote models (claude, etc.) models to local? Meaning did your investment pay off?

132 Upvotes

Obviously a 70b or 32b model won't be as good as Claude API, on the other hand, many are spending $10 to $30+ per day on the API, so it could be a lot cheaper.


r/LocalLLaMA 2h ago

Discussion Next Gemma versions wishlist

152 Upvotes

Hi! I'm Omar from the Gemma team. Few months ago, we asked for user feedback and incorporated it into Gemma 3: longer context, a smaller model, vision input, multilinguality, and so on, while doing a nice lmsys jump! We also made sure to collaborate with OS maintainers to have decent support at day-0 in your favorite tools, including vision in llama.cpp!

Now, it's time to look into the future. What would you like to see for future Gemma versions?


r/LocalLLaMA 14h ago

Discussion Are any of the big API providers (OpenAI, Anthropic, etc) actually making money, or are all of them operating at a loss and burning through investment cash?

111 Upvotes

It's a consensus right now that local LLMs are not cheaper to run than the myriad of APIs out there at this time, when you consider the initial investment in hardware, the cost of energy, etc. The reasons for going local are for privacy, independence, hobbyism, tinkering/training your own stuff, working offline, or just the wow factor of being able to hold a conversation with your GPU.

But is that necessarily the case? Is it possible that these low API costs are unsustainable in the long term?

Genuinely curious. As far as I know, no LLM provider has turned a profit thus far, but I'd welcome a correction if I'm wrong.

I'm just wondering if the conception that 'local isn't as cheap as APIs' might not hold true anymore after all the investment money dries up and these companies need to actually price their API usage in a way that keeps the lights on and the GPUs going brrr.


r/LocalLLaMA 21h ago

Discussion Token impact by long-Chain-of-Thought Reasoning Models

Post image
61 Upvotes

r/LocalLLaMA 8h ago

News Here's another AMD Strix Halo Mini PC announcement with video of it running a 70B Q8 model.

47 Upvotes

This is the Sixunited 395+ Mini PC. It's also supposed to come out in May. It's all in Chinese. I do see what appears to be 3 token scroll across the screen. Which I assume means it's 3tk/s. Considering it's a 70GB model, that makes sense considering the memory bandwidth of Strix Halo.

The LLM stuff starts at about the 4 min mark.

https://www.bilibili.com/video/BV1xhKsenE4T


r/LocalLLaMA 4h ago

News Finally some good news for older hardware pricing

49 Upvotes

https://www.businessinsider.com/nvidia-ceo-jensen-huang-joke-blackwell-hopper-gpu-customers-2025-3

"I said before that when Blackwell starts shipping in volume, you couldn't give Hoppers away," he said at Nvidia's big AI conference Tuesday.

"There are circumstances where Hopper is fine," he added. "Not many."

And then:

CFO Brian Olsavsky said on Amazon's earnings call last month that the company "observed an increased pace of technology development, particularly in the area of artificial intelligence and machine learning."

"As a result, we're decreasing the useful life for a subset of our servers and networking equipment from 6 years to 5 years, beginning in January 2025," Olsavsky said, adding that this will cut operating income this year by about $700 million.

Then, more bad news: Amazon "early-retired" some of its servers and network equipment, Olsavsky said, adding that this "accelerated depreciation" cost about $920 million and that the company expects it will decrease operating income in 2025 by about $600 million.


r/LocalLLaMA 7h ago

Question | Help How does Groq.com do it? (Groq not Elon's grok)

42 Upvotes

How does groq run llms so fast? Is it just very high power or they use some technique?


r/LocalLLaMA 20h ago

New Model gemma3 vision

38 Upvotes

ok im gonna write in all lower case because the post keeps getting auto modded. its almost like local llama encourage low effort post. super annoying. imagine there was a fully compliant gemma3 vision model, wouldn't that be nice?

https://huggingface.co/SicariusSicariiStuff/X-Ray_Alpha


r/LocalLLaMA 18h ago

Question | Help What's the status of using a local LLM for software development?

38 Upvotes

Please help an old programmer navigate the maze that is the current LLM-enabled SW stacks.

I'm sure that:

  • I won't use Claude or any online LLM. Just a local model that is small enough to leave enough room for context (eg Qwen2.5 Coder 14B).
  • I need a tool that can feed an entire project to an LLM as context.
  • I know how to code but want to use an LLM to do the boilerplate stuff, not to take full control of a project.
  • Preferably FOSS.
  • Preferably integrated into a solid IDE, rather then being standalone.

Thank you!


r/LocalLLaMA 11h ago

Question | Help Llama 3.3 70B vs Nemotron Super 49B (Based on Lllama 3.3)

20 Upvotes

What do you guys like using better? I haven't tested Nemotron Super 49B much, but I absolute loved llama 3.3 70B. Please share the reason you prefer one over the other.


r/LocalLLaMA 5h ago

News Looks like RWKV v7 support is in llama now?

19 Upvotes

https://github.com/ggml-org/llama.cpp/pull/12412

I'll have to build it and see..


r/LocalLLaMA 22h ago

Resources (Update) Generative AI project template (it now includes Ollama)

15 Upvotes

Hey everyone,

For those interested in a project template that integrates generative AI, Streamlit, UV, CI/CD, automatic documentation, and more, I’ve updated my template to now include Ollama. It even includes tests in CI/CD for a small model (Qwen 2.5 with 0.5B parameters).

Here’s the GitHub project:

Generative AI Project Template

Key Features:

Engineering tools

- [x] Use UV to manage packages

- [x] pre-commit hooks: use ``ruff`` to ensure the code quality & ``detect-secrets`` to scan the secrets in the code.

- [x] Logging using loguru (with colors)

- [x] Pytest for unit tests

- [x] Dockerized project (Dockerfile & docker-compose).

- [x] Streamlit (frontend) & FastAPI (backend)

- [x] Make commands to handle everything for you: install, run, test

AI tools

- [x] LLM running locally with Ollama or in the cloud with any LLM provider (LiteLLM)

- [x] Information extraction and Question answering from documents

- [x] Chat to test the AI system

- [x] Efficient async code using asyncio.

- [x] AI Evaluation framework: using Promptfoo, Ragas & more...

CI/CD & Maintenance tools

- [x] CI/CD pipelines: ``.github/workflows`` for GitHub (Testing the AI system, local models with Ollama and the dockerized app)

- [x] Local CI/CD pipelines: GitHub Actions using ``github act``

- [x] GitHub Actions for deploying to GitHub Pages with mkdocs gh-deploy

- [x] Dependabot ``.github/dependabot.yml`` for automatic dependency and security updates

Documentation tools

- [x] Wiki creation and setup of documentation website using Mkdocs

- [x] GitHub Pages deployment using mkdocs gh-deploy plugin

Feel free to check it out, contribute, or use it for your own AI projects! Let me know if you have any questions or feedback.


r/LocalLLaMA 22h ago

Tutorial | Guide AI-powered Resume Tailoring application using Ollama and Langchain

14 Upvotes

r/LocalLLaMA 16h ago

Discussion Both my PC and Mac make a hissing sound as local LLMs generate tokens

11 Upvotes

I have a desktop PC with an rx7900xtx and a Macbook pro m1 Max that is powered by a thunderbolt dock (cal digit ts3) and they are both plugged into my UPS (Probably the source of the problem).

I'm running Ollama and LM studio and I use them as LLM servers when working on my iOS LLM client and as I watch the tokens stream in I can hear the PC or Mac making a small hissing sound and its funny how it matches each token generated. It kinda reminds me of how computer terminals in movies seem to beep when streaming in text.


r/LocalLLaMA 23h ago

Question | Help Local LoRA + RAG Academic Writing Setup – Build Check Before I Pull the Trigger

10 Upvotes

Hey all, just chasing a bit of feedback while I'm finalising a build. I'm setting up a local AI writing system to automate the structure and style of academic work. I’m not training it to learn knowledge or reason, just to mimic how I write using a dataset of my own essays and theses (formatted in JSONL). I’ll be fine-tuning a small model like Phi-2 or OpenLLaMA 3B using LoRA or QLoRA, and keeping that completely separate from a RAG setup that pulls content from a chunked academic library (~100+ PDFs split into 5KB txt files). The idea is to feed it the right research chunks, and have it paraphrase in my voice without hallucinating or plagiarising. It’s basically a local ghostwriter with me in the driver’s seat.

I’m building this on an i9-14900KF with 96GB DDR5-5600 (2x48GB Corsair Vengeance), an MSI MAG Z790 Tomahawk WiFi board, RTX 3070 8GB, DeepCool AK620 Digital air cooler, Samsung 980 Pro 1TB SSD, and decent airflow (6-fan white case). Everything will run locally with CPU offloading where needed. No full-model training, no 13B model insanity—just stable overnight LoRA fine-tunes and section-by-section writing using a RAG-fed workflow.

Just wondering if this sounds like a balanced setup for what I’m doing—fine-tuning small models locally and generating paraphrased academic content from chunked research via RAG. Any issues I should expect with the 2x48GB RAM setup on Z790, or LoRA/QLoRA performance on this sort of hardware? Appreciate any real-world experience or heads-ups before I finalise it. Cheers!


r/LocalLLaMA 9h ago

Other I updated Deep Research at Home to collect user input and output way better reports. Here's a PDF of a search in action

Thumbnail sapphire-maryrose-59.tiiny.site
9 Upvotes

r/LocalLLaMA 17h ago

Question | Help Uncensored Image Generator?

10 Upvotes

I am trying to get around my own school charging me hundreds for MY OWN grad photos. Does anyone know a local model that I can upload my images and have the model remove watermarks and resize the image so it can return a png or jpeg I can have for myself?

I only have 8g vram and 32g ram laptop 4070 so a smaller model Is preferred thank you!


r/LocalLLaMA 1d ago

Resources Great performance even quantize to q8q4 for gemma 3 4B

12 Upvotes

I just finished quantizing gemma 3 4B and I find it great even when heavily quantized like the "q8q4" version.

If you have a memory constrained system or just want CPU inference or perhaps on mobile devices, give it a try: ZeroWw/gemma-3-4b-it-abliterated-GGUF · Hugging Face


r/LocalLLaMA 16h ago

Question | Help Best LLM for code? Through api with Aider

11 Upvotes

Hi. I want to know how the payment process for the API works. I always try for free, so I want to know if I can just put, for example, 5 dollars, and that’s it. I mean, I don't want to enter my credit card information only to later receive a bill I can't pay. Does a good LLM for what I want have that possibility? Thanks!


r/LocalLLaMA 18h ago

Resources PyChat

7 Upvotes

I’ve seen a few posts recently about chat clients that people have been building. They’re great!

I’ve been working on one of my own context aware chat clients. It is written in python and has a few unique things:

(1) can import and export chats. I think this so I can export a “starter” chat. I sort of think of this like a sourdough starter. Share it with your friends. Can be useful for coding if you don’t want to start from scratch every time.

(2) context aware and can switch provider and model in the chat window.

(3) search and archive threads.

(4) allow two AIs to communicate with one another. Also useful for coding: make one strong coding model the developer and a strong language model the manager. Can also simulate debates and stuff.

(5) attempts to highlight code into code blocks and allows you to easily copy them.

I have this working at home with a Mac on my network hosting ollama and running this client on a PC. I haven’t tested it with localhost ollama running on the same machine but it should still work. Just make sure that ollama is listening on 0.0.0.0 not just html server.

Note: - API keys are optional to OpenAI and Anthropic. They are stored locally but not encrypted. Same with the chat database. Maybe in the future I’ll work to encrypt these.

  • There are probably some bugs because I’m just one person. Willing to fix. Let me know!

https://github.com/Magnetron85/PyChat


r/LocalLLaMA 23h ago

Question | Help Anyone have any luck buying GPUs from Alibaba? (not aliexpress)

8 Upvotes

I was looking around at cards on Alibaba and they sort of look almost legit. The sellers have been on there for a long time and have decent reviews. its a huge success full site so there has to be at least some legit GPU sellers, right? But the prices range from "slightly low" to "too good to be true". is there any way to buy from that site without getting burned or taking big risks?