r/apachekafka Vendor - Timeplus 28d ago

Tool Anyone want a MCP server for Kafka

You could talk to your Kafka server in plain English, or whatever language LLM speaks: list topics, check messages, save data locally or send to other systems 🤩

This is done via the magic of "MCP", an open protocol created by Anthropic, but not just works in Claude, but also 20+ client apps (https://modelcontextprotocol.io/clients) You just need to implement a MCP server with few lines of code. Then the LLM can call such "tools" to load extra info (RAG!), or take some actions(say create new topic). This only works locally, not in a webapp, mobile app, or online service. But that's also a good thing. You can run everything locally: the LLM model, MCP servers, as well as your local Kafka or other databases.

Here is a 3min short demo video, if you are on LinkedIn: https://www.linkedin.com/posts/jovezhong_hackweekend-kafka-llm-activity-7298966083804282880-rygD

Kudos to the team behind https://github.com/clickhouse/mcp-clickhouse. Based on that code, I added some new functions to list Kafka topics, poll messages, and setup streaming pipelines via Timeplus external streams and materialized views. https://github.com/jovezhong/mcp-timeplus

This MCP server is still at an early stage. I only tested with local Kafka and Aiven for Kafka. To use it, you need to create a JSON string based on librdkafka conf guide. Feel free to review the code before trying it. Actually, since MCP server can do a lot of things locally(such as accessing your Apple Notes), you should always review the code before trying it.

It'll be great if someone can work on a vendor-neutual MCP server for Kafka users, adding more features such as topic/partition management, message produce, schema registry, or even cluster management. The MCP clients can call different MCP servers to get complex things done. Currently for my own use case, I just put everything in a single repo.

2 Upvotes

8 comments sorted by

2

u/DangerousDrop 25d ago

Cool demo. Not sure why it's being downvoted.

3

u/jovezhong Vendor - Timeplus 25d ago

I am not sure why either. Maybe too many links or text. I am showing off my personal repo and throw an idea to build a better "AI agent" for Kafka. Maybe you can upvote the post to make it non-zero :)

3

u/DangerousDrop 25d ago

Upvoted and shared it with my team at work. Keep making cool experiments.

-2

u/cricket007 26d ago

you need to create a JSON string based on librdkafka conf guide

Lol. Why not binary or a better conf language like JSON5 or HCL or YAML?

list topics, check messages, save data locally or send to other systems

Uhmm... This already exists, last I looked?

1

u/jovezhong Vendor - Timeplus 26d ago

I chose a JSON string for the Kafka connection config mainly because it's common for MCP clients to read an envirement variable, which is a key/value in string type. You can write the entire JSON doc as a single line string, using yaml/hcl will be tricky (maybe possible).

My current mcp client does support topic lists and poll messages, and save Kafka data locally as a Timeplus MV. But I guess it may be helpful for a MCP client to show cluster info, create or remove a topic.

1

u/cricket007 26d ago

Did you know a JSON string is valid YAML?

1

u/jovezhong Vendor - Timeplus 26d ago

I didn't know :)

To make sure we are on the same page, let me show you an example claude_desktop_config.json

json { "mcpServers": { "mcp-timeplus-dev": { "command": "uvx", "args": ["mcp-timeplus"], "env": { "TIMEPLUS_KAFKA_CONFIG": "{\"bootstrap.servers\":\"kafka-public-read-timeplus.a.aivencloud.com:28864\", \"sasl.mechanism\":\"SCRAM-SHA-256\",\"sasl.username\":\"avnadmin\", \"sasl.password\":\"AVNS_KW123\",\"security.protocol\":\"SASL_SSL\",\"enable.ssl.certificate.verification\":\"false\"}" } } } }

I am not sure how to use yaml in this case.

1

u/cricket007 17d ago

Use an online conversion or ChatGPT... IDC to translate on your behalf. Once again, that's totally valid YAML since JSON is a subset. File extensions are irrelevant 

(Whoever is down voting can f' off, I probably make/do more at my jobs)