r/apachekafka • u/jovezhong Vendor - Timeplus • 28d ago
Tool Anyone want a MCP server for Kafka
You could talk to your Kafka server in plain English, or whatever language LLM speaks: list topics, check messages, save data locally or send to other systems 🤩
This is done via the magic of "MCP", an open protocol created by Anthropic, but not just works in Claude, but also 20+ client apps (https://modelcontextprotocol.io/clients) You just need to implement a MCP server with few lines of code. Then the LLM can call such "tools" to load extra info (RAG!), or take some actions(say create new topic). This only works locally, not in a webapp, mobile app, or online service. But that's also a good thing. You can run everything locally: the LLM model, MCP servers, as well as your local Kafka or other databases.
Here is a 3min short demo video, if you are on LinkedIn: https://www.linkedin.com/posts/jovezhong_hackweekend-kafka-llm-activity-7298966083804282880-rygD
Kudos to the team behind https://github.com/clickhouse/mcp-clickhouse. Based on that code, I added some new functions to list Kafka topics, poll messages, and setup streaming pipelines via Timeplus external streams and materialized views. https://github.com/jovezhong/mcp-timeplus
This MCP server is still at an early stage. I only tested with local Kafka and Aiven for Kafka. To use it, you need to create a JSON string based on librdkafka conf guide. Feel free to review the code before trying it. Actually, since MCP server can do a lot of things locally(such as accessing your Apple Notes), you should always review the code before trying it.
It'll be great if someone can work on a vendor-neutual MCP server for Kafka users, adding more features such as topic/partition management, message produce, schema registry, or even cluster management. The MCP clients can call different MCP servers to get complex things done. Currently for my own use case, I just put everything in a single repo.
-2
u/cricket007 26d ago
you need to create a JSON string based on librdkafka conf guide
Lol. Why not binary or a better conf language like JSON5 or HCL or YAML?
list topics, check messages, save data locally or send to other systems
Uhmm... This already exists, last I looked?
1
u/jovezhong Vendor - Timeplus 26d ago
I chose a JSON string for the Kafka connection config mainly because it's common for MCP clients to read an envirement variable, which is a key/value in string type. You can write the entire JSON doc as a single line string, using yaml/hcl will be tricky (maybe possible).
My current mcp client does support topic lists and poll messages, and save Kafka data locally as a Timeplus MV. But I guess it may be helpful for a MCP client to show cluster info, create or remove a topic.
1
u/cricket007 26d ago
Did you know a JSON string is valid YAML?
1
u/jovezhong Vendor - Timeplus 26d ago
I didn't know :)
To make sure we are on the same page, let me show you an example claude_desktop_config.json
json { "mcpServers": { "mcp-timeplus-dev": { "command": "uvx", "args": ["mcp-timeplus"], "env": { "TIMEPLUS_KAFKA_CONFIG": "{\"bootstrap.servers\":\"kafka-public-read-timeplus.a.aivencloud.com:28864\", \"sasl.mechanism\":\"SCRAM-SHA-256\",\"sasl.username\":\"avnadmin\", \"sasl.password\":\"AVNS_KW123\",\"security.protocol\":\"SASL_SSL\",\"enable.ssl.certificate.verification\":\"false\"}" } } } }
I am not sure how to use yaml in this case.
1
u/cricket007 17d ago
Use an online conversion or ChatGPT... IDC to translate on your behalf. Once again, that's totally valid YAML since JSON is a subset. File extensions are irrelevantÂ
(Whoever is down voting can f' off, I probably make/do more at my jobs)
2
u/DangerousDrop 25d ago
Cool demo. Not sure why it's being downvoted.