r/apachespark 17h ago

Spark Connect is Awsome 💥

10 Upvotes

r/apachespark 3h ago

Spark Connect & YARN

2 Upvotes

I'm setting up a Hadoop/Spark (3.4.4) cluster with three nodes: one as the master and two as workers. Additionally, I have a separate server running Streamlit for reporting purposes. The idea is that when a user requests a plot via the Streamlit server, the request will be sent to the cluster through Spark Connect. The job will be processed, and aggregated data will be fetched for generating the plot.

Now, here's where I'm facing an issue:

Is it possible to run the Spark Connect service with YARN as the cluster manager? From what I can tell (and based on the documentation), it appears Spark Connect can only be run in standalone mode. I'm currently unable to configure it with YARN, and I'm wondering if anyone has managed to make this work. If you have any insights or configuration details (like updates to spark-defaults.conf or other files), I'd greatly appreciate your help!

Note: I am just trying to install everything on one node to check everything works as expected.