Is it possible to run Pyspark via Yarn-cluster mode in Jupyter in such a way it returns results locally in a User Interface? As i understand it this is only possible in yarn-client mode. The problem i have is multiple none technical users will use a UI to input a few params which will feed into my code. This will then execute on a single remote server in yarn-client mode (So the single remote sever connecting to a cluster). So i have concerns it will be over whelmed in yarn-client mode however conversely in yarn-cluster mode i dont think it would be possible to return results to the UI.
Any thoughts much appreciated.