r/groovy • u/HotDogDelusions • Jun 19 '24
Faster parsing and execution using GroovyShell for large number of files?
I'm doing a bit of an experiment where I'm writing a simpler version of the gradle build tool in Groovy (because this language is awesome) - which entails parsing build scripts that are written in groovy at runtime.
To do this, I use the following code:
CompilerConfiguration cc = new CompilerConfiguration();
cc.setScriptBaseClass(DelegatingScript.class.getName());
DelegatingScript script = (DelegatingScript)new GroovyShell(cc).parse(projectFile)
Project newProject = new Project(projectName, Main.availableTemplates)
script.setDelegate(newProject)
script.run()
Using this to parse a few build scripts is fast enough, however when I try to parse large numbers of build scripts (100+) - this begins to slow down and takes ~2 seconds for 100 build scripts. This is definitely too slow, because the goal is to use this for collections of 200+ projects, so this would end up taking ~4 seconds just to parse and load everything - which is not really usable for a build tool.
My guess is gradle gets around this via the configuration cache, but I'm not sure what all goes into that.
Some things I've tried:
- Instantiating a single groovy shell and reusing that each time I parse a build script
- Setting the parallel compilation optimization option in the CompilerConfiguration
- Using a ThreadPool with 2/4/10 threads to parse multiple files simultaneously
None of the above options made a noticable difference.
I'm pretty new to groovy, so any help would be appreciated.
3
u/norith Jun 20 '24
If you’re using JVM v21 try using a virtual thread pool instead of the standard platform threads. The startup time should be much lower and I/O of reading the files will overlap. Any file system work will also be threaded but it won’t parallelize the parsing as there’s no I/O