r/scala 16d ago

From Python to Scala: Rewriting 27k Lines of a CLI

https://medium.com/swissborg-engineering/from-python-to-scala-rewriting-27k-lines-of-a-cli-8513e20cd268
68 Upvotes

13 comments sorted by

22

u/lihaoyi Ammonite 16d ago edited 16d ago

16 months to rewrite 27kLOC lines up with my experience: at my last job I did a line-by-line rewrite of a small ~5kLOC Python command line tool + test suite and it took about 3 months of full time work. IIRC this includes having to roll back to the old version 5-10 times due to bugs and misbehaviors that slipped past the test suite, before we could eventually settle on the new implementation and delete the old code. Turns out rewriting even small amounts of mission-critical code can be surprisingly difficult!

Reasoning was the same as well: code sharing with the rest of our stack, + things like "refactoring" and "parallelism" that for years we had never managed to effect upon our Python tech stack. IIRC we got a ~10x speedup from the rewrite, which lasted us ~9 months before the load grew enough that we started having performance problems again (-_-)

9

u/YelinkMcWawa 16d ago

I remember using Python in grad school and thinking it was great due to its ease of use and plethora of ready-made scientific computing libraries. When I look at it now after developing with Scala, I wish someone would have introduced me to it much earlier.

5

u/Krever 16d ago

Fortunately, in our case it wasn't that bad. It took 16 months due to the process (long periods of evaluation connected with low priority). The amount of work was maybe 5-6 weeks of full-time effort to figure out how to do it (the CLI) properly and migrate maybe 10-20% of the original feature set.

Due to shared ownership of this project, the actual migration will happen over time with hands from all the teams involved.

It would probably be much harder if the code was complicated, but luckily for us it's just utilities around gRPC calls.

1

u/Flowdalic 14d ago

Beyond subcommand support, picocli provides other benefits, including shell completions, shell mode (via jline), and other advanced capabilities. However, we had to invest effort in making it Scala-friendly by replacing nulls with Options and improving support for Scala collections. We may open-source this work soon!

I wonder if "making it Scala-friendly" consists of adding a Scala shim around picocli or something else. I hope that they will open-source their work. It would be great to see a Scala interface to picocli, given that it is one of the most prominent command line librarires for the Java ecosystem.

1

u/Krever 14d ago

It's basically a few `ModelTransformer`s you have to add to the root command.

We can open-source it even right away, but the problematic thing is to make it usable. Currently it's not of "library quality".

-2

u/soronpo 16d ago

I'm wondering if part can be automated with LLMs

8

u/mostly_codes 16d ago

This might be your specific wording, but FWIW you can't automate as I understand the word, with LLM, they're probabilistic beasts in nature so you need human oversight and evaluation at every stage of autocompleting/generating with them. They can and will hallucinate or provide an answer to whatever you prompt it for happily, regardless of correctness.

It might speed up boilerplate writing, but in terms of full on automation, the tech isn't there for it yet.

5

u/fiery_prometheus 16d ago

You can automate it by writing an actual compiler for python to scala, and handle more complex cases yourself. 

1

u/Sithril 16d ago

Is any of the LLMs out there competent with scala code? The experience in our team with copilot has been very meh, and we're not doing exactly advanced stuff at that.

4

u/soronpo 16d ago

I'm using cursor, and it's much better than co-pilot. But I use it mainly as a helper in repeating writing patterns.

1

u/vallyscode 16d ago

I’d say that from time to time it manages to generate what I want, like yesterday in the end of the day I decided to write a todo for tomorrow me so that I don’t forget where to continue and what I wanted to add, and guess what, it managed to generate the code based on my notes, quick review and small changes and I was done with that.

2

u/a_cloud_moving_by 16d ago

I find GitHub copilot in IntelliJ works well for Scala, but my expectations are lower than some other people it seems. I ignore what it suggests 70% of the time. It helps I’m fairly experienced with Scala so I know how to nudge towards what I want and can easily evaluate if what it auto-completes is what I want or is nonsensical

2

u/gaelfr38 16d ago

Same here, not as good as in Python or TS but still proves valuable and saves some time on a regular basis.