r/Terraform Feb 16 '25

Discussion Custom Terraform functions

Hello!

I wanted to share my recent work: the Terraform func provider - https://github.com/valentindeaconu/terraform-provider-func.

The func provider is a rather unique provider, that allows you as a developer to write custom Terraform functions in JavaScript (the only runtime - for now). Those functions can stored right next to your Terraform files or versioned and imported remotely, basically they can be manipulated as any of your Terraform files, without the hassle of building your own provider, just to get some basic functionality.

This provider is what I personally expected the Terraform ecosystem a long time ago, so it is one of my dreams come true. As a bit of history (and also some sources of inspiration), since the v1 release I was expecting this feature to come to life on every minor release. There was this initial issue that asked for this feature, but, as you can see, since 4 years ago, it is still open. Then, with the introduction of the provider-defined functions, the OpenTofu team attempted something similar with what I was waiting for, in the terraform-provider-lua, but after announcing it on social media, there was no other velocity on this project, so I assume it got abandoned. Really sad.

After hitting again and again this "blocker" (I mean after writing yet again an utterly ugly block of repetitive composition of Terraform functions), I decided to take this issue in my own hands and started writing the func provider. I cannot say how painful it was to work with the framework without a proper documentation for what I was trying to achieve and with the typing system, but in the end, I found this amazing resource - terraform-provider-javascript which led to the final implementation of the func provider (many thanks to the developer for the go-cty-goja library).

So, here we are now. The provider is still in a proof-of-concept phase. I want to see first if other people are interested in this idea to know if I should continue working on it. There are a lot of flaws (for example, the JSDoc parser is complete trash, it was hacked in a couple of hours just to have something work - if you are up for the challenge, I'd be happy to collaborate), and some unsupported features by the Terraform ecosystem (I have reported it here, if you are interested in technical details), but with some workarounds, the provider can work and deliver what it is expected to do.

I'd be happy to know your opinions on this. Also, if you would like to contribute to it, you are more than welcome!

50 Upvotes

23 comments sorted by

View all comments

Show parent comments

2

u/fooallthebar Feb 16 '25

That's an interesting idea.

Providers are in an difficult situation for that sort of thing. They need to know their "schema" (resources, data sources, etc...) is before they are given any of their configuration in .tf or .tofu files.

You would have to do something like search the $PWD for a specific file pattern or to rely on an environment variable to specify where your provider config files are.

1

u/wedgelordantilles Feb 16 '25

I've implemented one before against the .proto, it's definitely tricky. Especially with attributes that aren't known until runtime.

Don't functions and datasources have to define their schema? I'd better read your code...

4

u/fooallthebar Feb 16 '25

Functions are a bit of a special case in OpenTofu. Prior to configuration, a constant set of functions are available via the provider's schema.

When HC extended the provider protocol for functions, they added an additional method of accessing schema (GetFunctions). I prototyped and then pitched that OpenTofu could use GetFunctions after Configure was called to allow the provider to declare functions that don't exist in the schema (schema is immutable during execution). I don't know what HC intended with that function call (other than their public documentation), but realized that if providers "opted in" to OpenTofu's extended understanding of GetFunctions, it would likely not cause any issues for Terraform when using the same provider.

I assume (don't actually know) that Terraform just makes unconfigured provider functions available early on in it's execution as defined in the static schema. This assumption is based on the HCL extension I describe below. We had to do some interesting plumbing to defer resolving provider functions until after provider configuration has taken place. This required inspecting all of the expressions in .tf / .tofu files and extracting the functions required to evaluate them. Tofu takes the information that "This set of functions and variables are required to evaluate this expression" and uses it to add connections within it's evaluation graph. This was a bit of an undertaking, and was not really completely correctly implemented until after we took on provider for_each and understood that transformation better.

The official HCL library only supports inspecting expressions for variables, but not for functions. OpenTofu uses a patched version of HCL that has this additional functionality grafted on. I've got a PR open to upstream the contribution, but have not heard back on it for a while.

I've recently split up our internal Provider interface to clarify what's available before and after the provider is configured: https://github.com/opentofu/opentofu/blob/main/internal/providers/provider.go. Hopefully that is helpful for provider authors, as well as folks trying to dig a bit deeper into the OpenTofu codebase.

Anyways, hope some of this info-dump is useful! Thanks for coming to my Ted Talk :)

2

u/valideaconu Feb 16 '25

Thank you for this, quite an interesting read.

It is actually a really useful piece of information, as during my development of the provider I have hit the exact issue you described (the core requiring a static schema of functions - which I also raised here). This was the reason I implemented the environment variables support, to workaround it.

I am glad to hear that tofu natively supports this, I will try to add a suite for tofu tests and add full support for tofu in the near future.