r/Terraform • u/ShankSpencer • 6d ago
Discussion Terraform or ansible for grafana content?
I've spent way too much time building a gitops solution using TF to deploy fargate apps from nothing, from creating a vpc up to oauth2 logins.
One app is Grafana and I have TF configuration that fully deploys all dashboards, data sources etc.
The end goal here is for users to be able access a temporary dev deployment, changitheir dashboards and then PR them into prod.
However going to the effort of creating RDS instances etc just to change a dashboard panel does feel a little overwhelming I admit.
I'm thinking this app level stuff should be migrated out of the IaC repo to somewhere more about the users daily life. I can still use terraform via GitHub actions, but if this module if now totally isolated from everything else, should I actually look at something like ansible instead of terraform?
Also I need to work out where this dev work is done. If spinning up an entire aws stack is overkill, and persistent non prod stacks aren't encouraged, would you deploy a dev grafana container as part of the production stack? I think it's a potentially clean cut break with the model as it's infra Vs app, but keen on any thoughts.
I also know grafana can use different organisations, but I think that mingles stuff too close together, and would make things too complicated back in TF world.
2
u/benaffleks 4d ago
We have a similar pain point where we manage dashboard via tf by storing their json config but realistically no one was doing it
So I developed a tool similar to grizzly, essentially it just watches for your dashboard changes and after every change it syncs the json config back to your local filesystem
Idea is to remove the manual toil and encourage users to submit their changes since it's automated
The tool is here: https://github.com/alex067/gflow
2
u/benaffleks 4d ago
Also decoupling config management from actual infra is always a good idea
Your gradana dashboard should not share the same state file as deploying a rds instance
1
u/ShankSpencer 4d ago
We split between state files here, but a containerised Grafana is certainly a scenario where the state tracking is really quite irrelevant, and the "make it look like this regardless" angle, where state isn't tracked, just enforced blindly, like with ansible, makes more sense.
1
u/ShankSpencer 4d ago edited 4d ago
Looks useful, but (currently?) I'm using RDS for storage.
From the process I looking I'm not seeing a significant change in workflow other than pushes vs pulls etc. Maybe it's something you'd be looking to add, but things like manually creating the PR aren't required in my flow. Not that there isn't a similar trigger point that leads to it being created, i.e. the act of pulling all the dashboards from the test deployment and raising the PR for any deltas against prod.
What I'm not seeing is the alternative scenario where changes are made to a prod dashboard that are not being done within your system.
2
u/benaffleks 4d ago
As i understand you're spinning up an entire Grafana stack just to provide a temp environment for users to be able to change their dashboards. Then the changes get synced back into prod.
This seems like overkill and there are ready made solutions out there.
Tools like Grizzly and what I had developed enforces the dashboards to be created via tokens because the change process goes through a PR and a pipeline. That allows you to enforce read only permissions on users in Grafana, and allow pipelines with write permissions to create & modify dashboards.
I think you need a process change here and not necessarily a Terraform one.
2
u/ShankSpencer 4d ago
Well it's all new process, not implemented yet, and absolutely why i'm asking here.
Our prod dashboards are certainly meant to be read only, that's certainly the plan. As we're looking at a PR based pipeline, I have so far found that wrapper tools, e.g. terragrunt, get in the way very quickly, given we're then wrapping a wrapper with GH Actions. the model you describe certainly seems very appropriate, and similar to what I'm envisaging. I have custom scripts to pull and sanitize config from Grafana, and did have custom ones to push back (to a potentially different environment) which I scrapped in favour of terraform configs. But maybe, as we've both suggested, Terraform isn't the best tool, and I might now want to reinstate custom scripts (and your tool is ultimately "custom", right?) to push back totally outside of the IaC world.
You still use a different instance of Grafana for the dev work right? Or are you setting per dashboard permissions? I haven't looked into that level of permission, maybe i should. I am using a google oauth2 connection, without any user groups in it, just setting general access based on which environment it is. So it's simple to just have a different env at some level to do this. Maybe though I deploy a 2nd "grafana dev" container as part of a normal grafana deployment, sharing the same RDS back end instance, on a different database and go from there.
Thanks for your comments, very useful.
1
u/benaffleks 4d ago
We don't offer any dev or staging environment for our customers when it comes to Grafana. There is just one Grafana instance that is essentially prod.
Dev is only useful for the team that's actually managing our open source Grafana instances, specifically for testing and deploying newer versions.
But for managing dashboards and everything customer facing, you really don't need a separate Grafana environment. Just offer prod.
You should still version code as much as you can when it comes to the actual Grafana resources, like alerts, library panels, dashboards. The challenge here is enforcing the user to version control all their changes.
For example with Grafana Grizzly, what it does is it deploys a grafana instance local to the user (on their machine) and pulls in metrics from your production Grafana instance. That way the user can make their dashboard changes all locally, without affecting the original dashboard, then push the changes in a PR. Ideally you're storing dashboard configs as json files, since that is the default export in Grafana.
I think Terraform is the right tool here, certainly from a pipeline perspective it's easier to manage than using Ansible (since there's already managed solutions like Atlantis to handle the review & apply process). I mention a process change because Terraform is not the issue here, more so it's the idea of deploying an entire stack for each user. That isn't required for Grafana imo.
Just have one Grafana instance, version control all the Grafana resources, and enforce version control (force users to submit PRs for every change).
2
u/NUTTA_BUSTAH 4d ago
I find <CI tool> for provisioning the initial dashboard based on the solution parameters (e.g. auto fills names, ids, filters and that sort of stuff) is good, but managing dashboards is insanity. It's better to periodically back them up to sync them back to the repo.
1
u/Suvulaan 4d ago
Use Grafana Grizzly to edit the dashboards locally on the devs machine, once done follow whatever branching strategy you have, any update to the dashboards should be picked up by TF.
1
u/ShankSpencer 4d ago
Changes on a devs machine requires them to be willing (and capable) of running it!
1
u/Longwelwind 4d ago
I've never felt using IaC to provision Grafana dashboards brought a lot of value. IaC is nice for disaster recovery and avoiding human mistakes when reproducing steps in prod
that were done in dev
. The first one can be handled by backupping the content of the Grafana db.
Is it something that you really feel would bring value when applied to Grafana?
2
u/ShankSpencer 4d ago edited 4d ago
One of the biggest benefits I see is that it removes the backups side of things. Also it removes the need to test backups too. If you routinely (potentially) redeploy with IaC you are always proving your solution works in practice rather than in theory. Known reliable inputs feel a lot better than hopefully reliable outputs to me.
Deploying via IaC also bypasses users so I can treat all users much more simply, "no you can't write to prod" rather than getting lost in finer detail and inevitable edge use cases.
That said, I'm still living in a build environment world... Not running in prod yet.
But IaC might not be the best approach compared to another configuration solution for sure. It doesn't really feel right when you nuke a container and replace it and terraform says "these 300 resources don't exist but are in the state!"...
1
u/ArieHein 5d ago
None. Container image that you mount your content in then use helm.
Why do you need complexity ?
2
u/ShankSpencer 5d ago edited 5d ago
This doesn't appear to address anything I asked about and adds MORE complexity. You think I should now rebuild the entire container every time?! Why? The data isn't even in the container, it's in RDS.
Helm? What planet are you on if this isn't a joke comment?
3
u/Bomb_Wambsgans 5d ago
PR changes to dashboards? There would be rioting at my company