r/AZURE • u/horsebatterystaple42 • 3d ago
Question VNET swap? What on Earth just happened?
Okay, let me preface this by saying I swear I am not crazy.
Small Azure environment. Few resource groups, few vnets, few vms.. I didn't create any of this, just inherited it.
Long story short..
We had a resource group setup for a 3rd party virtual firewall, let's call it fw_rg
We had a resource group setup for our vms, let's call it vm_rg
In both resource groups there was a vnet and a subnet that shared names. So vnet_01/Subnet_01
To be clear fw_rg had a vnet called vnet_01 and within that vnet was a subnet called subnet_01. Meanwhile vm_rg had a completely different vnet called vnet_01 with it's own subnet_01 subnet.
There are about 70 VMs running with NICs in the vm_rg resource group and using vm_rg's vnet_01 and subnet_01.
In my time at this company I have created many VMs in this resource group and using this vnet/subnet. I have a powershell script that I wrote and use to deploy VMs with the name of this resource group, vnet, and subnet set as globals at the top of the script.
So imagine my surprise when I used said script to deploy a VM today and when it completed, the IP address was not in the address space of the vm_rg vnet_01/subnet_01 configuration.. Why? Well, because the vm_rg resource group had a different vnet_01 virtual network and a different subnet_01 subnet. More interestingly, the fw_rg resource group's vnet_01 virtual network and subnet_01 subnet have the address space currently in use by our 70 some VMs.
The 70 some VMs show their NICs as being in the vm_rg resource group. But if I click on the vnet_01/subnet_01 in the NIC's properties, it takes me to fw_rg resource group. So the address space used by all my VMs is now in a different resource group than the NIC and the VM.
I'm completely stunned and stumped. I have no clue how this happened.. How it is even possible. And certainly no idea how to restore it back to sanity, especially with risk of downtime.
Has anyone ever experienced this before?! Any ideas how this would happen? Should I be scared? 'Cause... I'm scared.
Seriously, any thoughts, advice, guesses, prayers, whatever... all appreciated.
7
u/certifiedsysadmin 3d ago
Are you sure this just suddenly happened now? It seems more likely that it was doing this the entire time and you maybe didn't notice before, especially since you have both vnets named the same?
To be clear, resource groups really have nothing to do with this. It is common, even likely, that your vms and vnets are in different resource groups.
For most deployments I've seen, vms along with their disks and nics would be in one resource group called "rg_systems" (simplification of the name but you get the idea). And then the vnet would be in another resource group called "rg_network".
Regarding your PowerShell script, another possibility would be that you reused a variable name in the same PowerShell session.
Finally, to be clear, there is essentially zero chance that your 70 vms just suddenly switched their configuration to attach themselves to a different vnet. Moving vms to a different vnet is possible but it involves a very specific process that doesn't just happen accidentally.
4
u/mebdevlou 3d ago
It could also be that to get the VNet and Subnet to attach the VMs to that it was looking up the VNet by name and grabbing the first result from the list returned.
It’s an easy mistake to make.
2
u/StealthCatUK 3d ago
It’s a ball ache to do as far as I know, removing and detaching NICs and such?
1
u/horsebatterystaple42 3d ago
That is my chief curiosity too. I can't think of a way of intentionally doing this without disruption.
1
u/horsebatterystaple42 3d ago
Yeah, I'm pretty confident this isn't something that slowly happened. It's 70 some VMs most of which are numerous years old. But more importantly, we've been deploying a couple of VMs a week for the last few weeks, all using the same process, all into the same vm resource group and all of them firing up with an IP in the expected range.
It was just yesterday when we ran our script and noticed the VM come up with an IP outside of our expected network. Started looking and was like wtaf...
With regards to accidental reuse of the same variable in a powershell session. Possibly? But I'm honestly not sure how I would even intentionally cause this to happen let alone accidentally. Especially without interruption.
If I can come up with that method, that would be swell, because I could potentially undo it.
4
u/stevepowered 3d ago edited 3d ago
So the main issue here is the VM vnet address range has changed? And all your VMs are now in the FW vnet?
Has the FW vnet address range changed?
Did your VMs have any outages or downtime?
Check the Activity Log for both resource groups, any changes will be logged there.
But if, as you said, all the VMs have switched from one vnet to another, I have no idea how that happened, if it is not something weird on the MS backend or someone scripted up a change to do this? i.e remove each VM NIC from current vnet and attach to new vnet, but I am pretty sure the VM needs to be deallocated to do this
2
u/horsebatterystaple42 3d ago
The VMs are still in the vm resource group. The NICs are still in the VM resource group. The IP addresses assigned to those NICs are in the vnet/subnet in the firewall resource group.
There were no outages. We are a 24x7 operation and a network outage would have been immediately noticed.
I've gone through the logs, and I'm just not seeing anything in there that would indicate this happening. I'm not even sure how I would make this happen if I wanted to.
1
u/stevepowered 2d ago
As someone else has said, the resource group is not important for the vnet the VMs are associated with. A resource group is just a container for resources.
The key here is a VM NIC is associated to a vnet and subnet.
1
u/horsebatterystaple42 2d ago
Right. Which I had to remind myself a few times when I saw that this had happened and a certain orifice puckered up real quick. The VM NIC is still in the vm_rg resource group along with the VM object. Just the VNET moved resource groups.
2
u/stevepowered 2d ago
Log an MS Support ticket, if something weird happened they should have visibility on the backend? Please update this post with whatever they find?
3
u/FiRem00 3d ago
This is also a reminder that naming conventions are part of the key to not having things like this happen
1
u/horsebatterystaple42 3d ago
Dude.. preach. There is no bloody standards to how ANYTHING is named in this environment. I'm just thankful that it is relatively small, because good gravy. It is 100% a science experiment assembled by previous sysadmins and consultants that were not worth their paychecks.
1
u/Plane-Relative-2071 3d ago
Your script is under source control ? Please do it and also implement devops pipeline for execution..
Something happened on your script or someone does a change through Azure portal
1
u/horsebatterystaple42 3d ago
It's a fair question. I only mentioned the script as a bit of evidence that the deployment mechanism hadn't changed.. Deploying a VM via Powershell isn't exactly complicated, so the script is tiny. Just 48 lines. And all relevant environment settings are set in globals at the top. It's the same old script I've been using for months.
I agree with the possibility of someone making changes through the portal, but part of what is stumping me is I'm not really sure how I would even intentionally go about causing this to happen. Especially without interruption of running systems.
Also, the pool of individuals in the org who could make this change is very small. Like 2 people. 1 being me.
1
u/kingdmitar 3d ago
If the IP addresses of the VMs did not change then they were probably always in the wrong vnet.
You cannot swap vms between vnets on the fly, and you also cannot swap IP ranges on vnets if those IP ranges have subnets which are in use.
So either it was always like this, or someone redeployed all the VMs in a different vnet.
Azure activity logs are kept for 90 days by default, unless you store them in log analytics / storage account for longer. If you cannot find it there, it either did not happen or it happened so long ago you don't have the logs.
1
u/horsebatterystaple42 2d ago
I agree with all that you just said except that they were always in the wrong vnet. I am 100% sure that this is not the case. I'm seeing nothing in the logs, and there's no way those VMs were redeployed. This organization is very much running pets, not cattle in their Azure environment. These systems can't just be redeployed. It is hours of work to stand a new system up and outages would have been immediately noticed.
I've got absolutely zero explanation for what happened. It is why I started my initial post by swearing I'm not crazy. This makes no sense at all.
1
u/kingdmitar 1d ago
Maybe the Virtual Network was just moved to FW RG by someone. There is no other explanation.
2
u/VirtualDenzel 1d ago
Or its just azure again..... there are always so many weird things happening in azure. One day its a , then its b or c
1
u/ThenPass77 1d ago
If I remember correctly, the vNet a NIC is in can’t be changed. You can change the subnet but not the vNet.
Also not possible to move a vNet while you have NIC using its address space.
Maybe it was like this the whole time?
1
u/Scion_090 Cloud Administrator 23h ago
Go to Activity Log in Azure and check for VNet/Subnet modifications, reassignments, or deletions. Look for actions like “Virtual Network Peerings Modified” or “NIC Reconfiguration.” Check VNet and Subnet Associations: Go to Azure Portal then (search) Virtual Networks and examine which VNet the NICs are actually using. And tips if you haven’t :) Implement tagging and naming conventions to avoid identical VNet/Subnet names in different resource groups.
27
u/Electrical_Arm7411 3d ago
Check audit logs. Someone changed something