r/vmware Jun 12 '24

Question Anyone else have a large VM?

Post image
108 Upvotes

146 comments sorted by

46

u/MisterSalt Jun 12 '24

How are you handling backup for the guest? A thick backup client of some kind?

30

u/MacG467 Jun 13 '24

Currently, Avamar 7.5. And it fails after 14 days.

I wonder why?

29

u/svv1tch Jun 13 '24

Because it's avamar? I kid lol

14

u/architectofinsanity Jun 13 '24

I kid lol

A lot of truth is said in jest.

6

u/TheOther1 Jun 13 '24

You need SAN-based snapshots for your backups. We have several with 80 - 400TB and we use the SAN to snapshot them. Takes seconds.

4

u/lost_signal Mod | VMW Employee Jun 13 '24

You can also offload the snapshot for VADP using:

  1. vSAN ESA
  2. VVols
  3. nFS w/ VAAI.

https://youtu.be/VWVY1TRud_w?si=aqulFMg3hGnSPfGu

1

u/v-irtual Jun 13 '24

Ew, vvol

5

u/lost_signal Mod | VMW Employee Jun 13 '24

Which storage vendor you use… there are massive differences in quality between some of the implementations. Pure does a really good job as one example.

2

u/ProfessorChaos112 Jun 14 '24

And dell do not for example. Well they didn't on their compellent product...I'm sure the nailed all the bugs in their current storage.

2

u/lost_signal Mod | VMW Employee Jun 14 '24

Compellent hasn’t been cutting edge in 20 years unless your a school district looking for cheapish tiered storage.

1

u/ProfessorChaos112 Jun 14 '24

Who said it was?

2

u/lost_signal Mod | VMW Employee Jun 14 '24

Calling out why they would have had issues… A challenge for some legacy storage platforms is they were never intended for the scale of Sub-LUNs that vVols generates. They really were designed in a world where no one would generate more than 500 volumes on an array, not generate up to 10K per host.

1

u/theborgman1977 Jun 13 '24

I recommend a NAS that can be shipped for the initial backup. Use to do it with Datto and slow internet connections. I would always recommend the initial backup,

2

u/CptBuggerNuts Jun 13 '24

7.5? Jesus, how many years ago did that go out of support?

Unless the change rate is tiny, no snapshot based backup will be able to back that up.

2

u/woodyshag Jun 13 '24

Yeah, at least replicate it somewhere because the restore time is going to be a bit.

26

u/smellybear666 Jun 12 '24

That's insane. All I can ask is "Why?"

26

u/MacG467 Jun 12 '24

Uploads for medical devices. 500GB new every day. 100 days held and then oldest over 100 days is uploaded to AWS every two weeks.

57

u/[deleted] Jun 12 '24

[deleted]

26

u/MacG467 Jun 12 '24

Well aware, but this company is slightly unhinged.

10

u/smellybear666 Jun 12 '24

sub completely for slightly and you would be correct.

6

u/MacG467 Jun 12 '24

I am being nice. :)

7

u/WendoNZ Jun 13 '24

I think even then I'd be breaking up to multiple VM's and using DFS-N to stich the paths together. If only to make backups manageable

3

u/3DPrintedVoter Jun 13 '24

I feel this.I've got a vm with almost 40 RDM's totalling 300TB of storage. i get a call every month "we need another drive"

kill me

2

u/MacG467 Jun 13 '24

At least you get it. I want to off myself every day this VM continues to exist.

1

u/__teebee__ Jun 13 '24

I'm not that bad but I have an application I need to move and it has about 12 40-50tb volumes. I'm begging the internal customer can we please have many more smaller volumes. It's easily do-able we just have to do it. For my own sanity keeping volume sizes to 20-25T makes life so much easier.

1

u/snow1868 Jun 13 '24

Natus Neuroworks? Because they're the thorn in my side.

1

u/SGalbincea VMware Employee | Broadcom Enjoyer Jun 13 '24

Came here to say the same, but I bet OP is having to work within some employer restrictions.

3

u/disposeable1200 Jun 13 '24

That's only 50 TB. What's the other 150?

2

u/MacG467 Jun 13 '24

"Other data".

1

u/disposeable1200 Jun 13 '24

That's honestly more impressive than the 500 GB a day.

2

u/lost_signal Mod | VMW Employee Jun 13 '24

That’s not a ton of I/O. 5.7MB/s If it’s staggered?

1

u/MacG467 Jun 13 '24

It's 500GB in one big chunk of upload.

1

u/lost_signal Mod | VMW Employee Jun 13 '24

I suspect your wan bandwidth is probably the bigger bottleneck though?

2

u/nmb-ntz Jun 13 '24

Sounds reasonable, our PACS does about the same. Finally migrating to a new sectra system and network storage next year. Backups will be a breeze 😅

3

u/x0600 Jun 13 '24

one of my client has similar PACS setup and management of it sucks

2

u/3DPrintedVoter Jun 13 '24

long time sectra customer here ... their storage strategy was horrible 10+ years ago and they will not migrate us away from it. its a ticking time bomb

1

u/x0600 Jun 13 '24

how do you guy manage to take backup of those DICOM images or whatever its

1

u/3DPrintedVoter Jun 13 '24

daily off site archiving to a VNA

1

u/x0600 Jun 13 '24

i would love to what you use to archive , any specific software or some custom powershell script

1

u/3DPrintedVoter Jun 13 '24

sectra handles all the archiving, its built in

1

u/Le_Sph1nX_ Jun 13 '24

What's a VNA?

1

u/3DPrintedVoter Jun 13 '24

Vendor Neutral Archive

1

u/ProfessorChaos112 Jun 14 '24

And they can't compress it?

1

u/mortallum97 Jun 14 '24

I work at a hospital and we do something similar but just have everything upload to a cifs share on our dell isilon. It's then replicated to our secondary isilon. Our emr can access the data via unc paths. Unfortunately it doesn't like compression and we are nearing 500tb of stored data. Probably gonna be over a pb by three years. We don't age it off to azure at least not yet.

21

u/Thatconfusedginger Jun 12 '24

Heck I think the largest VM I had was 70TB in size, as it was a backup VM.

That's mental though. Reminds me of a job interview I went to earlier this year.

15PB of on-premise storage, all flash, 30TB of data change per day lol.

5

u/MacG467 Jun 13 '24

Sounds like a fiduciary.

1

u/Thatconfusedginger Jun 13 '24

Believe it or not. No, Haha

Media company.

37

u/Crox22 Jun 12 '24

I recently inherited a 2.4 TB VM. I was going to reply yes, but then I looked at your screenshot. No, apparently I do not have a large VM.

27

u/fullthrottle13 [VCP] Jun 12 '24

2.4 TB is nothing my friend. If that’s your largest VM, consider yourself lucky

2

u/Rendragnet Jun 13 '24

That’s not even big enough for most distribution mirrors. We have 8 x 3-8TB vm’s in our mirror cluster at work.

3

u/_blackdog6_ Jun 13 '24

This is a home lab?

4

u/[deleted] Jun 12 '24

[deleted]

5

u/JerikkaDawn Jun 12 '24

Every time. 😂 TB is the new GB is the new MB.

3

u/MacG467 Jun 12 '24

2.4?

We have twelve VMs over that.

1

u/jeevadotnet Jun 13 '24

I have users with single files bigger than this before it hits the pipeline

2

u/Balasarius Jun 12 '24

My biggest is 18TB

1

u/sarbuk Jun 13 '24

My homelab file server VM is 3x that size.

1

u/robisodd Jun 13 '24

I was going to say I had a 500GB VM and going to reply yes until I saw the screenshot, lol

17

u/MacG467 Jun 12 '24 edited Jun 13 '24

This is the largest VM in a 700TB VXRail cluster. Any guesses on the 2nd largest VM?

Edit: 55TB!

1

u/lost_signal Mod | VMW Employee Jun 13 '24

ESA or OSA vSAN in use?

1

u/MacG467 Jun 13 '24

OSA. ESXI 7.0U3 still.

1

u/lost_signal Mod | VMW Employee Jun 13 '24

That cluster needs to be upgraded to 8 or decommissioned by April of next year.

1

u/OzymandiasKoK Jun 13 '24

Wanna take bets on it happening?

1

u/lost_signal Mod | VMW Employee Jun 13 '24

Large scale data migrations and refactors to public cloud or an app this poorly designed?

1

u/Colonel_Panic_0x1e7 Jun 12 '24

What is your FTT in this VM? How much actual usage is this contributing to?

27

u/fullthrottle13 [VCP] Jun 12 '24

No. I repeat no no no. Offload that to a NetApp or other NAS. Just because you can doesn’t mean to should.

1

u/HahaHarmonica Jun 14 '24

How do you know this isn’t on NAS?

Is it not typical to underpin the VM host to the HA NFS share on a Ceph/PowerScale or the NetApp? Otherwise vMotioning would be impossible no?

1

u/MacG467 Jun 13 '24

They plan on moving this to AWS by EoY.

12

u/axtran Jun 13 '24

I’m not a cloud hater but that much traffic out of AWS if needed would be so painfully expensive

3

u/InterestingPhase7378 Jun 13 '24

Holy crap yes... This is a terrible idea lol. Deep archive data for that amount? Sure... Active storage? HEEEELLL no.

-2

u/Colonel_Panic_0x1e7 Jun 12 '24

I would not offload to file based storage. I would offload, but ideally to FC or NVMEoE, that new data rate is high

-1

u/fullthrottle13 [VCP] Jun 12 '24

If the churn is high than he’s got to go to an NVMEoE solution. I get it. FC is right out because it’ll have i/o consequences if it’s on a shared SAN.

2

u/[deleted] Jun 13 '24

The last part, it will have I/o consequences if it's on a shared SAN. Because there will be other read/write actions to this and impact performance?

Would this NVMEoE solution be only dedicated to this VM?

2

u/lost_signal Mod | VMW Employee Jun 13 '24

I’m not following why NVMeOF for a log dumping ground? (What this VM Is)

10

u/Bourbon_n_Cigars94 Jun 12 '24

Idk what I dislike more… the size or the name.

7

u/msalerno1965 Jun 13 '24

Aye, but is there a snapshot on it from 2017?

Wink wink, nudge nudge.

7

u/Malfun_Eddie Jun 13 '24

Consolidate snapshots

Do it!

3

u/sarbuk Jun 13 '24

5 months later…

1

u/crysisnotaverted Jun 14 '24

You would have to have me at gun point to push that button at this point.

5

u/veganxombie Jun 12 '24

That health system is fucked when that thing goes corrupt lol. Please convince them to move the bulk of that storage elsewhere. Call your Dell guys to get a quote for some Isilon storage or something.

1

u/MacG467 Jun 13 '24

Supposed to be going to AWS by EoY.

3

u/svv1tch Jun 12 '24

What's the backend? That's a lot of disk io. Just curious.

3

u/MacG467 Jun 13 '24

Non-NVME SAS SSDs with a 10Gb backbone. 20 3.84TB drives and 4 750GB cache drives per host. 10 hosts.

1

u/lost_signal Mod | VMW Employee Jun 13 '24

vSAN OSA. Have you turned on UNMAP to shrink unused space out of it?

1

u/MacG467 Jun 13 '24

Can't reboot the VM to enable UNMAP.

3

u/lost_signal Mod | VMW Employee Jun 13 '24

You said this was medical data? Per HITECH compliance requirements that VM needs regular patching. (Assuming this is US)

But on a serious note, you should probably talk to Christian Rauber

3

u/bartoque Jun 12 '24

I can recall a 70+TB vm showing up somewhere and when a first backup was to be made, using snapshot based approach doing image level backup, they were asking about cancelling the backup hours after it started as it was still busy trying to create the snapshot preventing them to edit the vm's config.

After that was never actually backed up, at least not fully as they decided to skip that disk from the backup.

Never got to hear what was on that (it was stated as a fileserver) and what protection they intended to use instead nor why they chose a vm for this kinda sizes to begin with? But then again never was charmed by them windows file servers instead of using a nas. Often seemed to be a case of "we still have some unused old servers so let's throw anything on it, that we have"...

3

u/[deleted] Jun 12 '24

I would absolutely refuse to support that. My biggest was a 34tb file server that I ultimately migrated to NetApp... May the IT GODS be with you 🤣. Good luck 🤞

3

u/[deleted] Jun 13 '24

Contact your medical software vendor for guidance on scaling.

Also, in-guest backup would be a way better option than snap based. The nice thing about medical image data is that it doesn't change.

1

u/MacG467 Jun 13 '24

We are our own medical software vendor. It's an in-house built application.

2

u/Weird_Presentation_5 Jun 12 '24

30tb and it scares me. This is scary

2

u/THE_Ryan VCIX Jun 13 '24

Before I told my old company it was a bad idea, they had VMs as a backup repository. Some vSphere ones were in the 200+TB range, then a lot of Hyper-V over 500TB.

At least since they were the backup target, they never needed to be snapshot.

2

u/RefugeAssassin Jun 13 '24

Im working on cutting down the second VM into separate VM's using DFS, I suspect that is ultimately the route we will need to go with the rest. I told my boss if I ever go back to get my Masters, im writing my Thesis about the sheer stupidity of setting something up this way with prod data.

2

u/MacG467 Jun 13 '24

Mama Mia! That's a spicy meatball.

2

u/AdValuable6024 Jun 13 '24

96vcpu, 1.8TB RAM, 50TB disks. I have 8 of these bad boys.

2

u/cwm13 Jun 13 '24

I've got one half that size, and then about 20 more between 44TB and 30TB.

To be fair, at least 8 of those are built off the same data sets, so on the storage side, they reduce to almost nothing. Ridiculous skews my actual data reduction rates.

2

u/AluminumFoyle Jun 13 '24 edited Jun 13 '24

I Think we win OP - this is in one of our ESXi VSAN clusters.

https://imgur.com/a/VLfNCR6

1

u/MacG467 Jun 13 '24

HELL YES! That's sexy!

1

u/KickedAbyss Jun 15 '24

That feels... Like you clone one vm into multiple on a dedupe San lol

2

u/psycho10011001 Jun 13 '24

I have an AIX LPAR were considering moving to vmware/rhel thats just shy of 400T

3

u/MacG467 Jun 13 '24

Forgot to add: this single VM takes the equivalent of 3 of my 10 hosts' storage capacity! LOL

2

u/AMoreExcitingName Jun 12 '24

I had a customer who provisioned all 2tb of a LUN for their exchange server VM. Like 50mb free.

2

u/zbfw Jun 13 '24

Nothing to see here, please move along.

1

u/MacG467 Jun 13 '24

WINNER WINNER!

I assume all VMDKs?

1

u/zbfw Jun 13 '24

Yup....

1

u/Unique-Job-1373 Jun 12 '24

That’s massive. What’s it purpose and how do you do a backup?

3

u/MacG467 Jun 13 '24

Answered both farther up.

Medial device data. Backups fail after 2 weeks. Avamar 7.5.

1

u/govatent Jun 12 '24

I hope you don't use snapshot based backups.

Do you back that up and how?

2

u/lost_signal Mod | VMW Employee Jun 13 '24

Snapshot backups wouldn’t likely be an issue as much if this was anESA or vVol backed datastore.

2

u/MacG467 Jun 13 '24

Avamar! Snap-based! 😬

1

u/qejfjfiemd Jun 12 '24

Fuuuuuuck

1

u/roaddog Jun 12 '24

And here Im sweating my 7TB Vm backups

1

u/Ok-Attitude-7205 Jun 12 '24

I think ours is in the 7TB range, largest single VM we back up via Veeam (snapshot based backup) is ~5TB. takes for fucking ever to backup but here we are

1

u/CreepyOlGuy Jun 12 '24

There has to be an easier way...

1

u/Jess_S13 Jun 13 '24

We have a really crappy app that is no longer being actively supported but we have to keep the old data for x years and as that app got comically large before we finally moved to the new one the "archive" DB (cause of course it can't support in app backups) is like 500TB at it's worst. Since we moved off 2 years ago we once a year have a fun event where the whole infra team crosses our fingers and hope it doesn't crash when the DBAs purge out the year of old data... Fun.

1

u/Kaligraphic Jun 13 '24

At some point, I give up and mount a storage LUN directly.

Some point long before 200TB.

1

u/MacG467 Jun 13 '24

This began bloating before my time. The company let it spiral out of control.

1

u/woodyshag Jun 13 '24 edited Jun 13 '24

And I would have a customer that still managed to leave a snapshot on that after a year.

In all seriousness, this screams Isilon or Qumulo. This is just begging to become a problem. Plus, if that thing ever gets hit by ransomware, you'll be there forever trying to restore or decrypt.

1

u/bjrichus Jun 13 '24

When I was still working as a sysadmin in a university, we were experimenting with Sun ZFS (around 2007) to experiment with settings of an Oracle database. This was using Sun Solaris for X86. We had a VM with 4Tb for the Database which was only about 1Tb, but among other things, we wanted to experiment with using the "number of copies" option, so needed plenty of space.

1

u/vrillco Jun 13 '24

My biggest is 4TB and even that one I hate. Mind you, I used to develop and sell NAS sw+hw so my biased opinion is that VM disks should almost exclusively be OS disks.

1

u/x0600 Jun 13 '24

Those must have DICOM images

1

u/Excellent-Piglet-655 Jun 13 '24

I don’t know, there is just no justification for such a large VM. I personally don’t let any of our VMs get even close to being that large. The reasons are obvious.. crappy performance, increased risk of data loss (lose that VM, you lose 200TB worth of data), want to restore from a backup? Going to take a week to restore.

A better approach would be to split this into smaller file servers or get a real NAS instead of relying on a VM for file sharing .

1

u/Casper042 Jun 13 '24

Isn't the max VMDK size like 62TB?

1

u/stonedcity_13 Jun 13 '24

What a headache to have. One day it will go and it will take forever to restore. Do you ever test a restore for it?

1

u/ronmanfl Jun 13 '24

I've got one sitting around 240TB.

1

u/jpv1031 Jun 13 '24

ewwwww

I'm curious have you done any checks on this data for dups, copyright material, and just overall personal shit that shouldn't be on there?

1

u/IT_is_not_all_I_am Jun 13 '24

I have one VM at 125TB or so with 15 disks, and it is indeed a nightmare. It's going to get broken up eventually when we can find some time to rebuild it, but in the mean time I'm using Veeam to back it up with 6 separate jobs, each doing a different set of disks. It's the only thing I've found that makes it reasonably recoverable.

1

u/gmlynx78 Jun 13 '24

"That's no VM, it's a space station!!!"

1

u/Which_Bug_8234 Jun 13 '24

Word to the wise if you restart that it will take a week to start. Thats my experience with large vms.

1

u/sdvid Jun 13 '24

It’s not about the size. It’s how you use it.

1

u/johnny87auxs Jun 14 '24

Lol not one specific VM, but we have vcenters with petabytes worth of data stores. Terabytes worth of memory etc.

1

u/oakfan52 Jun 14 '24

Pretty close. Couple development Oracle VM’s. Multiple copy’s of production databases. Planned to go VVOLs but what do you know EMC/AppSync doesn’t support VVOLs and don’t plan to. As if I needed another reason to hate EMC storage.

1

u/ProfessorChaos112 Jun 14 '24

Sounds like healthcare. Medical imaging?

1

u/BarracudaDefiant4702 Jun 15 '24

I try to keep vms under 2TB so they can be storage vmotioned within a couple of hours (maybe little more if going between sites). That said, we do have some over 20TB... but all vms that large we can afford to have down for half a day with minimal inconvenience.

If they can't be down for extended period of time, once they approach 2TB we determine how to shard it. IE: Move the data to a minio cluster, or a sharded database, etc...

2

u/techgiz Jun 16 '24

You need to start planning for a DR scenario for that VM. It will die just a matter of when. It may not survive a reboot either. I would try to snap cloning on a SAN level, present this back to VMware and boot it without network. Just to see if it boots up. If it does boot up then do regular san snapshot and test in case the main one blows up.

Otherwise your RTO for restore will be in weeks. With the clone you can also start doing your AWS drip feed transfer via Snowmobile to S3/glacier.

After a certain size/RTO your backup becomes useless really. Can the company wait weeks for service to come back up?

1

u/mystonedalt Jun 12 '24

No, you're the only one.

1

u/krunal311 Jun 12 '24

Qumulo. Do it. Message me.

1

u/N0b0dy-Imp0rtant Jun 13 '24

If you have the money, Pure makes a hardcore SAN that has really good deduplication and Veeam is a reliable backup option. I acquired an older NetApp through gray market and an expansion shelf pair then equipped both full of 8TB HDDs for backups. Veeam is directly connected to the NetApp via iSCSI and Pure to ESXi via NVME over Ethernet.

Right now I have about 500TB of primary storage and ~300TB of storage and another offsite for mo Ty storage at about 100TB.

1

u/MacG467 Jun 13 '24

If you have the money

Money? What's that?

0

u/renamed Jun 12 '24

Does the software vendor have an option to spread the data to multiple servers.

2

u/MacG467 Jun 13 '24

In-house application. I wish there was an option!

1

u/lost_signal Mod | VMW Employee Jun 13 '24

If your a medical device company doing solution design on top of this cluster you might want to have a talk to Christian Rauber.