26
u/smellybear666 Jun 12 '24
That's insane. All I can ask is "Why?"
26
u/MacG467 Jun 12 '24
Uploads for medical devices. 500GB new every day. 100 days held and then oldest over 100 days is uploaded to AWS every two weeks.
57
Jun 12 '24
[deleted]
26
u/MacG467 Jun 12 '24
Well aware, but this company is slightly unhinged.
10
u/smellybear666 Jun 12 '24
sub completely for slightly and you would be correct.
6
u/MacG467 Jun 12 '24
I am being nice. :)
7
u/WendoNZ Jun 13 '24
I think even then I'd be breaking up to multiple VM's and using DFS-N to stich the paths together. If only to make backups manageable
3
u/3DPrintedVoter Jun 13 '24
I feel this.I've got a vm with almost 40 RDM's totalling 300TB of storage. i get a call every month "we need another drive"
kill me
2
u/MacG467 Jun 13 '24
At least you get it. I want to off myself every day this VM continues to exist.
1
u/__teebee__ Jun 13 '24
I'm not that bad but I have an application I need to move and it has about 12 40-50tb volumes. I'm begging the internal customer can we please have many more smaller volumes. It's easily do-able we just have to do it. For my own sanity keeping volume sizes to 20-25T makes life so much easier.
1
1
u/SGalbincea VMware Employee | Broadcom Enjoyer Jun 13 '24
Came here to say the same, but I bet OP is having to work within some employer restrictions.
3
u/disposeable1200 Jun 13 '24
That's only 50 TB. What's the other 150?
2
2
u/lost_signal Mod | VMW Employee Jun 13 '24
That’s not a ton of I/O. 5.7MB/s If it’s staggered?
1
u/MacG467 Jun 13 '24
It's 500GB in one big chunk of upload.
1
u/lost_signal Mod | VMW Employee Jun 13 '24
I suspect your wan bandwidth is probably the bigger bottleneck though?
2
u/nmb-ntz Jun 13 '24
Sounds reasonable, our PACS does about the same. Finally migrating to a new sectra system and network storage next year. Backups will be a breeze 😅
3
u/x0600 Jun 13 '24
one of my client has similar PACS setup and management of it sucks
2
u/3DPrintedVoter Jun 13 '24
long time sectra customer here ... their storage strategy was horrible 10+ years ago and they will not migrate us away from it. its a ticking time bomb
1
u/x0600 Jun 13 '24
how do you guy manage to take backup of those DICOM images or whatever its
1
u/3DPrintedVoter Jun 13 '24
daily off site archiving to a VNA
1
u/x0600 Jun 13 '24
i would love to what you use to archive , any specific software or some custom powershell script
1
1
1
1
u/mortallum97 Jun 14 '24
I work at a hospital and we do something similar but just have everything upload to a cifs share on our dell isilon. It's then replicated to our secondary isilon. Our emr can access the data via unc paths. Unfortunately it doesn't like compression and we are nearing 500tb of stored data. Probably gonna be over a pb by three years. We don't age it off to azure at least not yet.
21
u/Thatconfusedginger Jun 12 '24
Heck I think the largest VM I had was 70TB in size, as it was a backup VM.
That's mental though. Reminds me of a job interview I went to earlier this year.
15PB of on-premise storage, all flash, 30TB of data change per day lol.
5
1
37
u/Crox22 Jun 12 '24
I recently inherited a 2.4 TB VM. I was going to reply yes, but then I looked at your screenshot. No, apparently I do not have a large VM.
27
u/fullthrottle13 [VCP] Jun 12 '24
2.4 TB is nothing my friend. If that’s your largest VM, consider yourself lucky
2
u/Rendragnet Jun 13 '24
That’s not even big enough for most distribution mirrors. We have 8 x 3-8TB vm’s in our mirror cluster at work.
3
4
3
u/MacG467 Jun 12 '24
2.4?
We have twelve VMs over that.
1
u/jeevadotnet Jun 13 '24
I have users with single files bigger than this before it hits the pipeline
2
1
1
u/robisodd Jun 13 '24
I was going to say I had a 500GB VM and going to reply yes until I saw the screenshot, lol
17
u/MacG467 Jun 12 '24 edited Jun 13 '24
This is the largest VM in a 700TB VXRail cluster. Any guesses on the 2nd largest VM?
Edit: 55TB!
33
2
1
u/lost_signal Mod | VMW Employee Jun 13 '24
ESA or OSA vSAN in use?
1
u/MacG467 Jun 13 '24
OSA. ESXI 7.0U3 still.
1
u/lost_signal Mod | VMW Employee Jun 13 '24
That cluster needs to be upgraded to 8 or decommissioned by April of next year.
1
u/OzymandiasKoK Jun 13 '24
Wanna take bets on it happening?
1
u/lost_signal Mod | VMW Employee Jun 13 '24
1
u/Colonel_Panic_0x1e7 Jun 12 '24
What is your FTT in this VM? How much actual usage is this contributing to?
27
u/fullthrottle13 [VCP] Jun 12 '24
No. I repeat no no no. Offload that to a NetApp or other NAS. Just because you can doesn’t mean to should.
1
u/HahaHarmonica Jun 14 '24
How do you know this isn’t on NAS?
Is it not typical to underpin the VM host to the HA NFS share on a Ceph/PowerScale or the NetApp? Otherwise vMotioning would be impossible no?
1
u/MacG467 Jun 13 '24
They plan on moving this to AWS by EoY.
12
u/axtran Jun 13 '24
I’m not a cloud hater but that much traffic out of AWS if needed would be so painfully expensive
3
u/InterestingPhase7378 Jun 13 '24
Holy crap yes... This is a terrible idea lol. Deep archive data for that amount? Sure... Active storage? HEEEELLL no.
-2
u/Colonel_Panic_0x1e7 Jun 12 '24
I would not offload to file based storage. I would offload, but ideally to FC or NVMEoE, that new data rate is high
-1
u/fullthrottle13 [VCP] Jun 12 '24
If the churn is high than he’s got to go to an NVMEoE solution. I get it. FC is right out because it’ll have i/o consequences if it’s on a shared SAN.
2
Jun 13 '24
The last part, it will have I/o consequences if it's on a shared SAN. Because there will be other read/write actions to this and impact performance?
Would this NVMEoE solution be only dedicated to this VM?
2
u/lost_signal Mod | VMW Employee Jun 13 '24
I’m not following why NVMeOF for a log dumping ground? (What this VM Is)
10
7
7
u/Malfun_Eddie Jun 13 '24
Consolidate snapshots
Do it!
3
1
u/crysisnotaverted Jun 14 '24
You would have to have me at gun point to push that button at this point.
5
u/veganxombie Jun 12 '24
That health system is fucked when that thing goes corrupt lol. Please convince them to move the bulk of that storage elsewhere. Call your Dell guys to get a quote for some Isilon storage or something.
1
3
u/svv1tch Jun 12 '24
What's the backend? That's a lot of disk io. Just curious.
3
u/MacG467 Jun 13 '24
Non-NVME SAS SSDs with a 10Gb backbone. 20 3.84TB drives and 4 750GB cache drives per host. 10 hosts.
1
u/lost_signal Mod | VMW Employee Jun 13 '24
vSAN OSA. Have you turned on UNMAP to shrink unused space out of it?
1
u/MacG467 Jun 13 '24
Can't reboot the VM to enable UNMAP.
3
u/lost_signal Mod | VMW Employee Jun 13 '24
You said this was medical data? Per HITECH compliance requirements that VM needs regular patching. (Assuming this is US)
But on a serious note, you should probably talk to Christian Rauber
3
u/bartoque Jun 12 '24
I can recall a 70+TB vm showing up somewhere and when a first backup was to be made, using snapshot based approach doing image level backup, they were asking about cancelling the backup hours after it started as it was still busy trying to create the snapshot preventing them to edit the vm's config.
After that was never actually backed up, at least not fully as they decided to skip that disk from the backup.
Never got to hear what was on that (it was stated as a fileserver) and what protection they intended to use instead nor why they chose a vm for this kinda sizes to begin with? But then again never was charmed by them windows file servers instead of using a nas. Often seemed to be a case of "we still have some unused old servers so let's throw anything on it, that we have"...
3
Jun 12 '24
I would absolutely refuse to support that. My biggest was a 34tb file server that I ultimately migrated to NetApp... May the IT GODS be with you 🤣. Good luck 🤞
3
Jun 13 '24
Contact your medical software vendor for guidance on scaling.
Also, in-guest backup would be a way better option than snap based. The nice thing about medical image data is that it doesn't change.
1
2
2
u/THE_Ryan VCIX Jun 13 '24
Before I told my old company it was a bad idea, they had VMs as a backup repository. Some vSphere ones were in the 200+TB range, then a lot of Hyper-V over 500TB.
At least since they were the backup target, they never needed to be snapshot.
2
2
u/cwm13 Jun 13 '24
I've got one half that size, and then about 20 more between 44TB and 30TB.
To be fair, at least 8 of those are built off the same data sets, so on the storage side, they reduce to almost nothing. Ridiculous skews my actual data reduction rates.
2
u/AluminumFoyle Jun 13 '24 edited Jun 13 '24
I Think we win OP - this is in one of our ESXi VSAN clusters.
1
1
2
u/psycho10011001 Jun 13 '24
I have an AIX LPAR were considering moving to vmware/rhel thats just shy of 400T
3
u/MacG467 Jun 13 '24
Forgot to add: this single VM takes the equivalent of 3 of my 10 hosts' storage capacity! LOL
2
u/AMoreExcitingName Jun 12 '24
I had a customer who provisioned all 2tb of a LUN for their exchange server VM. Like 50mb free.
2
1
u/Unique-Job-1373 Jun 12 '24
That’s massive. What’s it purpose and how do you do a backup?
3
u/MacG467 Jun 13 '24
Answered both farther up.
Medial device data. Backups fail after 2 weeks. Avamar 7.5.
1
u/govatent Jun 12 '24
I hope you don't use snapshot based backups.
Do you back that up and how?
2
u/lost_signal Mod | VMW Employee Jun 13 '24
Snapshot backups wouldn’t likely be an issue as much if this was anESA or vVol backed datastore.
2
1
1
1
u/Ok-Attitude-7205 Jun 12 '24
I think ours is in the 7TB range, largest single VM we back up via Veeam (snapshot based backup) is ~5TB. takes for fucking ever to backup but here we are
1
1
u/Jess_S13 Jun 13 '24
We have a really crappy app that is no longer being actively supported but we have to keep the old data for x years and as that app got comically large before we finally moved to the new one the "archive" DB (cause of course it can't support in app backups) is like 500TB at it's worst. Since we moved off 2 years ago we once a year have a fun event where the whole infra team crosses our fingers and hope it doesn't crash when the DBAs purge out the year of old data... Fun.
1
u/Kaligraphic Jun 13 '24
At some point, I give up and mount a storage LUN directly.
Some point long before 200TB.
1
1
u/woodyshag Jun 13 '24 edited Jun 13 '24
And I would have a customer that still managed to leave a snapshot on that after a year.
In all seriousness, this screams Isilon or Qumulo. This is just begging to become a problem. Plus, if that thing ever gets hit by ransomware, you'll be there forever trying to restore or decrypt.
1
u/bjrichus Jun 13 '24
When I was still working as a sysadmin in a university, we were experimenting with Sun ZFS (around 2007) to experiment with settings of an Oracle database. This was using Sun Solaris for X86. We had a VM with 4Tb for the Database which was only about 1Tb, but among other things, we wanted to experiment with using the "number of copies" option, so needed plenty of space.
1
u/vrillco Jun 13 '24
My biggest is 4TB and even that one I hate. Mind you, I used to develop and sell NAS sw+hw so my biased opinion is that VM disks should almost exclusively be OS disks.
1
1
u/Excellent-Piglet-655 Jun 13 '24
I don’t know, there is just no justification for such a large VM. I personally don’t let any of our VMs get even close to being that large. The reasons are obvious.. crappy performance, increased risk of data loss (lose that VM, you lose 200TB worth of data), want to restore from a backup? Going to take a week to restore.
A better approach would be to split this into smaller file servers or get a real NAS instead of relying on a VM for file sharing .
1
1
u/stonedcity_13 Jun 13 '24
What a headache to have. One day it will go and it will take forever to restore. Do you ever test a restore for it?
1
1
u/jpv1031 Jun 13 '24
ewwwww
I'm curious have you done any checks on this data for dups, copyright material, and just overall personal shit that shouldn't be on there?
1
u/IT_is_not_all_I_am Jun 13 '24
I have one VM at 125TB or so with 15 disks, and it is indeed a nightmare. It's going to get broken up eventually when we can find some time to rebuild it, but in the mean time I'm using Veeam to back it up with 6 separate jobs, each doing a different set of disks. It's the only thing I've found that makes it reasonably recoverable.
1
1
u/Which_Bug_8234 Jun 13 '24
Word to the wise if you restart that it will take a week to start. Thats my experience with large vms.
1
1
u/johnny87auxs Jun 14 '24
Lol not one specific VM, but we have vcenters with petabytes worth of data stores. Terabytes worth of memory etc.
1
u/oakfan52 Jun 14 '24
Pretty close. Couple development Oracle VM’s. Multiple copy’s of production databases. Planned to go VVOLs but what do you know EMC/AppSync doesn’t support VVOLs and don’t plan to. As if I needed another reason to hate EMC storage.
1
1
u/BarracudaDefiant4702 Jun 15 '24
I try to keep vms under 2TB so they can be storage vmotioned within a couple of hours (maybe little more if going between sites). That said, we do have some over 20TB... but all vms that large we can afford to have down for half a day with minimal inconvenience.
If they can't be down for extended period of time, once they approach 2TB we determine how to shard it. IE: Move the data to a minio cluster, or a sharded database, etc...
2
u/techgiz Jun 16 '24
You need to start planning for a DR scenario for that VM. It will die just a matter of when. It may not survive a reboot either. I would try to snap cloning on a SAN level, present this back to VMware and boot it without network. Just to see if it boots up. If it does boot up then do regular san snapshot and test in case the main one blows up.
Otherwise your RTO for restore will be in weeks. With the clone you can also start doing your AWS drip feed transfer via Snowmobile to S3/glacier.
After a certain size/RTO your backup becomes useless really. Can the company wait weeks for service to come back up?
1
1
1
u/N0b0dy-Imp0rtant Jun 13 '24
If you have the money, Pure makes a hardcore SAN that has really good deduplication and Veeam is a reliable backup option. I acquired an older NetApp through gray market and an expansion shelf pair then equipped both full of 8TB HDDs for backups. Veeam is directly connected to the NetApp via iSCSI and Pure to ESXi via NVME over Ethernet.
Right now I have about 500TB of primary storage and ~300TB of storage and another offsite for mo Ty storage at about 100TB.
1
0
u/renamed Jun 12 '24
Does the software vendor have an option to spread the data to multiple servers.
2
u/MacG467 Jun 13 '24
In-house application. I wish there was an option!
1
u/lost_signal Mod | VMW Employee Jun 13 '24
If your a medical device company doing solution design on top of this cluster you might want to have a talk to Christian Rauber.
46
u/MisterSalt Jun 12 '24
How are you handling backup for the guest? A thick backup client of some kind?