Sorry to hear you are having such trouble. We use a fair amount of multicast applications, and have never had issues with dropped packets. We don't have the vm density on our hosts to require splitRxMode, the default config has been fine so far. Only multicast issue I can remember is the spam caused by multi nic vmotions running longer than your mac table timouts, but that was fixed some time ago.
Just to be clear, is the VM in question the target of the multicast or one of the recipients?
You only mention the one server, a rather old dell 2950. Have you tried moving the VM in question to more modern hardware? Can't really complain ESXi isn't ready for mission critical applications when your running it on a server model Dell won't even warranty anymore 