What you need to do to stop VMware eating all your available RAM is to limit the "Reserved Memory" amount, which by default is set to whatever amount is installed. You'll find the setting in Edit>Preferences: