I don't have a working example, but in the x0_i0_spells library there is a spellsIsTarget() function in which you can specify what target type will be affected by a spell. the Lexicon goes into detail about the three main options you have with that function. But if you only want hostiles, you could set it up like this in the AOE's OnEnter event:
if (spellsIsTarget(oTarget, SPELL_TARGET_SELECTIVEHOSTILE, GetAreaOfEffectCreator())) {
//Do the hostile-affecting-only AOE stuff here
}
If you're not looking to modify AOE scripts though, the Lexicon description of that spellsIsTarget function give some module switches you can enable or disable which (should?) determine what target types get negatively affected by (all?) AOEs.