Boss will not attack you if you stand outside of melee range. However, you can still attack her. http://www.wowhead.com/npc=20885/dalliah-the-doomsayer