Does your date and the Ex. still confer with one another?  Or more to the point, do they discuss you?  If so, you will never be more than an outsider in their lives because they already have a special bond that you don’t.