Should I continue dating my divorce attorney?

He wouldn't take no for an answer, and I finally caved under pressure. We've been in a serious relationship for a few months now. Hes over my house all the time, and he plays with my kids. My trials coming up in a few months, and my ex-husband has been seen driving by my house. I think he knows? Can this hurt my divorce settlement when I go to court if he and his attorney tell the judge that I'm dating my attorney? Help!