Should I continue to date my divorce attorney?

He would not take no for an answer, so I finally gave in under pressure. We've been seeing each other for a few months now, and he even does things with my kids. My trials comimg up soon, and I'm afraid my ex might find out. If he does, and he and his attorney bring this up to the judge in court can this hurt my divorce settlement?Help!