Does this sound like a sex-only affair to you?

Yes, he is jeopardizing his job and your family for even considering such a thing. Maybe he thought that by hiring her, it made things easier for them to "get together" whenever they felt like it. Still, though, working together would give them a level of intimacy that is beyond just sleeping together.