I’ve certainly had those; in fact, quite a few servants I use now were some I used to avoid for some reason or another.
For me it would be Teach. I made a post some time ago lamenting on his design (and I do still want another servant after his history) because of how impactful Blackbeard’s story was to my homestate. Since then, though, he’s actually grown on me; he has some funny scenes and isn’t really as bad as a lot of people told me he was. On top of that, too, he’s one hell of a farmer.
I dunno, what do you guys think?