Haha, I got this guy after finishing the arena quests in previous play. But I just told him bug off and he didn't bother me again.
Which reminds me... what about that corpse in a small lake somewhere around the Imperial City? In its inventory there is a letter speaking about great love. I found similar letter in a house somewhere in Cheydinhal in one of previous plays but it all is still a mystery to me.
By the way, if the guy is casting off comments as I battle, I might take him with just for fun. And... try to get him killed of course.