How sensitive are men about going soft during sex?

Basically this happened yesterday with a casual hook up and i made what I thought was a light hearted comment about it (to show it wasn't an issue), next thing I know guy has bailed and that's the end of it.
I already know penises have a mind of their own and it's not an issue for me, but did I mess up by mentioning it it? Does drawing attention to it, even gently, embarrass someone?
Have you ever bailed because of something like that? What should I have done instead?
Edit: I said "those things always have a mind of their own" and sort of grinned.
Edit 2: he had already gotten off at this point too, were working on a second round so I figured it was to be expected anyway?


Source.