It's a possibility. Though it's just as possible that it would not shake him at all.

[ Hannibal shifts to tuck his hands into his pockets idly with his posture open and casual, but his expression is still fairly serious and thoughtful. ]

That's why it's unpredictable. It could make him slip, or it may not do a thing at all. I couldn't guess with much certainty which it might be without knowing him better.