It doesn't matter too much what their intentions are. [ He knows what Suigetsu would like them to be and he doesn't want to allow anyone to give him an excuse. The more commotion they cause, the more eyes they'll have on them. The more attention, the harder it'll be to get out. Of that, he's convinced. ]

I don't doubt that you're too strong for most if not all of them. Defend yourself if you need to, but otherwise ignore their goading.