If you’re looking for an equivalent to ISIS in the 13th or 14th century, you’re looking for an agent that is not abiding by the then extant international rules of conduct, and that is reviled universally, by coreligionists and outsiders alike, as being beyond the pale.
The Ottomans started as ghazis, so they were more aggressive about their expansion than Muslims had been recently. But they were not aliens from a different planet, the way ISIS is to everyone else.
ISIS is also small, but I don’t think that’s a useful differentiator. Osman’s emirate of Bithynia started out pretty small too.
The closest equivalent, I’ll suggest, is Genghis Khan.
EDIT: Dimitris Almyrantis is right in comments: it’s the Assassins. And Dimitris, post more about the Kharijy?