There are no rules, but there are trends.
- -ish is used for country names that the English would have been familiar with in the Middle Ages.
- -ese is used for country names that the English learned of via the Italians or Spanish. That includes East Asia.
- -(i)an is used as a default for new-fangled country names, or names which look like they are Latin. That includes all country names ending in -y and -ia.
- -i is used for Arabic-speaking countries; it is an Arabic suffix.