The ascendancy of English as a world language of diplomacy is after WWII. Before WWII, it was French, and keeping French around as the second world language in international organisations (including the Olympic movement) is mostly legacy.
There are countries where French gets you further than English, but those countries now are not European countries (other than France, Wallonia, Luxembourg and Suisse Romande); they are the African Francophonie.