Thank you OED:
four < *fowr < Middle English fower < feower < Old English feower
forty since 15th century; fourty Middle English up to 17th century < Middle English fourti (and, in parentheses, forti) < feouwerti < Old English feowertig .So the forti spelling was apparently occasional in Middle English, but not regular.
This took some hunting, and OED wasn’t as much help as I’d hoped. I ended up going to Jespersen A Modern English Grammar on Historical Principles, §4.69
The regular development was fowər > fowr > foːr > fɔːr.
Jespersen notes, though, that vowels are shortened a lot of the time when they are in the first part of a compound, or preceding a consonant cluster. That’s why five goes to fifty with a short vowel—already in the Ormulum (12th century). You can see that with an original long o in nose > nostril, holy > holiday. You can also see that with an original diphthong ow: know > knowledge, Gloucester /glowsester > gl ɔstə/, rowlock [rɔlək], and, Jespersen says, forty from four.
Now, not so fast. The spelling tells Jespersen that fowərti went to fɔrti with a short o, just like know went to knowledge with a short o. But I pronounce the vowels in four and forty the same, long. What’s happened?
One of the following three things, and possibly all three:
- forty is spelled according to the original Early Modern English pronunciation, and used to be pronounced /fɔrti/ accordingly. It is pronounced /fɔːɹti/ now by analogy with four.
- forty is pronounced /fɔːɹti/ in my dialect, because that’s what r’s do; so my dialect has gotten rid of any distinction between [fɔːɹti] and [fɔɹti]. Americans?
- EDIT: I read English-language vowel changes before historic /r/ the hoarse/horse merger correctly, mourning and morning are pronounced differently in South Carolina, Alabama, and in Scots. If there is a historical survival of short o forty, it will be found in those dialects.
- Likeliest: fourty (with a long vowel) and forty (with a short vowel) were both around in Early Modern English as different pronunciations—as the Mediaeval spelling forti hints at. The short vowel version was acting like fifty, in shortening the ow before the –ty. The long vowel version kept the historical form, whether because of analogy with four, or because the diphthong in fowrty was a late development.
The fourty spelling dies out quickly in the 1600s, and I am convinced by the guess in Why is ‘forty’ spelled without a ‘u’ in Canadian/British English? : the King James Bible happened to choose forty instead of fourty, at a time of orthographic whimsy and/or phonetic instability. And that was that.
What I think is less likely is that four and fourty were pronounced identically in Early Middle English, and the King James switched spelling to forty just because. Things like that do happen, English being English; but because forti was a mediaeval spelling, I think there was an underlying pronunciation difference.
Obligatory disclaimer: this is all a guess.