During antiquity, did anyone in Greece or Rome recognize similarities between Greek and Latin languages and hypothesized relationships between them?

Yup.

Aeolism: Latin as a Dialect of Greek/Aeolism: Latin as a Dialect of Greek is a paper on that. And Are there any accounts of the Romans realizing linguistic similarity between Latin and Germanic languages? • /r/AskHistorians is a Reddit thread of it.

The locus classicus is Dionysius of Halicarnassus, but the idea was doing the rounds:

LacusCurtius • Dionysius’ Roman Antiquities

The language spoken by the Romans is neither utterly barbarous nor absolutely Greek, but a mixture, as it were, of both, the greater part of which is Aeolic; and the only disadvantage they have experienced from their intermingling with these various nations is that they do not pronounce all their sounds properly.