If you’re being brought up to speak Esperanto or Klingon or Lojban or (in the case of Itamar Ben-Avi) Revived Hebrew [yes, I’m calling Eliezer Ben-Yehuda’s work a made up language], the main issue you’d run into is not having anyone but your parents, and maybe occasionally your parents’ weirdo friends, to use the language with.
That is actually a very common dealbreaker for kids with Esperanto, and the parents end up acquiescing; there may be 10k denaskaj Esperantistoj (native speakers of Esperanto) that are still engaged with the language, but there are a lot more that aren’t. This got addressed in the surveys behind Peter Forster’s book The Esperanto Movement. I haven’t asked him personally, but I think it’s a big reason why Alec Speers gave up and D’Armond Speers acquiesced, with Klingon. Itamar, unfortunately, was not given the option, which is why he could only talk to his dog as a kid.
(I know someone bringing up his kid to speak Lojban, and my Facebook feed has intermittent reports of how it’s going; but I haven’t been following it. Lojban is certainly going to be a lot more alien than Klingon.)
A second issue, which I’ve heard for Esperanto and which D’Armond certainly reported for Klingon, was the lack of vocabulary that you can use with a kid around the house. It’s not necessarily that Esperanto lacks such vocabulary, but that Esperantists usually don’t learn that vocabulary, because that’s not the context in which they use the language. Just as people who learn foreign languages formally usually don’t end up learning the word for armpit. So you may grow up with circumlocutions or ad hoc words.
Chomskyans may mutter darkly that if you are brought up to speak a made up language, that will warp your language acquisition FOREVAH, and that bringing up a kid to speak Klingon is somehow child abuse. I even heard that from non-Chomskyans.
Poppycock. Kids survived being brought up in slave plantations creolising their parents’ pidgins without sustaining brain damage; the brain is a flexible thing, far more flexible than knob-twiddling universal parameters gives it credit for; and in any case, no kid is being brought up with no exposure ever to natural languages in parallel. (Not even Itamar. Poor kid.)