Couples with fertility problems are three times more likely to have a child with serious conditions like autism and cerebral palsy
The extra risk is likely to be caused by health problems that make it difficult for these couples to conceive in the first place, scientists believe.
Fertility treatments, such as IVF, may contribute too, an American Society for Reproductive Medicine meeting heard.
But the experts stressed the overall risk was still relatively low.
They said couples should be counselled about the risks and encouraged to improve their health before undergoing fertility treatment.
Professor Mary Croughan, who led the University of California research on 4,000 women and their children aged up to six years, explained those with fertility problems were also more likely to have other health problems, such as heart disease and diabetes, and were more at risk of pregnancy and labour complications.
She said: "What has caused them to be unable to conceive goes on to cause problems.
"It is as if a brick wall has stopped you becoming pregnant. Treatment allows you to climb over the wall, but it is still there and it goes on to cause problems."
Her team found the risk of five conditions - autism, mental retardation, cerebral palsy, seizures and cancer - was 2.7 times higher among the children born to 2,000 women who experienced fertility problems than among those born to the 2,000 women who did not have difficult conceiving.
For autism alone, the risk was four times higher.
Moderate developmental problems such as attention deficit hyperactivity disorder, learning disabilities or serious sight or hearing disorders were also 40% more common in the children born to the couples who struggled to start a family.
Infertility link to autism in children
Children born after IVF treatment 'face higher health risks'
Older eggs pass on fertility risk
Women risk passing infertility to children if they delay motherhood
Older mothers 'put their daughters at risk of infertility'