Hmm...that's strange.
Are the 20 fry in the same tank as the adults you've lost?
If so, then I'd say you might be looking at a bully who is stressing and picking at the others to death. Just a guess though. Have you noticed one of them bullying the others incessantly or anything like that? Probably you would've mentioned that already. Just throwing some ideas out there...
All I know is that if there's something in the water responsible for killing adult fish, then the fry would almost certainly succumb to it as well. You've got me stumped.
BV