Exile target creature. That creature's controller reveals cards from the top of their library until they reveal a creature card. That player puts that card onto the battlefield, then shuffles the rest into their library.
If the player reveals their entire library without revealing a creature card, that player just shuffles their library.
If the target creature is an illegal target by the time Transmogrify tries to resolve, the spell won't resolve. No player reveals cards or shuffles their library.