Viewer numbers cool as competition hots up on X Factor