In try_to_bring_up_master(), we tear down the master's component list
for each error case, except for devres group failure.  Fix this
oversight by making the code less prone to such mistakes.

Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>