This reverts commit 7d3ecfd796.
This change had caused a regression with swr-haptics driver probe
as the soundwire master couldn't enumerate swr haptics. Revert
this until we can understand why soundwire enumeration fails.
Change-Id: Ie5840247013a1a3fab15758e9ace7b131619be49
Signed-off-by: Subbaraman Narayanamurthy <subbaram@codeaurora.org>