In his letter to the committee, dated April 9, Justice Yashwant Varma said he was "recusing" himself from the proceedings because "a fair inquiry is not being conducted"