He had a notorious relationship with Lady Grosvenor but married the Hon. Anne Horton, widow of Christopher Horton and eldest daughter of Simon Luttrell, 1st Earl of Carhampton, on 2nd October 1771.