Elwes was born in Westminster, London, and is the third and youngest son of portrait-painter Dominick Elwes and interior designer Tessa Georgina Kennedy, who is of Croatian and Anglo-Irish descent.