Gregg Wallace answered accusations of inappropriate behaviour by saying the claims came from ‘a handful of middle-class women of a certain age’.