Stocks closed lower Wednesday but booked another year of big gains as a tumultuous year in markets came to an end.