What can I do about builders incorrect estimate?

A builder estimated the amount of boulder rocks needed for him to edge our flower beds and I went and bought the amount he advised. However, now he has finished the job he says he over-estimated the amount of rocks needed, and sorry but we have some left over. We are pretty cross as we have paid for rocks which are not needed, and cannot take them back to the supplier. Should we ask the builder to deduct the cost of the rocks from his bill, or is it just our loss?