With the overclocking software (usually MSI Afterburner) you lower the core voltage by mV increments. Or you can lower power limit percentage. Most ppl use the percentage. Then lower it by 5 and see how stable the system runs for about 30mins or so. If that works, drop it another 5%. If it starts giving issues, then bump it up 1% until it becomes stable.
All this while you can bump the Memory speed up (not necessarily gpu speed) will help get better performance, stable results, lower power consumption.
This is, of course, super quick explanation and I may be off. I'm sure mike will chime in with more details, but this is how I understand it.