The advantages to be gained from an active CRO program depend on a number of factors. Do you have enough traffic? How big is your margin? How good is your CRO agency?
A/B testing (which is also sometimes known as split testing or bucket testing) allows you to compare the performance of two versions of a web page or app to see which one performs better for a defined success metric or conversion.