I assure you it is the right tool for the job!
No, you're thinking of the recolor tool as an ordinary paint brush. It's not. Think of it as a selective brush. It will only recolor the bits that fall within the range of the Secondary color (modified by the tolerance setting in the toolbar). This means that you can be sloppy with the margins because the white background is not even close to the original color of the apple