First off, let's get something out in the open. As an agile software developer, I have a couple of strongly held beliefs.
Design isn't optional.
Architecture isn't optional.
DTSTTCPW doesn't mean "no up-front design, we'll just wing it." [That's DO THE SIMPLEST THING THAT COULD POSSIBLY WORK for all y'all who don't speak acronym.]
When we fail to refactor our new code into the existing code, we get odd bits of functionality that stick out from the existing design like warts on a bowling ball. The idea behind refactoring is that we intend to make sure that we *always* have a round ball. Refactor doesn't mean just change a signature or two. Sometimes it means we need to go back to the design white board and figure out what we need to have *now* given the current functionality requirements. We almost always need to change the existing code's design and integrate the new code so that all appears cohesive, each and every time we write new code. Iterative development doesn't intend that we make a bunch of atomic changes and just glue them together.
Code design at each check-in should be able to withstand scrutiny of any outside reviewer, and the design should be up to engineering standards. We still call it software *engineering* not software *art*. Many teams I see are missing this point completely and are producing code that should have been re-designed rather than just having new code glued on. With each check-in we should be able to look over our classes and functionality and say, "yes, this is something that would withstand the scrutiny of a code reviewer not on my team, without excuses or explanation." If I have to couch it and say "well, we will be fixing that in the next release" or "we'll revise that in a later check-in" this is technical debt, and therefore a process smell. Developers that are stuck in this rut need to be re-trained that they aren't "done" with the code until it is production-quality.
If there are warts on your software, this is a very loud process smell, and it should be addressed directly. Make sure that the team understands that this is not an acceptable way to work on an agile team. Re-design in refactoring and adding new code should be common, and discussed among the team after the stand-up meeting daily. "Yesterday I re-factored and re-engineered the Widget classes, so that they now have the new functionality we need in story XXX and are more streamlined to what we are delivering this sprint. Today I plan to have a half-hour review with the team so everyone is on the same page with the re-design. Then I will incorporate all the feedback, get the final code reviewed and coordinate the check-in with the rest of the team to make sure I don't block anyone. I'll make the appointment for the review with everyone after the stand-up."
Make sure that everyone understands that the code as a whole should be well-designed at each commit point, and that this isn't optional. It needs to be expected behavior on the team. This also means that estimation of effort may (and should) go up, for complexity on tasks in both story points and hours. We don't look at revising design as "additional work" just like we don't view unit tests as "additional work." These things are both fundamental parts of the iterative process. Without adhering to these principles, iterative development just won't work, and we might as well go back to waterfall.
If your team has the process smell of needing to budget additional work for refactoring the design, perhaps it's time to go to the team and discuss what Agile software development really means, and that it doesn't mean lower-engineering quality software.