Thursday, August 21, 2008
"Obviously, Development should be responsible for the acceptance tests. The developers are the ones on the team who are the best at writing good code. The developers are best able to turn the acceptance criteria into valid executable specifications. Development then can use the automated acceptance tests to drive the development TDD cycle."


"In QA, we never totally trust the developers. We do trust them to do the right thing. However, we have a saying in QA - Trust but Verify. QA should be responsible for the acceptance tests. At the end of the day, it is our job to make sure that the software meets the criteria for acceptance, so it's really a QA ballpark. Dev's should focus on unit tests."

Two strong opinions, and really I think both are right. Development should be using acceptance tests to drive an ATDD cycle. Developers are (usually) better at writing code than SDETs, so they do a more efficient job on acceptance tests. But QA probably has a more even overall view of the software, and from a different and more independent perspective, so they should be writing the acceptance tests as well as the other test automation. QA also has the responsibility of signing off on the code, so they should be writing the acceptance tests.

So, this may be one way to resolve the issue. Have developers and testers both work on automated acceptance tests for the first story. When the tests are completed, developers are free to begin working on the unit tests and code to satisfy the acceptance tests. QA can then begin test case generation and automation implementation.

I believe that this solution is the best of both worlds, because Development and QA will be completely in sync and on the same page for all of the stories where the acceptance tests are co-developed. There won't be any ambiguity on whether the software meets criteria, and both teams will be in agreement as to what is passing and what isn't. The tests will be well-written, and they will run as efficiently and accurately as possible. They will be written with multiple perspectives - the customer (who supplied the acceptance criteria), QA who makes sure that the test accurately guarantees the outcome, and Development, who makes automation quicker to implement and more efficient.

With automated acceptance tests co-written by dev and test, we have a great common point of communication, and a good example of how teamwork can really help software shine and inspire confidence and trust in the eye of the customer.
Thursday, August 21, 2008 8:21:44 PM (Pacific Standard Time, UTC-08:00)  #    Disclaimer  |  Comments [0]  |  Trackback
I have posted the acceptance test code in both Selenium and WatiN on the AgileBUG.com site. This code is a short demo and example of how we can do acceptance testing using both tools. The presentation is a short one on what ATDD is, and why we should do it. It serves as a prelude to the tools demo.

Join us at the Owl and Thistle tonight in person for a pint and some good agile discussions!


ABN | TDD | testing
Thursday, August 21, 2008 9:22:20 AM (Pacific Standard Time, UTC-08:00)  #    Disclaimer  |  Comments [0]  |  Trackback
 Saturday, May 10, 2008
Security has always been a challenge in software development. Being in an environment that has rapid ship cycles and iterative development does add challenges of its own when it comes to security.

Here are a few of the key concepts I intend to flesh out in the next few weeks:
  • Security Requirements
    • how to come up with security stories
    • how much is enough?
    • how much is too much?
  • Using automated tools
    • what tools are available
      • use static code analysis tools, and pay attention to their results.
      • I recommend also doing file and network fuzzing on system entry points, but don't have any good tool recommendations. Got some? Please leave comments!
    • web site testing vs web service testing
    • application testing
    • how do the fit into automation frameworks
  • Security Documentation (Threat Models)
    • Designing in Security as Feature 0
    • Iterative Threat Modeling
    • Who Reads the Threat Model?
    • How do we turn threat models into automated acceptance tests?
  • security testing strategies
    • white route (internal folks, given the internals of the system)
    • black route (for-hire hackers, given only an objective to accomplish, and no system information)
  • security-oriented code reviews
    • how to train developers and testers to look for security defects
  • security vs. performance
    • Sometimes mitigations incur a performance hit. How do we avoid this, and what are some alternatives?


This is an Agile blog, so this is the first production release of this article ... More features (content) will become available over time, so stay tuned to this RSS feed for updates and new content, as they emerge.
Saturday, May 10, 2008 11:54:11 AM (Pacific Standard Time, UTC-08:00)  #    Disclaimer  |  Comments [0]  |  Trackback
 Thursday, April 10, 2008

We all know what design patterns are in software design and development. These kinds of patterns also are recognized in unit and other kinds of tests as well.

While it is not necessarily a new idea, it is a good idea. Here are some links I have found on the subject. Further research should yield a whitepaper soon, if I ever get time to write it.

Brian Marick's testing.com: http://www.testing.com/test-patterns/patterns/

A great example at CodeProject: http://www.codeproject.com/KB/architecture/autp5.aspx

RBSC: http://www.rbsc.com/pages/TestPatternList.htm

TypeMock Unit test patterns for .net: http://www.typemock.com/Docs/TestPatterns.html

Book: xUnit Test Patterns: Refactoring Test Code

TDD | testing
Thursday, April 10, 2008 12:17:50 PM (Pacific Standard Time, UTC-08:00)  #    Disclaimer  |  Comments [0]  |  Trackback
 Friday, March 14, 2008
BDD, ATDD, UTDD, DSL's ... when will it all end...
 
The drive toward business-driven testing has never been stronger. Developers are seemingly now finding a higher and higher bar when it comes to customers' expectations of quality and features. Our tools are getting better, and we can deliver more software, faster. But, our methodologies haven't necessarily changed enough to satisfy today's customer expectations.
 
Enter Business-Driven Design...
 
Business driven design is a concept that enables us to take business requirements and current priorities and turn them into a software design through Acceptance Test-Driven Development. The business requirements that drive the need for the software are turned into specific criteria that allow the business to decide what the criteria are that will allow them to use a feature and have it meet their business need. Rather than the old-school way of gathering requirements, and having a requirements document and a functional specification, we now turn to individual small criteria that decide if the software is acceptable to meet the need. Some of the criteria map directly from functional requirements, and others may not have been captured in a traditional requirements gathering and specifying model.
 
Domain-Specific Languages (DSL's) are key to success in Acceptance Test-Driven Development. DSL's give us a way to communicate with the customer and domain experts in their terms. When we capture criteria in this manner, it becomes quite clear to those with domain knowledge, what is meant and what is desired. There is no need for a "translator" between the customer and the developers (this used to be called "Business Analyst"). The developers model the code in terms of the language the customer already uses. This mechanism leads to better communication, better encapsulation, and better object-oriented development.
 
Acceptance Test-Driven Development [ATDD] gives us a mechanism to use DSL's and direct customer involvement in making sure the software we deliver meets the needs. When we take the criteria and turn them into automated acceptance tests, it is far easier for the customers to see that they are getting what they asked for. It's also easier for the developers to have a target to shoot for, and have a goal to meet. This way, they are more focused on delivering a specific unit of functionality that the customer needs rather than (as so often happens) some "new feature" that they thought might be useful.
 
Much care needs to be put into the way that acceptance criteria are gathered and then automated. If there is something that is missed, it could critically affect the design. This is an opportunity for customers and developers to collaborate and get it right. The customer needs to understand that if it isn't on the acceptance criteria list, it isn't going to be in the software... Performance criteria, interoperability with other systems, and other criteria like these are often missed. Customers should have many opportunities to review and re-review the criteria before they are approved. Even still, sometimes things are missed. This is why it is important for the customer to be involved at all stages of the development process. The customer shouldn't just be involved in the criteria gathering, then come back later for their product. If things are missed, they will likely become apparent and turn up in daily work. If the customer is there to be consulted, decisions can be made about how to integrate missed criteria, and how to capture these better in the future.
 
Business-Driven Design is a business-centric, collaborative, agile mechanism for delivering quality software to today's demanding customer.
testing | DSL | TDD
Friday, March 14, 2008 12:19:07 PM (Pacific Standard Time, UTC-08:00)  #    Disclaimer  |  Comments [0]  |  Trackback