Bits 'n Widgets - testing

Automated Acceptance Tests - Who Should Write Them, Dev or QA?

John Boal — Fri, 22 Aug 2008 04:21:44 GMT

"Obviously, Development should be responsible for the acceptance tests. The developers are the ones on the team who are the best at writing good code. The developers are best able to turn the acceptance criteria into valid executable specifications. Development then can use the automated acceptance tests to drive the development TDD cycle."

"In QA, we never totally trust the developers. We do trust them to do the right thing. However, we have a saying in QA - Trust but Verify. QA should be responsible for the acceptance tests. At the end of the day, it is our job to make sure that the software meets the criteria for acceptance, so it's really a QA ballpark. Dev's should focus on unit tests."

Two strong opinions, and really I think both are right. Development should be using acceptance tests to drive an ATDD cycle. Developers are (usually) better at writing code than SDETs, so they do a more efficient job on acceptance tests. But QA probably has a more even overall view of the software, and from a different and more independent perspective, so they should be writing the acceptance tests as well as the other test automation. QA also has the responsibility of signing off on the code, so they should be writing the acceptance tests.

So, this may be one way to resolve the issue. Have developers and testers both work on automated acceptance tests for the first story. When the tests are completed, developers are free to begin working on the unit tests and code to satisfy the acceptance tests. QA can then begin test case generation and automation implementation.

I believe that this solution is the best of both worlds, because Development and QA will be completely in sync and on the same page for all of the stories where the acceptance tests are co-developed. There won't be any ambiguity on whether the software meets criteria, and both teams will be in agreement as to what is passing and what isn't. The tests will be well-written, and they will run as efficiently and accurately as possible. They will be written with multiple perspectives - the customer (who supplied the acceptance criteria), QA who makes sure that the test accurately guarantees the outcome, and Development, who makes automation quicker to implement and more efficient.

With automated acceptance tests co-written by dev and test, we have a great common point of communication, and a good example of how teamwork can really help software shine and inspire confidence and trust in the eye of the customer.

Selenium and WatiN code examples, ATDD presentation

John Boal — Thu, 21 Aug 2008 17:22:20 GMT

I have posted the acceptance test code in both Selenium and WatiN on the AgileBUG.com site. This code is a short demo and example of how we can do acceptance testing using both tools. The presentation is a short one on what ATDD is, and why we should do it. It serves as a prelude to the tools demo.

Join us at the Owl and Thistle tonight in person for a pint and some good agile discussions!

How to Implement Software Security on an Agile Team

John Boal — Sat, 10 May 2008 19:54:11 GMT

Security has always been a challenge in software development. Being in an environment that has rapid ship cycles and iterative development does add challenges of its own when it comes to security.

Here are a few of the key concepts I intend to flesh out in the next few weeks:

Security Requirements

how to come up with security stories
how much is enough?
how much is too much?

Using automated tools

what tools are available

MetaSploit - multiple vulnerabilities: http://www.metasploit.com/
Fiddler - HTTP/S proxy, inspector, injector, manipulator. Fun for the whole family. http://www.fiddlertool.com/
WireShark - network protocol analyzer and sniffer: http://www.wireshark.org/
various other network tools like Snort, RetinA, NetStumbler can be automated and scripted.

use static code analysis tools, and pay attention to their results.
I recommend also doing file and network fuzzing on system entry points, but don't have any good tool recommendations. Got some? Please leave comments!

web site testing vs web service testing
application testing
how do the fit into automation frameworks

Security Documentation (Threat Models)

Designing in Security as Feature 0
Iterative Threat Modeling
Who Reads the Threat Model?
How do we turn threat models into automated acceptance tests?

security testing strategies

white route (internal folks, given the internals of the system)
black route (for-hire hackers, given only an objective to accomplish, and no system information)

security-oriented code reviews

how to train developers and testers to look for security defects

security vs. performance

Sometimes mitigations incur a performance hit. How do we avoid this, and what are some alternatives?

This is an Agile blog, so this is the first production release of this article ... More features (content) will become available over time, so stay tuned to this RSS feed for updates and new content, as they emerge.

Test Patterns - recurrent design patterns in testing

John Boal — Thu, 10 Apr 2008 20:17:50 GMT

We all know what design patterns are in software design and development. These kinds of patterns also are recognized in unit and other kinds of tests as well.

While it is not necessarily a new idea, it is a good idea. Here are some links I have found on the subject. Further research should yield a whitepaper soon, if I ever get time to write it.

Brian Marick's testing.com: http://www.testing.com/test-patterns/patterns/

A great example at CodeProject: http://www.codeproject.com/KB/architecture/autp5.aspx

RBSC: http://www.rbsc.com/pages/TestPatternList.htm

TypeMock Unit test patterns for .net: http://www.typemock.com/Docs/TestPatterns.html

Book: xUnit Test Patterns: Refactoring Test Code

BDD, ATDD, DSL's - It's not your father's TDD anymore...

John Boal — Fri, 14 Mar 2008 20:19:07 GMT

BDD, ATDD, UTDD, DSL's ... when will it all end...

The drive toward business-driven testing has never been stronger. Developers are seemingly now finding a higher and higher bar when it comes to customers' expectations of quality and features. Our tools are getting better, and we can deliver more software, faster. But, our methodologies haven't necessarily changed enough to satisfy today's customer expectations.

Enter Business-Driven Design...

Business driven design is a concept that enables us to take business requirements and current priorities and turn them into a software design through Acceptance Test-Driven Development. The business requirements that drive the need for the software are turned into specific criteria that allow the business to decide what the criteria are that will allow them to use a feature and have it meet their business need. Rather than the old-school way of gathering requirements, and having a requirements document and a functional specification, we now turn to individual small criteria that decide if the software is acceptable to meet the need. Some of the criteria map directly from functional requirements, and others may not have been captured in a traditional requirements gathering and specifying model.

Domain-Specific Languages (DSL's) are key to success in Acceptance Test-Driven Development. DSL's give us a way to communicate with the customer and domain experts in their terms. When we capture criteria in this manner, it becomes quite clear to those with domain knowledge, what is meant and what is desired. There is no need for a "translator" between the customer and the developers (this used to be called "Business Analyst"). The developers model the code in terms of the language the customer already uses. This mechanism leads to better communication, better encapsulation, and better object-oriented development.

Acceptance Test-Driven Development [ATDD] gives us a mechanism to use DSL's and direct customer involvement in making sure the software we deliver meets the needs. When we take the criteria and turn them into automated acceptance tests, it is far easier for the customers to see that they are getting what they asked for. It's also easier for the developers to have a target to shoot for, and have a goal to meet. This way, they are more focused on delivering a specific unit of functionality that the customer needs rather than (as so often happens) some "new feature" that they thought might be useful.

Much care needs to be put into the way that acceptance criteria are gathered and then automated. If there is something that is missed, it could critically affect the design. This is an opportunity for customers and developers to collaborate and get it right. The customer needs to understand that if it isn't on the acceptance criteria list, it isn't going to be in the software... Performance criteria, interoperability with other systems, and other criteria like these are often missed. Customers should have many opportunities to review and re-review the criteria before they are approved. Even still, sometimes things are missed. This is why it is important for the customer to be involved at all stages of the development process. The customer shouldn't just be involved in the criteria gathering, then come back later for their product. If things are missed, they will likely become apparent and turn up in daily work. If the customer is there to be consulted, decisions can be made about how to integrate missed criteria, and how to capture these better in the future.

Business-Driven Design is a business-centric, collaborative, agile mechanism for delivering quality software to today's demanding customer.