Category Archives: presentations

automation, mewt, presentations, regression, talks

Regression Testing: The Bane of the Tester

2 Comments

I was fortunate enough to be invited to the 2nd Midlands Exploratory Workshop on Testing (MEWT) which took place on the 13th September at the wonderful Attenborough Nature Centre.

(You can read more about the background to the MEWT concept and the day’s events on the MEWT blog.)

The theme of this year’s workshop was “Regression Testing”.

I’ve set out my presentation in long form below but if you want the original slides here they are: MEWT – Regression Testing

Initial thoughts

When I first read what the theme of this year’s workshop was I thought “Well, that’s a bit obvious. Surely everyone knows what Regression Testing is?”

It turns out there’s no such thing as a silly question, especially when it comes to testing.

So let me just set out what I understand regression testing to be, because I know already (thanks, Twitter) that there’s a difference of opinion in the room as to what that might be.

Again, I’m not going off what the “dictionary definition” says. I’ve deliberately not looked it up. I must present what I understand it to be, not what I’d like it to be, and certainly not what some other source says it is.

Regression Testing is…

Ensuring what has already been tested, and found to be good, remains so as the software continues to be developed.

Ideally you would re-run all your previous tests every time there is a change.

What’s the problem?

Q: Why is it the “bane of the tester”?

A: There is rarely (if ever) enough resource to cover a full regression test every time it is needed.

Reducing the load

There are some common tactics to reduce the workload…

1. Don’t retest everything.

  • Wait until there are enough changes built up to form a release, then test that.
    • Reduces effort with minimal risk.
  • Only test significant changes.
    • How do you determine “significant”?
  • Only test those areas likely to have been affected.
    • Again, sensible, but how is this determined?

2. Don’t do it all yourself.

  • Spread the testing effort out across the team.
    • Rerunning tests manually is dull. Rotate the tests. Share the pain.
  • Outsource the testing.
    • Rerunning tests manually is dull. Let factory testers do it.
  • Automate the testing.
    • Let the computer do the work. Now you have two problems.

Coverage

Regression Tests are…

  • All the tests you have run to check against requirements.
  • All the tests you have run to check the fixes are correct.

That’s a lot of tests.

Regression Testing vs Sapient Testing

Regression Testing is the antithesis of Sapient Testing.

  • Use sapient testing to explore the way ahead.
  • Use regression testing to shore up the road behind.

 

The one thing to take away from the workshop was that everyone came away with a different idea of what Regression Testing should be and could be than when they started that day. (As I’ve already said.)

I’d say that, on that basis alone, it was a worthwhile exercise.

99-second, crisis, presentations, software, talks, testbash

99-Second Talk: Software Crisis?

1 Comment

As delivered in under 1:39 at TestBash 2.0, Brighton, 22nd March 2013.

[This is from my notes. It probably isn’t what I actually blurted out. Video evidence to follow, probably…]

While thinking about what to talk about today for some reason a term that I had not really encountered since university days came to mind. Has anyone heard of the term Software Crisis?

[One person put their hand up: Michael Bolton IIRC.]

Well there’s a reason for that. It was coined back in 1968 at the NATO Conference on Software Engineering held in Germany. It was (is?) an emotive, provocative term that posited this: with the increase in complexity of software made possible by rapid advances in computing power, how can we ever successfully manage such projects? We can do anything with a computer we can set our minds to. There is no bound to things that are virtual, and whose mechanics are invisible. To paraphrase Fred Brooks, there will never be enough human resources to build (provably) correct software.

  • So, is the term an anachronism?
  • If the term no longer has currency [it seems that way by show of hands hand], are we still in crisis or out of the woods?
  • Are software development methodologies are simply various ways of managing crises?

meetup, presentations, talks

Five Misleading Software Testing Concepts

1 Comment

This week I gave a lightning talk about Five Misleading Software Testing Concepts at the STC Meetup in Cambridge. They were:

  1. Automated Testing
  2. Quality Assurance
  3. Best Practices
  4. Bug Counts
  5. Code Complete

My cue cards can be found here.

[Edit: 29/03/12] I also gave this talk at the STC Meetup in Birmingham last night. Here’s a blogged response already.

[Edit: 03/04/13] Once again (and for the final time I think) I gave this talk exactly a year later at the inaugural Sheffield Test Gathering on 28th March.

Permit me some self-promotion here:

.@pmberry2007 provoking some lively discussion at the first #shefTest. twitter.com/WhyAyeMac/stat…
— Iain Mc (@WhyAyeMac) March 28, 2013

bdd, cucumber, gherkin, meetup, presentations

Cucumber is inherently compromised

6 Comments

Background

I attended the STC Meetup in Birmingham on Tuesday evening and was struck by a realisation towards the end of Simon Knight‘s lightning talk on the pros and cons (they were mostly cons) of Cucumber. More on this epiphany later…

By happy coincidence I had recently started to look into Cucumber as a concept because our development team on the SAMS Programme here at Nottingham are trying to move towards embracing continuous integration as a methodology. I have read and digested the Cuke4Ninja how-to PDF and it seems, on the surface, to be addressing some of the failings of “traditional” software development. Its main thrust is the use of Gherkin, a form of Structured English, to enable requirements specifications, or derivations thereof, to be put forth in much tighter and more precise terms than are commonly used. Bluntly the idea is that by aiming to reduce ambiguity, better software will result faster. Riding on the coat-tails of this are programming language libraries (Cucumber is by far the dominant one) to make Gherkin statements executable and thus testable in some limited sense.

Criticism

With good timing James Bach (also at the Meetup) had recently weighed in with some strident criticism of the Behaviour-Driven Development (BDD) methodology that Cucumber is a tool for.

Realisation

I realised during Simon’s presentation that Cucumber tries to position itself at the intersection of Analysis, Development and Testing (visualise a Venn diagram of these roles) and is therefore essentially a compromise between all three.

To quote from Cuke4Ninja (page 13 of the PDF):

Cucumber is a functional test automation tool for lean and agile teams. It supports behaviour-driven development, specification by example and agile acceptance testing. You can use it to automate functional validation in a form that is easily readable and understandable to business users, developers and testers. [my emphasis]

The point I raised during the Q&A was that since each group already produces their own form of documentation particular to their viewpoint (Business Analysts write requirements specifications, Developers have their code, Testers produce test plans, etc) how can any documentation possibly sit between all three and not be compromised? Who would author such documentation? Who would own it? And who would realistically countenance adding what would be a fourth layer to the mix when documentation is hard enough to do anyway? It is yet another set of documents to be kept in synch with the true requirements, which are hard enough to ever nail down.

Caution

If BDD is half of what it’s cracked up to be, such an investment might be worthwhile. There are some interesting concepts there and not all are without merit. But it is the nature of people to overstate the worth of something new so we need to exercise caution while at the same time being open to new ideas. I’m reminded once again of Fred Brooks’ paper No Silver Bullet which still stands as an undefeated caveat to the entire software industry 25 years since it was published.

Futher Work

I leave you with these notes I jotted down after the session. Despite the scepticism above, I still intend to do this, though I admit my enthusiasm has been somewhat tempered:
Cucumber/Gherkin
– learn it
– apply it
– report on your experiences
– bask in the warm glow/icy frost of your peers

Watch this space.