Hardcore Test Automation

How SQLite Is Tested is an extremely detailed account of how the team behind the world’s most popular embedded database verify its robustness.

It’s worth at least a skim read to see what they do. Some things jump out:

  • Code required to comprehensively check other code is necessarily significantly larger and more complex than that which it’s testing. We already know that, but in SQLite’s case the testing code is 690 times larger than the source code.
  • Regression testing must include specific checks for every bug ever reported, which are then used to show that the fixes hold for each release. Again, no surprises there, but the consequence is that this effort quickly snowballs into a lot of extra work and new releases take longer and longer to deliver.
  • Static analysis has never proved useful (for them): not a single bug has been detected using this method. Even though this is mostly because their automated testing does such a thorough job it’s worth investigating how that might apply to your testing.
  • They only achieved 100% branch coverage in August of this year. Clearly a high watermark for them but also a goal that takes a very long time to realise.

Leave a Reply