Software Testing and Automation Lessons from a Real Automation Project

Here at Beta Breakers, we have been providing automated quality assurance solutions for quite a while. Having recently completed a number of fairly complex projects—on time and under budget—we had gotten to feeling pretty good about our ability to scope and fulfill an automation effort. We assumed that our recent good fortune had been due to our expertise. Then a project that appeared somewhat ordinary at the outset ended up a bit over budget and behind schedule. Here is what happened:

Our client needed us to use a tool that we had not used before. We performed a quick proof-of-concept that demonstrated we could integrate our work into their tool and concluded it was an insignificant obstacle. However, while our POC had verified the integration into a dev environment, the client needed integration into their production environment, which proved to be much more time-consuming than we anticipated.

Due to unexpected and unconventional behavior on the part of some of the custom program objects, automating the most critical areas of the program was much more difficult than we anticipated. Compounding this challenge was our tight estimate, which did not leave much room for error in either the schedule or budget. Though the culprits here were pretty subtle, we learned a couple things: make sure your proof-of-concepts are applicable and they address the areas of risk that are high priority; and always include a reasonably generous margin for the budget and timeline. Most importantly, beware of variables. Even though it is impossible to anticipate everything, at least acknowledge that there is always the potential for an unknown variable to have an impact on scope. These can be applied to everything we do at Beta Breakers, from software testing to quality assurance.