Taming Flaky Tests in a Non-Deterministic World
24/02/2022, 10:30am
Speaker
Abstract
As software evolves, developers typically perform regression testing to ensure that their code changes do not break existing functionality. During regression testing, developers often waste time debugging their code changes because of spurious failures from flaky tests, which are teststhat nondeterministically pass or fail on the same code. These spurious failures mislead developers because the failures are due to bugs that existed before the code changes. My work on characterizing flaky tests has helped open the research topic of flaky tests, and many companies (e.g., Facebook, Google, Microsoft) have since highlighted flaky tests as a major challenge in their software development.
In this talk, I will describe my recent work on taming flaky tests. Two prominent kinds of flaky tests are order-dependent flaky tests, which pass when run in one order but fail when run in a different order, and async-wait flaky tests, which pass if an asynchronous call finishes ontime but fail if it finishes too late. My results include the first automated techniques to (1) fix order-dependent flaky tests, fixing 92% of such flaky tests in a public dataset; (2) reduce the number of spurious failures from order-dependent flaky tests,reducing such failures by 73%; and (3) speed up async-wait flaky tests while also reducing their spurious failures, speeding up such tests by 38%. Overall, my work has helped detect more than 3000 flaky tests and fix more than 1000 flaky tests in over 300 open-source projects.
Speaker Bio
Dr. Wing Lam is an Assistant Professor in the Computer Science department atGeorge Mason University. He is also a Part-Time Applied Research Scientist at Fraunhofer CMA. Dr. Lam works on several topics in software engineering, with a focus on software testing. His research improves software dependability by characterizing bugs and developing novel techniques to detect and tame bugs. Wing has published in top-tier conferences such as ESEC/FSE, ICSE, ISSTA, OOPSLA, and TACAS. His techniques have helped detect and fix bugs in open-source projects and haveimpacted how Microsoft and Tencent developers test their code. Wing has been awarded several fellowships and scholarships, including a Google - CMD-IT Dissertation Fellowship Award. More information is available on his web page: https://cs.gmu.edu/~winglam/