Choosing the best repo for your test automation code

While the debate between app repo and test code repo approaches continues, a hybrid solution offers a practical compromise [View this email in your browser]( by Leonid Khudisman | [Read online at Ministry of Testing]( Where should the project's test code live? A decades-old debate Ever since the engineering community began to include software testing as part of the Software Development Life Cycle (SDLC), we've been arguing about the proper home for test automation code. Should test automation code coexist in the same repository as the application code it is verifying (the "app repo")? Or is it better to have it live in a separate repository, away from the main codebase (a "test code repo")? This debate is almost as intense as the tabs versus spaces controversy. This article explores the arguments for both sides, highlighting the pros and cons of each approach. It proposes a hybrid solution based on the author's experience and discussions with engineering team members. The article emphasizes the importance of a "culture of quality" and the role of quality engineers as quality advocates. It also discusses the implementation of pre-commit hooks and the use of tags in pytest to create a fast feedback loop and efficient continuous integration and delivery / development (CI/CD) pipeline. The conclusion calls for larger studies to guide best practices in the QA community. What's the debate about? Who are these modern-day [Hatfields and McCoys]( Let’s look at these groups more closely. For clarity, I will not discuss the related controversy of whether automation code should be written in the same language as the main codebase. We will leave that topic for another time. App repo advocates say: easier to maintain test code in the app repo App repo advocates feel strongly that anything having to do with the app codebase should reside in that app codebase. This would include unit testing as well as any test automation. The decision is often made based on the test automation framework used and the idea that it is easier to maintain test automation code once it is part of the app repo. Test code repo advocates say: consider test code complexity Test code repo advocates tend to consider the complexity of the test automation project. And they believe that only app code should be in the app repo. Trial and error is our best source of information As I was writing this article, I tried to find out if anyone had ever conducted a formal study that would indicate how these decisions are made and where the testing code usually resides. Unfortunately, I found no such studies. Therefore, I have to rely on anecdotal evidence and my own experience in this matter. Let’s dive deeper into the pros and cons of keeping test automation code in the application repository versus maintaining it in a separate test code repository. 👋 [New Course: Prompting for Testers]( As Generative AI continues to gain popularity, knowing how to use it effectively can make a big difference in your daily testing tasks and long-term career goals. A key skill in making the most of this technology is prompt engineering—the practice of designing and refining prompts to get the best results from large language models (LLMs). Pros and cons of both approaches The pros of using the app repo - Keeping automation code in the app repo ensures tests are always in sync with the latest application code changes - Everyone can track changes to both application and test code in a single repository - CI/CD pipelines can more easily access and run tests alongside application code builds - Improved collaboration between developers and QA, fostering better communication and shared ownership of quality - App and test code share a consistent environment - Changes to application code can be immediately tested without switching between repos - Reduced maintenance overhead since there is only one repo to manage - Developers and QA can review each others' code - Shared dependencies can be managed in one place - Easier debugging of test failures The cons of using the app repo - Combining application and test code can make the repository more cluttered and harder to navigate - With more code in the same repository, the likelihood of merge conflicts increases - Running tests as part of the same pipeline can slow down the build process, especially if the test suite is extensive - Test code might expose sensitive information or internal APIs that should not be accessible from the main application codebase - Keeping test code in the app repo can limit the ability to reuse tests across multiple projects or applications - Developers might be less focused on writing and maintaining high-quality test code if it is mixed with application code, leading to potential neglect of testing practices - As the project grows, maintaining a single repository for both application and test code can become unwieldy and harder to manage - Mixing application and test code can blur the lines between development and testing responsibilities - Managing dependencies for both application and test code in the same repository can lead to conflicts and increased complexity - Different teams might require different access permissions, and having both application and test code in the same repository can complicate access control management The pros of using a separate test code repo - Keeping the app repository focused on application code can make it easier to navigate and manage - Test code can be versioned independently from the application code - A separate test code repo can facilitate the reuse of test code across multiple projects - Different access permissions can be set for the test code repo, allowing only specific team members to modify test scripts - Separate CI/CD pipelines can be set up for the test code repo, allowing for more specialized and optimized testing workflows - Reduced app repo size by keeping the test code separate - Distinguishing between application and test code makes it easier to manage and understand each part of the project - Dependencies specific to testing can be managed independently, avoiding potential conflicts with application dependencies - Teams can work on test code without affecting the main application code, allowing for parallel development and testing efforts - Having a separate test code repo can make it easier to scale testing efforts The cons of using a separate test code repo - Keeping test code in sync with application code changes can be challenging - Keeping versions of test code aligned with application code versions can be challenging - Changes to application code may not be immediately reflected in tests, potentially delaying the discovery of issues - Harder for developers and QA to collaborate - Integrating tests from a separate repo into CI/CD pipelines can be more complex and require additional configuration - Managing two separate repositories requires more effort and time for maintenance - Reviewing changes across two repositories can be time-consuming and complex - Shared resources or utilities might be duplicated across repos - Developers may have less visibility into test code, potentially leading to a disconnect between development and testing efforts - New team members may need to set up and understand two separate repositories, increasing the learning curve Presenting: a hybrid solution to the repo question Wow. Good points on both sides. No wonder this argument has never been resolved definitively. One thing I can do is provide a compromise that may work for many organizations. Why is there a need to choose one approach when, in reality, a hybrid approach would work best? I will base my proposal on my own experience and my conversations over the years with members of the engineering team, on both the dev and QA sides. A testing plan is just like baklava because it has many layers, is carefully constructed, and when done right, it's a sweet success :) Which code should live in the app repo? Unit tests belong in the app repo In my experience and speaking with other members of the QA community, the best home for unit tests is the app repo. I do not think we get a lot of arguments that unit testing should live outside in its own test code repo. Contract tests work best in the app repo The second type of test I would include in the app repo is contract tests. Contract testing is a testing method that ensures that two systems, such as services or microservices, can communicate and interact with each other as expected by verifying that the "contract" between them is correctly implemented. Normally, I would argue for integration testing to live outside of the app repo for reasons I will list later. However, contract testing is a special kind of integration testing. It is the best way to ensure that services can communicate correctly without the overhead of full end-to-end tests. Instrumenting the tests in the app repo I would include both unit and contract tests in the pre-commit hook with lint, static code, and security analysis tools. This practice creates a fast feedback loop for the developers and ensures the pull requests created abide by the processes installed. Which code should live in the test code repo? Short answer: all other types of tests! I recommend that every other type of test should live outside the app repo in its own test code repo. Take a step back and think: why do we want to use different layers of testing? What results do we want to achieve? Add infrastructure and limit mocking in the test code repo When you are planning to do integration, functional, accessibility, end-to-end, and other types of testing, you want to add infrastructure as one of the components under test and you want to limit mocking as much as possible. Grouping tests in the test code repo with pycode tags I am also a believer in the DRY (Don't Repeat Yourself) principle and feel that it makes sense to keep test automation code in the same repo to reduce duplication of code and logic to improve maintainability and reduce errors. You should use something like the concept of "tags" in pytest to group relevant tests in multiple suites of tests. Those suites of tests can then be used in the CI/CD pipeline. In the CI/CD pipeline, after the pre-commit hook passes and the pull request is ready to be created, you need to start adding testing layers by using the suite tags we spoke about previously. In this scenario, you need to create access to only one test code repo. Then you can run relevant tests. Notifying the right people of test failures The final details of concern to a good QA advocate are to ensure that the team gets notified of failures quickly, making data visible. Unfortunately, in my experience and speaking to others, once a CI/CD pipeline runs, no one monitors it to completion. We need to make sure we give fast feedback and notify the right people when failures occur. And we also must collect historical data to monitor trends and identify improvements. The hybrid repo approach at a glance Our exploration of testing strategies has highlighted the importance of a comprehensive approach to ensuring the quality and reliability of software. From pre-commit hooks to CI/CD pipelines, and from unit testing to security testing, each step plays a crucial role in the overall testing process. Here is a table to recap the key components of this process, emphasizing the need for further research and standardization in the QA community. Table 1: Test types and instrumentation appropriate to app repos and test code repos Pre-Commit Hook (App Repo) CI/CD Pipeline (Test Code Repo) Unit testing Smoke testing Contract testing Sanity testing Lint API testing Static code analysis UI testing Static security analysis Security testing The QA community should consider conducting larger studies to guide best practices but, in the meantime, our experience is the only thing we can rely on. To wrap up In conclusion, while the debate between app repo and test code repo approaches continues, a hybrid solution offers a practical compromise. By keeping unit tests and contract tests in the app repo and other types of tests in a separate repo, teams can benefit from the advantages of both approaches. This strategy supports a culture of quality and facilitates shift-left testing, ultimately leading to more efficient and effective software development processes. "I found a community in the Ministry of Testing, a form of belonging which helped me grow personally and in my career." — Kim Knup [Upgrade your software testing career with Ministry of Testing]( [Website]( [LinkedIn]( [YouTube]( [Twitter]( [Instagram]( Copyright © 2024 Ministry of Testing, All rights reserved. You have opted to join this email list. Our mailing address is: Ministry of Testing 19 New RoadBrighton, East Sussex BN1 1UF United Kingdom [Add us to your address book]( Want to change how you receive these emails? You can [update your preferences]( or [unsubscribe from this list](.