Automating screenshot and log file capture for Allure reports

Discover how a custom Allure Attachments Handler streamlines debugging and enhances test reports. [View this email in your browser]( by Nick Karamaniolas | [Read online at Ministry of Testing]( "Strategic tooling can lead to substantial improvements in software development workflows. It guarantees that comprehensive logging and diagnostic information are consistently captured and reported, supporting robust testing practices and fostering a culture of continuous improvement." Solving problems and automating work across projects: our shared QA library If your team has to navigate the complexities of multiple software projects, you know that it demands more than just expertise. It calls for innovative solutions. My team builds several different products. To ensure that we can keep delivering consistent, high-quality results across all of those projects, we developed a shared library that unifies test-oriented functionality across our projects and promotes reusability. The components of our shared library do many different things for us, and today I'll share with you how one of those components helps us report test outcomes and understand why a test failed. A star of our QA library: our very own Allure Attachments Handler At the heart of our reporting process is Allure, a lightweight test report generation tool that produces detailed and interactive HTML reports for test execution. It supports multiple languages and testing frameworks, making it versatile for different development environments. Allure helps teams visualize test results by providing a clear and comprehensive overview of the test execution process, allowing teams to better understand test outcomes and pinpoint issues quickly. (You can find out more about Allure [here]( Allure out of the box is a great solution, but without some custom work, you can't automatically attach artifacts like screenshots and log files to test runs. And so we developed the Allure Attachments Handler. Designed to support both Selenium and Playwright testing frameworks, it ensures comprehensive documentation and reporting of test artifacts, which streamlines the debugging process and enhances the clarity of test outcomes. This powerful tool assists debugging and reporting efforts by seamlessly integrating screenshots, log files, and Playwright trace files into Allure reports. [A standard Allure report with no attachments.]( 👋 [New Course: Prompting for Testers]( As Generative AI continues to gain popularity, knowing how to use it effectively can make a big difference in your daily testing tasks and long-term career goals. A key skill in making the most of this technology is prompt engineering—the practice of designing and refining prompts to get the best results from large language models (LLMs). Which problems were we trying to solve with respect to Allure? Before we developed the Allure Attachments Handler, our teams lacked clear, useful and concise reporting from Selenium and Playwright tests. Each project operated in isolation, resulting in duplicated efforts and inconsistent approaches to capturing and reporting test artifacts. The resulting increase in development time and cost made it difficult to standardize testing practices across the organization. And the lack of a unified method for capturing and reporting test artifacts often led to insufficient debugging information, hindering our ability to identify and resolve issues quickly. Build or buy? Exploring the testing toolbox Before we wrote a single line of code for a solution or spent any money on a third-party solution, we explored some alternatives and approaches that could potentially address our challenges. Here are some of the notable alternatives: - pytest-html: This pytest plugin generates attractive HTML reports. While useful, it lacked the comprehensive artifact attachment capabilities we needed, especially for capturing visual elements and detailed log files. The plugin's simplicity and ease of use were appealing, but it did not provide the depth of reporting necessary for our multi-faceted test environments. - Custom-built solutions: We considered developing a custom reporting solution tailored to our specific needs. This approach would give us full control over features and integrations. However, it also meant significant initial development and ongoing maintenance efforts, diverting resources from core development tasks. Creating a solution from scratch would have required a considerable investment in time and expertise to achieve the level of functionality and reliability we required. Ultimately, we chose to develop the Allure Attachments Handler. This decision was driven by our need for a unified, flexible solution that could integrate seamlessly with our existing testing frameworks and provide comprehensive reporting capabilities. Our solution incorporated several existing libraries to streamline the development process and ensure robust functionality: - Selenium and Playwright: We use these well-established testing frameworks to support both browser automation and comprehensive artifact collection. - Allure reports: By building on the powerful Allure reporting framework, we ensured that our reports would be visually appealing and rich in detail, meeting the needs of various stakeholders. - Existing pytest plugins: We used pytest plugins to integrate with our test suites, allowing us to build on familiar tools and minimize the learning curve for our team. By combining these established libraries with our custom enhancements, we created a solution that addressed our specific needs while benefiting from the reliability and community support of widely used tools! Our solution at a glance Our primary focus for this tool was to capture and attach detailed testing artifacts to Allure reports, enhancing our test reports with crucial data we would need in case of a failed test. The key features are: - Screenshots and log files for Selenium: The Handler automates the attachment of browser state screenshots, DOM sources, cookies, and console log files, providing a comprehensive view of the test environment at any time. - Trace files for Playwright: For projects using Playwright, the Handler includes trace files in the reports, which are crucial for a detailed analysis post-test execution. These trace files encompass screenshots, network log files, and more, offering a granular view of the test conditions and outcomes. - Cross-platform compatibility: Designed with versatility in mind, the Handler supports both Selenium and Playwright frameworks, ensuring that teams can use its functionality regardless of the underlying technology stack. This integration not only simplifies the debugging process but also significantly boosts the efficiency of our reporting mechanisms. This allows stakeholders like engineers, managers, product owners (and hey, even us testers!) to gain deeper insights into the outcomes of automated tests. Inside the Allure Attachments Handler: a technical deep dive Here, we delve into how the Handler operates with both Selenium and Playwright, providing insights under the hood. Integration with Selenium For Selenium-based tests, the Handler employs a multi-faceted approach to capturing test artifacts: - Conservative use of WebDriver initialization: The webdriver property initializes the Selenium WebDriver instance only when first accessed, reducing unnecessary overhead for tests that don't need it. - Attaching log files and screenshots: Methods like attach_selenium_screenshot and attach_selenium_logs capture and attach various Selenium artifacts (such as screenshots, page source, cookies, console log files, and network log files) to the Allure report for detailed test analysis. - Network log file processing: The _process_selenium_network_logs method filters and formats raw performance log files from Selenium to include specific XMLHttpRequest (XHR) events, which are then attached to the Allure report. Using Selenium Webdriver and attaching useful info to Allure reports... [Read the full post...]( 👋 Level up your software testing career with MoT Professional Membership. [Available for individuals and teams]( [Website]( [LinkedIn]( [YouTube]( [Instagram]( Copyright © 2024 Ministry of Testing, All rights reserved. You have opted to join this email list. Our mailing address is: Ministry of Testing 19 New RoadBrighton, East Sussex BN1 1UF United Kingdom [Add us to your address book]( Want to change how you receive these emails? You can [update your preferences]( or [unsubscribe from this list](.