How to run automated regression tests on generative AI applications

As LLM-powered products and features make their way into mainstream usage, testing them becomes more and more important. ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ [View this email in your browser](~/AAAHUQA~/RgRnrhs-P0TbaHR0cHM6Ly9iZXJyaW1hLmVvY2FtcGFpZ24xLmNvbS93ZWItdmVyc2lvbj9lcD0yJmxjPTg3MDdlZTg2LTUyZjAtMTFlZS1hZjUyLTRiNTM4MmJhM2I5NCZwPTYyZWNjNTJlLWM3NWEtMTFlZS1iYjBkLTYzYjgzYTgxYTlmNCZwdD1jYW1wYWlnbiZ0PTE3MDc4NDEwODYmcz1iMTAyMDQwMzA0ZDkwNjhlZWVkM2I4OTYwODBlMTNkZDRkMGVjYzEyNmFjMTViYWU1MDRhNGVhYzhlYjRiNjQ2VwVzcGNldUIKZcM-lstlTPMeblIbdHJpc3RyYW1iYWxkd2luOTFAZ21haWwuY29tWAQAAGQd) [QA Wolf logo](~/AAAHUQA~/RgRnrhs-P0UgNWU4M2I3ZjU3NzY4MTBjMTZiNTcwZjQ5YmIwMzdhN2JEgmh0dHBzOi8vc3J2LmJ1eXNlbGxhZHMuY29tL2Fkcy9sb25nL3gvVEM2V1FMSUhUVFRUVFRGU0pTM0NQVFRUVFRUVkxRQ0RLV1RUVFRUVEI0REtXN1ZUVFRUVFRFSk1LNkFVVDVRWVBLN000T0xZVjdSSFRBSVVaUUJGNkxCWkNSWVRXBXNwY2V1Qgplwz6Wy2VM8x5uUht0cmlzdHJhbWJhbGR3aW45MUBnbWFpbC5jb21YBAAAZB0~) How to run automated regression tests on generativeAI applications As LLM-powered products and features make their way into mainstream usage, testing them and their underlying models becomes more and more important. But how do you create reliable, deterministic tests for inherently random outputs? In our post “[6 things to know about automated testing of genAI apps](~/AAAHUQA~/RgRnrhs-P0UgNWU4M2I3ZjU3NzY4MTBjMTZiNTcwZjQ5YmIwMzdhN2JEgmh0dHBzOi8vc3J2LmJ1eXNlbGxhZHMuY29tL2Fkcy9sb25nL3gvVEM2V1FMSUhUVFRUVFRGU0pTM0NQVFRUVFRUVkxRQ0RLV1RUVFRUVEI0REtXN1ZUVFRUVFRFSk1LNkFVVDVRWVBLN000T0xZVjdSSFRBSVVaUUJGNkxCWkNSWVRXBXNwY2V1Qgplwz6Wy2VM8x5uUht0cmlzdHJhbWJhbGR3aW45MUBnbWFpbC5jb21YBAAAZB0~)” we share the techniques we’ve been using for our clients, and what we’ve learned from building and maintaining thousands of black box regression tests for genAI applications. [Learn how to automate black-box testing of your generative AI](~/AAAHUQA~/RgRnrhs-P0UgNWU4M2I3ZjU3NzY4MTBjMTZiNTcwZjQ5YmIwMzdhN2JEgmh0dHBzOi8vc3J2LmJ1eXNlbGxhZHMuY29tL2Fkcy9sb25nL3gvVEM2V1FMSUhUVFRUVFRGU0pTM0NQVFRUVFRUVkxRQ0RLV1RUVFRUVEI0REtXN1ZUVFRUVFRFSk1LNkFVVDVRWVBLN000T0xZVjdSSFRBSVVaUUJGNkxCWkNSWVRXBXNwY2V1Qgplwz6Wy2VM8x5uUht0cmlzdHJhbWJhbGR3aW45MUBnbWFpbC5jb21YBAAAZB0~) You’ll learn about - Creating assertions for probabilistic outcomes - When to employ different testing techniques - Cost controls and token management [Read the post](~/AAAHUQA~/RgRnrhs-P0UgNWU4M2I3ZjU3NzY4MTBjMTZiNTcwZjQ5YmIwMzdhN2JEgmh0dHBzOi8vc3J2LmJ1eXNlbGxhZHMuY29tL2Fkcy9sb25nL3gvVEM2V1FMSUhUVFRUVFRGU0pTM0NQVFRUVFRUVkxRQ0RLV1RUVFRUVEI0REtXN1ZUVFRUVFRFSk1LNkFVVDVRWVBLN000T0xZVjdSSFRBSVVaUUJGNkxCWkNSWVRXBXNwY2V1Qgplwz6Wy2VM8x5uUht0cmlzdHJhbWJhbGR3aW45MUBnbWFpbC5jb21YBAAAZB0~) About QA Wolf QA Wolf provides QA as a Service for more than 150 companies: Test creation, execution, and maintenance 24 hours a day. With QA Wolf, your team can get to 80% automated test coverage in less than 4 months. We build and maintain your automated test suite, plus provide unlimited parallel test runs. Visit our [website](~/AAAHUQA~/RgRnrhs-P0UgNGQ2OWE1NWI0M2ZlNDRmNjE2NDkwYjVlM2RkODIwMTZEI2h0dHBzOi8vd3d3LnFhd29sZi5jb20vY2FzZS1zdHVkaWVzVwVzcGNldUIKZcM-lstlTPMeblIbdHJpc3RyYW1iYWxkd2luOTFAZ21haWwuY29tWAQAAGQd) to learn more. You received this email because you subscribed to our list. You can [unsubscribe](~/AAAHUQA~/RgRnrhs-P4QWAWh0dHBzOi8vYmVycmltYS5lb2NhbXBhaWduMS5jb20vdW5zdWJzY3JpYmU_ZXA9MiZsPWNhYTY1ODU4LTRhYzYtMTFlZS04MDE2LTRkY2U5YmEwOGU2ZCZsYz04NzA3ZWU4Ni01MmYwLTExZWUtYWY1Mi00YjUzODJiYTNiOTQmcD02MmVjYzUyZS1jNzVhLTExZWUtYmIwZC02M2I4M2E4MWE5ZjQmcHQ9Y2FtcGFpZ24mcHY9NCZzcGE9MTcwNzgzODIyOCZ0PTE3MDc4NDEwODYmcz0wZTZjOTdiODRiNzBiOTc1OGI2YjE0NzVhOTBkY2E4NmY0ZmE3OWQ3ZTE3ZjFlNzBjNmRmNmM3NDNmNzIyYTFkVwVzcGNldUIKZcM-lstlTPMeblIbdHJpc3RyYW1iYWxkd2luOTFAZ21haWwuY29tWAQAAGQd) at any time. 71 Balmain Street Cremorne Vic 3121 Australia [Powered by EmailOctopus](~/AAAHUQA~/RgRnrhs-P0TSaHR0cHM6Ly9iZXJyaW1hLmVvY2FtcGFpZ24xLmNvbS9yZXdhcmRzP2xjPTg3MDdlZTg2LTUyZjAtMTFlZS1hZjUyLTRiNTM4MmJhM2I5NCZwPTYyZWNjNTJlLWM3NWEtMTFlZS1iYjBkLTYzYjgzYTgxYTlmNCZwdD1jYW1wYWlnbiZ0PTE3MDc4NDEwODYmcz1iM2VmNWY1ZWE0NTVkMzk2N2RkOWRiZWM2ZDBkYWU4Zjc4ODQwY2YwYjM4ZTY4OGYwYjgxOWJhOTYxY2U2ZGFhVwVzcGNldUIKZcM-lstlTPMeblIbdHJpc3RyYW1iYWxkd2luOTFAZ21haWwuY29tWAQAAGQd)