QA Engineer, AI
About Applied Business Software, Inc.,
Applied Business Software, Inc., (ABS), the maker of The Mortgage Office , is the industry leader in private lending and loan management technology. Since 1978, our software has powered thousands of lenders, including private money firms, municipalities, CDFIs, Tribal Nations, universities, franchisors, third-party servicers and other non-bank lenders, helping them streamline, automate, and scale their lending operations with confidence.
About the role
We are hiring for a QA Engineer, (AI), who will be responsible for validating both: Deterministic systems (APIs, backend logic, UI flows) and Non-deterministic AI systems (LLM outputs, prompt behavior, edge cases). We are building an AI-first product where core functionality depends on prompt engineering and LLM behavior.This position requires someone to design, execute, and automate testing strategies for LLM-based features, and AI-driven workflows. Additionally, they will test end-to-end product flows across frontend, backend, and APIs.
What you'll do- Test end-to-end product flows across frontend, backend, and APIs of an event based system
- Design structured test cases for prompt-based features
- Build and maintain test datasets
- Define evaluation criteria for AI outputs
- Run regression testing when prompts or models change Automate repeatable tests
- Track quality metrics over time (output quality, failure rate, latency issues)
- 3+ years in QA / software testing, preferably for LLM based product
- Strong Python skills (Automation, LLM, Evaluations)
- Experience testing APIs and full-stack systems
- Ability to think in edge cases and break systems intentionally
- Experience working with LLM-based products
- Experience testing prompt engineering products
- Comprehensive health, dental, and vision coverage.
- 401(k) with company funded match.
- Mission-driven company with a 45+ year legacy and industry-leading products trusted by thousands of lenders.
- Collaborative, supportive team known for long tenure, deep expertise, and a culture built on trust and integrity.
- Help modernize and evolve the leading platform in private lending technology.
- Beautiful San Mateo office with a tight-knit culture.
- Competitive compensation and room for professional growth.
- Performance-based bonuses.
- Generous PTO.
- Modern workspace with on-site amenities.
- Awesome company Christmas party.
The pay range for this role is:
120,000 - 140,000 USD per year (Hybrid, San Mateo, CA)