Job Details

POSITION SUMMARY

The AI Quality Assurance Analyst is the quality champion for Allied's emerging AI capabilities, ensuring that every model, integration, and automation works reliably and safely within our business processes. Working alongside developers, product owners, andbusiness users, the Analyst will design tests that push the limits of our applications and help deliver trustworthy AI solutions.

ESSENTIAL FUNCTIONS:

• Test AI features integrated into core systems (e.g., Dynamics CRM, QicLink claims platform, Docuvantage) to ensure they function correctly in end-to-end workflows.

• Write and automate test cases for AI outputs and traditional software features; define clear pass/fail criteria (using statistical sampling when appropriate) to validate AI results.

• Own regression testing for AI capabilities: continuously re-run critical test scenarios as models are updated or data changes, to catch issues early.

• Contribute to the overall test strategy and perform adversarial or edge-case testing (e.g., stress tests, prompt injections) to evaluate the robustness and security of AI components.

• Document and recommend solutions for issues found; coordinate with the AI Product Owner and SMEs to clarify requirements for fixes and improvements.

• Collaborate closely with developers and the Product Owner to report bugs, verify fixes, and improve requirements. Emphasize shared accountability for quality across the team.

EDUCATION

• Bachelor's degree in related field or equivalent work experience required.

EXPERIENCE AND SKILLS:

• 7+ years in software Quality Assurance/testing roles, with significant experience in enterprise systems or complex integrations required.

• Proven expertise in test automation (UI and API) and building regression test suites in CI/CD environments.

• Familiarity with AI/ML system testing principles. Comfortable validating outputs that may vary statistically and determining acceptable accuracy thresholds.

• Strong analytical skills to identify edge cases and pinpoint defects across interconnected systems (frontend, backend, data, model).

• Excellent documentation and communication skills for writing test plans, reporting issues, and guiding teams on quality best practices.

• Experience in industries like insurance or healthcare is a plus (provides domain context for Allied's processes).

• Test automation tools such as Selenium or Cypress for web interface testing, and Postman or REST-assured for API testing.

• Testing frameworks like PyTest or JUnit, integrated with CI/CD pipelines (Azure DevOps, Jenkins) for continuous testing.

• Version control (Git) for maintaining test scripts, and monitoring/logging tools (Azure Monitor, Splunk) to investigate issues.

• Performance testing or security testing tools for evaluating AI systems under load or threat scenarios.

POSITION COMPETENCIES

• Job Knowledge
• Time Management
• Accountability
• Communication
• Initiative
• Customer Focus

PHYSICAL DEMANDS:

This is a standard desk role - long periods of sitting and working on a computer are required.

WORK ENVIRONMENT

Remote

Here at Allied, we believe that great talent can thrive from anywhere. Our remote friendly culture offers flexibility and the comfort of working from home, while also ensuring you are set up for success. To support a smooth and efficient remote work experience, the internet connection must be obtained through a cable broadband or fiber optic internet service provider with speeds of at least 100Mbps download/25Mbps upload. Reliable internet service is essential for staying connected and productive.

The company has reviewed this job description to ensure that essential functions and basic duties have been included. It is not intended to be construed as an exhaustive list of all functions, responsibilities, skills, and abilities. Additional functions and requirements may be assigned by supervisors as deemed appropriate.

Compensation is not limited to base salary. Allied values our Total Rewards, and offers a competitive Benefit Package including, but not limited to, Medical, Dental, Vision, Life & Disability Insurance, Generous Paid Time Off, Tuition Reimbursement, EAP, and a Technology Stipend.

Allied reserves the right to amend, change, alter, and revise, pay ranges and benefits offerings at any time. All applicants acknowledge that by applying to the position you understand that the specific pay range is contingent upon meeting the qualification and requirements of the role, and for the successful completion of the interview selection and process. It is at the Company's discretion to determine what pay is provided to a candidate within the range associated with the role.

Protect Yourself from Hiring Scams

Important Notice About Our Hiring Process

To keep your experience safe and transparent, please note:

• All interviews are conducted via video.
• No job offer will ever be made without a video interview with Human Resources and/or the Hiring Manager.

If someone contacts you claiming to represent us and offers a position without a video interview, it is not legitimate. We never ask for payment or personal financial information during the hiring process.

For your security, please verify all job opportunities through our official careers page: Current Career Opportunities at Allied Benefit Systems

Your security matters to us-thank you for helping us maintain a fair and trustworthy process!