Best practices for vetting AI vendors

With artificial intelligence (AI) in the spotlight, more vendors are branding themselves as “AI-powered”—whether or not that label holds up under scrutiny. Dawn Anderson, longtime MSP leader and current consultant for The Hardenbergh Group, encourages MSPs to push for clarity when evaluating such solutions.

“Ask for a real-world demo. Don’t just take their word for it,” Anderson advises. Ask to see it in action—how the tool works, how it sounds, what it actually does.

Look for integration with your existing systems. AI tools shouldn’t require reinventing your workflow. Instead, ask vendors how their technology fits into your existing platforms and processes.

Demand auditability. Any AI tool used in credentialing or payer enrollment must track actions, maintain records, and support compliance needs. 

Prioritize tools that reduce—not increase—your workload. If an “AI-powered” solution still relies heavily on manual inputs or creates new layers of oversight, it may not be mature enough to deliver real benefits.

Anderson also cautions MSPs to look out for vague marketing language. Ask vendors directly: What exactly is the AI doing? Is it pulling data? Making phone calls? Generating draft document language? And is it acting independently or just streamlining manual steps?

She urges credentialing leaders not to be dazzled by buzzwords. You’ll hear vendors throw around terms like “machine learning” or “natural language processing,” she says. But if they can’t show how their product saves time or reduces errors, be skeptical.

Another key factor? Audit trails and documentation. For compliance and legal protection, anything AI touches has to be traceable. That includes call recordings, follow-up logs, and even the prompts used to generate content.

Finally, she reminds MSPs to advocate for user control. You should be able to override, edit, or guide the AI, she says. It’s a partner—not a black box.

Editor’s note: This article was excerpted from our Credentialing Resource Center Journal newsletter.