However, Gogia raised a deeper philosophical question regarding the centralization of this process. Because SAP provides the training content, the AI that simulates the customer, and the algorithm that grades the interaction, the company has become the sole arbiter of what constitutes "competence." This "closed-loop" system, Gogia suggested, is something that the industry will need to watch closely as it matures.
Questions of Governance and Transparency
Despite the general praise for the move toward practical assessment, several experts raised concerns about the governance and transparency of an AI-led grading system. Robert Kramer cautioned that while a roleplay is a step forward, it is not a perfect proxy for consulting ability.
"A successful roleplay doesn’t necessarily make someone a capable consultant," Kramer said. He noted that real-world ERP implementations are rarely "textbook" exercises. They are often complicated by decades of legacy processes, poor data quality, and competing business priorities. In many cases, the best advice a consultant can give is to delay an implementation until a customer is truly ready—a nuance that an AI might not be programmed to appreciate.
The "black box" nature of AI grading is another point of contention. Frank Dickson raised concerns about how the AI "umpire" is calibrated. "Every roleplay is graded by AI, so the strike zone has to be the same for every batter," Dickson said. He noted that SAP has yet to provide clear details on how scoring is audited or how a professional might appeal a grade they believe is unfair.
The financial aspect also adds a layer of complexity to the transparency issue. Because staying certified requires a paid subscription to the SAP Learning Hub, SAP is effectively selling the training and then acting as the sole grader of the result. Dickson argued that this makes transparency regarding scoring more important, not less.
Furthermore, there is the question of what tools are permitted during the assessment. Previous SAP exams allowed candidates to use AI assistants. However, SAP has not explicitly stated whether a professional can use an AI tool to help them navigate the AI roleplay. "If AI assistance is allowed, the roleplay partly measures how well someone works with an assistant," Dickson noted, adding that while this might be a valid skill to test, SAP needs to be clear about the rules of engagement.
Looking Ahead: Closing the Adoption Gap
The ultimate success of the AI-powered Stay Certified program will likely be measured by the quality of the consultants it produces. Robert Kramer suggested that SAP should provide metrics to prove that this new method actually leads to better professional outcomes, rather than just a more streamlined administrative process.
"If it does, this could help close the ERP adoption gap," Kramer said. The "adoption gap" refers to the distance between the theoretical capabilities of a software suite and the actual value a customer realizes from it. By ensuring that consultants are trained to handle the human and business elements of a project through roleplays, SAP may be able to ensure that its software is implemented more effectively across the globe.
As the program rolls out to its massive user base, the eyes of the IT industry will be on SAP to see if this "simulator" model becomes the new standard for professional certification in the age of artificial intelligence.
