Agentic testing: how we test ISMS software in a simulated company
In agentic testing, an AI receives an assignment in everyday language and no predefined path, and it operates the software itself, the way a person would. Conventional tests check whether software does what was specified in advance. An agentic test also asks whether someone who only knows their goal gets there with the software. In our view, this matters most when many roles work with one piece of software and phrase their assignments in their own words. OdySecure is meant to guide many roles through the introduction of an information security management system (ISMS). That is why we also test OdySecure agentically, in a simulated company in which agents act on assignments.
What conventional tests do well
A unit test checks a single function with defined inputs, for example whether a calculation from two values returns the expected result. A scripted UI test clicks through a defined path in the application. In manual acceptance testing, people check selected processes before a release.
All three test against a predefined expectation. That makes them repeatable and unambiguous. Our backend and frontend tests run in CI before a change is accepted into OdySecure.
An ISMS, however, takes shape in processes across several roles and approvals, and in decisions people make based on what the software shows them. Our simulation combines both. A chain works through the introduction of an ISMS with people in different roles, and a persona finds its own way from an assignment in everyday language. This also makes defects visible that sit between the functions.
A company that does not exist
Kessler & Rauh Armaturen GmbH makes valves and fittings for water utilities and the chemical industry. It has 340 employees, its headquarters and a plant in Siegen, a second plant in Olpe and a sales office in Linz. It works with 15 suppliers. Names, addresses and suppliers are invented.
24 people with their own roles act in the scenario, from management and information security through plant management, purchasing and data protection to the contacts at a supplier. Each person signs in with their own account and works in a separate browser session. For four-eyes approvals, for example, information security requests, its deputy reviews and management approves. After each step the test checks the state from the other person's point of view.
25 stations, 253 steps
The chain follows the introduction of an ISMS. It starts with the assignment from management and the setup of tenant, modules, users and roles. The team then imports existing documents, defines context and scope, assesses business processes, assets and protection needs, and writes the policy.
Next come risk methodology and risk appetite, assessing and treating risks, topic policies, controls and the statement of applicability. NIS-2 under the German BSIG follows with an assessment for each plant. After that the chain covers evidence, a supplier assessment with a questionnaire, business continuity, a security incident with vulnerabilities and an awareness training. It ends with the treatment plan.
That makes 25 stations, some with sub-stations, and 253 steps in total. A full run takes about 21 to 26 minutes.
The image above shows the company's floor plan with offices and production halls, as the stage of a recorded run from 2 November displays it. Each figure stands for a person. The speech bubble shows the step in progress, and a green tick shows that the person has finished. The text view remains the authoritative record of the result.
Every step has a fixed check rule
The steps run as automated tests with Playwright, a tool for browser testing. Each step ends with one of four results: passed, failed, not measurable or not performed. The test checks what the interface displays and what is stored in the data.
A fixed rule decides the result. No language model judges whether a step succeeded, so every result stays verifiable. An AI writes the tests, and a second, independent model reviews every new version.
An AI persona finds its own way
An AI persona from information security works through assignments in everyday language on its own, for example this one: “We now have to comply with NIS-2 as well. Add it in OdySecure and see what we are still missing.” A second assignment begins with: “A customer requires TISAX from us.”
The persona gets no path to follow. It finds one through menus, pages and labels and uses OdySecure in the browser the way a person would. The run therefore checks whether the interface leads someone to the goal who knows only their assignment. At the end the persona reports to management what it has added and which requirements are still open.
Each assignment runs in several counted passes, and the starting state is restored after every pass. In the runs on 3 November the persona completed both assignments in every pass, using between 44 and 85 steps. We run the language model behind the persona ourselves, and its requests go through OdyControl, our control layer for enterprise AI.
What happens between the functions
One example: on 31 October the assignment “comply with NIS-2” activated the EU directive as the framework. OdySecure activated exactly the framework the persona had selected. For a company in Germany it was still the wrong one, because NIS-2 applies here through the BSIG, which transposes the directive into German law.
In our view, a test that checks only whether the selected framework was activated would have passed. The defect became visible in the persona run, because the selected framework did not fit the assignment.
When the simulation finds a defect, we fix it in the product code, never only in the test setup. In this case the names and descriptions of the frameworks now show whether they refer to the EU directive or to the German law. The directive also points to the BSIG. We measured again on the test environment on 3 November, and the same assignment activated the BSIG. A later run checks in each case whether a defect is gone.
What each type of test answers
A unit test answers whether a function calculates correctly. A scripted UI test answers whether a known path works. The simulation answers whether a company with its roles, and with assignments in its own words, reaches the right result. For ISMS software, we see the third question as the one that decides its value in everyday work.
The simulation runs in our own test tenants on our test environment, never in production and never in customer tenants.
The page on the Cybervize platform shows how the platform is built. The page on Agentic ISMS describes how agents work within the ISMS.