Codex Agentic Test Automation Kit

This locally installable multi-agent toolkit for test automation was developed during a previous professional role at a Belgian service organization. At 5A Technologies, we build on the resulting expertise in agentic orchestration and test automation. Users could provide test information through CSV, Excel, user stories or free-form text. The kit analyzed the request, selected an appropriate workflow and then built a test solution within the existing repository.

• Context: test automation within existing repositories
• Project: locally installable private multi-agent toolkit from a previous professional role
• Status: completed internal tool
• Role of 5A Technologies: carrying forward expertise in agentic architecture, workflow orchestration, test automation, bounded quality loops and local developer tooling
• Core principles: one shared automation contract, bounded agent responsibilities, local execution and execution evidence

Key design principle
A generated file did not automatically count as completed automation. The flow explicitly distinguished between generated, executable, successful and blocked work and allowed quality gates to determine whether execution could continue.

The solution supported several types of automation:

• UI tests;
• API tests;
• unit tests;
• combinations of multiple test layers;
• extension or repair of existing test suites;
• analysis-only when implementation was not yet responsible.

Four specialized agents worked together: an Analysis Agent, Architecture Agent, Implementation Agent and Quality Agent. A separate Codex session coordinated the run, guarded its boundaries and received the final handover.

Test automation often starts with unstructured or inconsistent input: test cases in files, user stories with acceptance criteria, separate functional descriptions, missing technical context and repositories using different frameworks.

An engineer would normally need to interpret that input first, investigate the existing architecture, choose an approach, write code, run tests and repair failures in a targeted way. When UI, API and unit tests come together, the handoff between analysis, architecture, implementation and quality control can also become difficult to trace.

The Agent Kit brought these steps together in one repeatable, local workflow. Every responsibility remained visible: the coordinator guarded the run, specialized agents delivered bounded output and quality gates determined whether the flow could continue.

The toolkit was installed locally as a private npm package. After initialization, it inspected the target project, loaded the permitted configuration and normalized the available input into one shared automation contract.

A workflow router then selected a predefined flow for:

• a new or existing UI test suite;
• API test automation;
• unit-test generation;
• combined feature automation;
• repair of failing or incomplete tests;
• analysis without implementation.

The selected workflow determined which agents were needed, the order in which they worked, which compatible adapters were reused and which quality conditions had to be met.

1. The user supplied test information through CSV, Excel, user stories or free-form text.
2. The coordinating Codex session bounded the run and loaded the permitted project configuration.
3. The input was normalized into one shared automation contract.
4. The workflow router selected the appropriate flow, agents and compatible adapters.
5. The Analysis Agent validated scenarios, acceptance criteria, repository context and missing information.
6. The Architecture Agent translated that analysis into an executable test and implementation plan.
7. The Implementation Agent created or repaired tests and ran them locally through the repository tools.
8. The Quality Agent checked the execution evidence and returned problems to Analysis, Architecture or Implementation in a targeted way.
9. After release or a controlled stop, the user received the final handover with results, evidence, blockers and remaining actions.

CSV, Excel, user stories and free-form text were first translated into a shared structure. That structure described the source, requested test layers, scenarios, conditions, steps, expected results and permitted changes.

This meant the agents did not need separate implementation logic for every input format. The contract created a traceable boundary between the user's request, the repository context and the selected execution.

The source format itself was not treated as proof of completeness. Missing information remained explicitly visible and could lead to a targeted question, an analysis-only result or a human blocker.

The toolkit did not impose one test framework on every project. It first inspected the language, repository structure, existing tests and available commands. It then selected a compatible adapter or extended existing patterns.

Depending on the repository, UI, API or unit-test adapters could be reused, for example. This did not mean that every possible framework was built in by default. The existing architecture was preferred; a new dependency or test layer was considered only when the configuration allowed it.

Feature flags and access boundaries determined which workflows were active, whether dependency changes were permitted, whether automated repair could start and when the iteration loop had to stop.

Coordinating Codex session

The coordinating Codex session was separate from the four specialized agents. It started and bounded the orchestration, guarded the selected workflow, tracked run status and iteration, and eventually received the reviewable final report.

Analysis Agent

The Analysis Agent translated the user's request into an executable assignment. The agent validated scenarios, linked acceptance criteria, investigated relevant repository context, identified missing information and determined what could and could not be automated.

Its output was a structured analysis contract for the next handoff. When the basis was insufficient, the flow could deliberately end as analysis-only.

Architecture Agent

The Architecture Agent determined how the tests should fit into the existing project. The agent selected compatible adapters and, where necessary, designed suites, fixtures, page objects, clients, mocks, test data, execution commands and concrete quality gates.

Those choices were transferred to the Implementation Agent as an executable implementation plan.

Implementation Agent

The Implementation Agent created or repaired tests and supporting code. The agent could extend existing tests, add helpers and test data, adjust configuration within its boundaries and link every change to the relevant scenarios.

The implementation was followed by an initial local execution through the repository tools.

Quality Agent

The Quality Agent did not accept a solution simply because code had been generated. The agent checked the build, types, lint, test discovery, actual execution, skips, assertions, alignment with acceptance criteria, reuse, duplication, error handling and remaining risks.

The verdict distinguished between generated, executable, successful and blocked work.

A rejection did not automatically trigger a complete restart. The Quality Agent produced a structured feedback package and sent it to the agent best placed to repair the problem:

• a misinterpreted scenario returned to Analysis;
• a structural or architectural problem returned to Architecture;
• a code or configuration error returned to Implementation;
• every new version passed through Quality again.

This kept every handoff targeted and traceable. The coordinating session tracked which version was active, why a repair loop started and which quality gate had to run again afterwards.

Bounded stop conditions

The loop stopped when all mandatory quality gates passed, when the configurable iteration maximum was reached or when an external blocker required human input, access or a product decision.

When blocked, the partial result was retained. The handover recorded which step could not responsibly continue and which action was required to proceed.

Agents worked through bounded filesystem, process, framework and test-runner adapters. Those tools could inspect the repository, change permitted files and execute the existing build, type, lint and test commands.

The results were returned to Quality as execution evidence. A generated file without successful checks therefore did not automatically count as completed automation.

The workflow remained locally executable and aligned with the repository in which the user already worked. No public rendering service or autonomous publication path was required.

An MCP memory layer using SQLite retained durable project context and semantically reusable information, such as previously investigated repository structure, architecture decisions, normalized requirements, scenario identifiers, quality findings and known commands.

Agents could retrieve relevant context without starting every iteration entirely from scratch. When source files or repository content changed, outdated context was invalidated. The cache therefore remained supportive and was not blindly treated as current truth.

A local React and Vite board could receive live events through Server-Sent Events. It displayed the selected workflow, current agent, iteration, handoffs, commands, test results, changed files, quality findings, blockers and final status.

The board did not make decisions itself. It was an observability layer that allowed users to follow why the flow continued, returned to an agent or stopped in a controlled way.

Package, configuration and input

• TypeScript, Node.js and npm for local installation and execution;
• parsers and JSON Schema for normalizing different sources;
• YAML or JSON configuration and feature flags to bound behavior for each repository.

Orchestration and repository adapters

• typed state-machine workflows for selection, handoffs, retries and stop conditions;
• Codex with separate agent responsibilities for analysis, architecture, implementation and quality;
• filesystem, process, framework and test-runner adapters for local execution.

Memory, events and visualization

• an MCP memory server with SQLite for context and semantic caching;
• React, Vite and Server-Sent Events for the local Agent Board;
• build, lint, test and traceability checks as technical and functional quality gates.

After release or a controlled stop, control returned to the coordinating Codex session. The user received a final report containing:

• completed, partial and non-automated work;
• the distinction between generated, executable, successful and blocked work;
• files created or changed;
• tests and quality checks performed;
• commands for running the suite again;
• external blockers, risks and remaining actions.

Success was not reported merely because code had been written.

At 5A Technologies, we build on experience across:

• the end-to-end agentic architecture;
• the predefined workflows and routing;
• the shared automation contract;
• the responsibilities and handoff contracts of the four agents;
• the implementation and repair cycles;
• the quality gates and bounded feedback loop;
• the local package and feature configuration;
• the MCP memory and caching strategy;
• the live visualization board;
• reporting and technical handover.

The Agent Kit brought diverse test input together in one reviewable automation process. The orchestrator selected the appropriate workflow, specialized agents delivered bounded intermediate results and the Quality Agent returned problems to analysis, architecture or implementation in a targeted way.

This project shows how an agentic test toolkit:

• divides multi-agent orchestration into separate responsibilities for analysis, architecture, implementation and quality;
• uses automation contracts and targeted handoffs to keep every transition traceable;
• bounds repair loops through quality gates, stop conditions and human blockers;
• aligns test architecture and local developer tooling with an existing repository;
• uses MCP memory and observability as support without turning them into autonomous decision layers;
• treats code generation as successful only when execution evidence and quality checks support that conclusion.

The public project description observes the following boundaries:

• the organization remains anonymized;
• internal repository, package and configuration names are not disclosed;
• prompts, credentials, test data, application names and company screenshots remain outside the public content;
• no counts, savings, adoption or success rates are claimed;
• support for every possible framework is not claimed;
• technologies and operation are described only at architecture level;
• internal implementation details are not published.