Streamline AI Agent Tasks: Playbook Vs. Direct Artifact Delegation

When you're working with AI agents, especially in complex creative processes like story generation, one of the biggest hurdles is ensuring they understand exactly what they need to produce. It's a bit like giving someone a recipe: you wouldn't just say "bake a cake"; you'd specify the type of cake, the ingredients, and the steps. Currently, our AI models often struggle because they don't have this clarity upfront. They dive into creative work, like designing a plot, only to hit a wall when they try to save their output as a specific artifact. This leads to a frustrating cycle of guessing the correct structure, failing validation, and then iterating poorly. The root cause? Agents don't know upfront what artifacts they must produce and what those artifacts look like. The delegation process, as it stands, is a little too loose. Tasks like "design the story structure" leave too much room for interpretation, forcing the agent to guess, which, as we've seen, doesn't lead to valid results.

To tackle this head-on, we propose making the delegate function much stricter by introducing two explicit modes: Playbook Mode and Direct Artifact Mode. This approach aims to provide agents with the precise guidance they need, significantly reducing errors and improving the quality of the artifacts they produce. By clearly defining the scope and expected output of each task, we can empower agents to work more efficiently and effectively, moving us closer to seamless AI-assisted creation.

Playbook Mode: Guided Creative Journeys

Playbook Mode is designed for scenarios where a task is part of a larger, predefined workflow. Think of it as following a detailed set of instructions for a specific phase of a project. When an agent receives a delegation in Playbook Mode, it looks something like this:

{
  "playbook": "story_spark",
  "picking_up_from": "topology_design",
  "context": {...}
}

This tells the agent, "We're executing the story_spark playbook, and you're specifically picking up from the topology_design phase." To understand what's expected, the agent can then call a new tool, consult_playbook(phase). The response from this tool provides a comprehensive overview of the current phase within the playbook. This includes the overall workflow, the specific steps involved, and the criteria for completion. Crucially, it also provides a list of artifact types that this phase is expected to produce. The instructions within the playbook response will then guide the agent to "Call consult_schema for each artifact type before creating it." This ensures that the agent understands the required structure before any creative work begins, preventing the common issue of generating invalid artifacts. This mode ensures consistency and adherence to established processes, making complex workflows manageable for AI agents.

Direct Artifact Mode: Focused and Specific Tasks

On the other hand, Direct Artifact Mode is for tasks that are more focused and target specific outputs. This mode is ideal when an agent needs to create or modify a particular artifact without necessarily following a multi-step playbook. A delegation in this mode would look like this:

{
  "artifact_type": "topology",
  "task": "create initial structure for mystery story"
}

Or, if the task involves updating existing artifacts:

{
  "artifact_ids": ["passage-001", "passage-002"],
  "task": "rewrite to increase tension"
}

In Direct Artifact Mode, the delegation explicitly names the artifact_type or provides artifact_ids for updates. Furthermore, the relevant schema for that artifact type is injected directly into the agent's context. This provides the agent with immediate access to the structural requirements for the artifact it needs to create or modify. This mode offers a simple, bounded scope, ensuring the agent knows precisely what structure to adhere to and what task to perform. It cuts out ambiguity, allowing for quick and accurate generation or modification of individual components. This is particularly useful for smaller, well-defined tasks or for making specific adjustments to existing content.

Rejection of Vague Tasks: Embracing Clarity

A core principle of this stricter delegation model is the rejection of vague tasks. Delegations that don't clearly fall into either Playbook Mode or Direct Artifact Mode should be rejected by the system. For instance, a task like "Design the story structure" is too ambiguous. Under this new model, such a delegation would be flagged. If the intention is to follow a specific workflow, it should be explicitly stated as a playbook execution, like "Execute story_spark playbook, starting from topology_design phase." If the task is too complex to fit neatly into the Direct Artifact Mode but doesn't align with an existing playbook, the recommended course of action is to create a new playbook. This encourages a more structured approach to defining and managing workflows, ensuring that all tasks have a clear purpose and expected outcome. By enforcing this clarity, we prevent the guesswork that currently plagues AI agent operations, leading to more reliable and predictable results.

Separation of Concerns: Distinct Roles for Tools

To maintain clarity and prevent confusion, we are advocating for a strict separation of concerns between playbook information and schema information. This means that different tools will be responsible for providing different types of information:

This separation offers several significant benefits. Firstly, it provides a clear mental model for the agent. When the agent's context is summarized, it's easy to distinguish between information related to the workflow and information related to the structure of an artifact. Secondly, it eliminates redundancy. Instead of schemas being embedded within playbook responses (which could be repeated across multiple phases), the schema is fetched independently via consult_schema only when needed. This keeps the playbook concise and focused on the workflow. Thirdly, the agent clearly knows what each tool provides. By calling both consult_playbook and consult_schema, the agent understands it's getting workflow details from one and structural definitions from the other. While this does introduce an extra tool call per artifact type when using playbooks, we consider this an acceptable trade-off for the gains in clarity and reduced errors.

Context Summarization Consideration: Maintaining Focus

As AI agents process information, their context windows can fill up, requiring summarization. This is where the separation of concerns becomes particularly vital. When tool outputs are summarized, we need to ensure the agent can still easily understand what information pertains to what. For example, if consult_schema('topology') output is summarized, it should be clearly represented as something like: "you requested the topology schema; request again if needed." Similarly, a summarized output from consult_playbook('topology_design') should indicate: "you requested details for the topology_design phase; request again if needed." If we were to mix schemas directly into playbook responses, it could become confusing during summarization. An agent might struggle to differentiate between the workflow steps and the artifact structure definitions, especially if the same schema is relevant to multiple phases. By keeping these separate, we ensure that during context summarization, the agent can maintain a clear understanding of what information is available and how to access it again if necessary. This structured approach prevents information overload and maintains the agent's ability to effectively retrieve and utilize critical data for its tasks.

Open Questions: Refining the Model

While the proposed model offers a significant improvement, there are a few open questions we need to address to fully refine it:

1. Should consult_playbook also return schemas for artifact types it lists, or strictly defer to consult_schema? Our current proposal is strict deferral, but we're considering if bundling could offer convenience, albeit at the cost of redundancy and potential confusion during summarization.
2. How should runtime validate delegation format? We need to define the process for validating that incoming delegations adhere to the strict Playbook or Direct Artifact Mode formats. Should it result in a specific error message, or should the system attempt auto-correction where possible?
3. Does this require changes to how Showrunner is prompted about delegation? The Showrunner is a key component in how tasks are delegated. We need to ensure its prompting strategy aligns with and effectively utilizes these new, stricter delegation modes.

Addressing these questions will help us implement a robust and user-friendly system for AI agent delegation.

Related Discussions

This RFC is directly related to resolving issues where AI agents fail to produce valid artifacts. You can find more context and related problems in:

• #331 - save_artifact validation feedback: This issue highlights the symptom of the root cause we are addressing here – agents producing invalid artifacts.
• Plotwright failing to create valid topology artifacts: This problem was discovered during the investigation that led to this proposal, underscoring the need for clearer delegation and structure.

By implementing these stricter delegation models, we aim to build more reliable and predictable AI systems. For further insights into improving AI agent workflows and artifact generation, you might find the resources at OpenAI helpful.