Skip to content

feat(workflow): workflow runner and public API (Part 5/8) - #592

Open
kalenkevich wants to merge 1 commit into
feat/workflows_part4from
feat/workflows_part5
Open

feat(workflow): workflow runner and public API (Part 5/8)#592
kalenkevich wants to merge 1 commit into
feat/workflows_part4from
feat/workflows_part5

Conversation

@kalenkevich

Copy link
Copy Markdown
Collaborator

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

Problem: With the engine core and node types in place, the workflow needs a runnable orchestrator and a public API.

Solution — Part 5 of 8 (dynamic scheduling + rehydration were merged into the runner PR because they are runner-coupled and share its tests). Stacked on Part 4.

Included:

  • workflow.ts — the Workflow orchestrator: triggers, routing/fan-out, dynamic entry, resume/fast-forward.
  • workflow_agent.tsBaseAgent adapter so a Workflow runs under the ADK Runner (streams via the shared AsyncQueue).
  • dynamic_node_scheduler.ts + utils/rehydration_utils.ts — dynamic scheduling and event-driven state reconstruction for resume.
  • node.ts — the node() user API.
  • workflow/index.ts + core/src/index.ts — public barrel; typedoc.json marks internal AsyncQueue/ScheduleDynamicNode/NodeContextOptions as intentionally-not-exported.
  • register_builtin_nodes.ts — side-effect module imported by node()/workflow so the built-in Function/Tool/Parallel builders register even when those entry points are imported directly (not via the barrel).

Adapted to Part 1's review APIs (AsyncQueue, commonPrefixOf).

Testing Plan

  • Unit tests added/updated; all pass locally.

Tests (53): workflow, workflow_advanced, routing, parallel, dynamic_workflow, dynamic_resume, resume, runner_integration, auth_gate, hitl. Full core suite 2444 green; docs:check clean; tsc clean.

Manual E2E: N/A (covered by the runner integration tests; LLM-agent-as-node tests land in Part 6).

Checklist

  • I have read CONTRIBUTING.md.
  • I have performed a self-review.
  • Commented hard-to-understand areas.
  • Added tests.
  • New and existing unit tests pass locally.
  • Manually tested end-to-end.
  • Dependent changes merged.

Additional context

Stacked split — merge in order (…Part 4 → Part 5 → Part 6 → …).

Part 5/9 of the feature/workflows split. Brings the engine together into a
runnable workflow and exposes the public surface.

- workflow.ts: the Workflow orchestrator — triggers, routing/fan-out, dynamic
  entry, and resume/fast-forward.
- workflow_agent.ts: BaseAgent adapter so a Workflow runs under the ADK Runner
  (streams events via the shared AsyncQueue).
- dynamic_node_scheduler.ts + utils/rehydration_utils.ts: dynamic scheduling
  and event-driven state reconstruction for resume.
- node.ts: the node() user API.
- workflow/index.ts + core/src/index.ts: public barrel exports; typedoc.json
  marks the internal AsyncQueue/ScheduleDynamicNode/NodeContextOptions as
  intentionally-not-exported.
- register_builtin_nodes.ts: side-effect module imported by node()/workflow so
  the built-in Function/Tool/Parallel builders are registered even when those
  entry points are imported directly (not via the barrel).

Adapted to Part 1's review APIs: workflow_agent uses AsyncQueue, workflow uses
the commonPrefixOf util function.

Tests (53): workflow, workflow_advanced, routing, parallel, dynamic_workflow,
dynamic_resume, resume, runner_integration, auth_gate, hitl. Full core suite
green (2444), docs:check clean, tsc clean.

The LLM-agent-as-node tests (node_api, multi_agent, llm_agent) land in Part 7
with the agent builder.

@AmaadMartin AmaadMartin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the public-API surface, the runner loop and the resume path against the files at head. Good things first: it reuses the shared AsyncQueue rather than adding a second queue, there is no any/@ts-expect-error anywhere in the diff, and I checked the new barrel names against common.ts/index.ts for collisions (none today). Three things I would not ship as-is: intentionallyNotExported hides three types that are genuinely on NodeContext's public surface, WorkflowConfig marks a strictly-required choice optional, and the fast-forward path hands user code an object cast to NodeContext that is missing every method. The rest are nits.

Comment thread typedoc.json
Comment on lines +13 to +17
"intentionallyNotExported": [
"AsyncQueue",
"ScheduleDynamicNode",
"NodeContextOptions"
]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a nit. These three names are silenced in the docs build rather than exported, but all three are reachable from the public surface of NodeContext, which this PR exports from @google/adk.

"intentionallyNotExported": [
  "AsyncQueue",
  "ScheduleDynamicNode",
  "NodeContextOptions"
]
// core/src/workflow/node_context.ts
readonly channel: AsyncQueue<Event>;    // :45
scheduler?: ScheduleDynamicNode;        // :77
constructor(opts: NodeContextOptions)   // :85

AsyncQueue is not exported from core/src/index.ts or core/src/common.ts today, so someone writing a custom node cannot name the type of ctx.channel, and cannot construct a NodeContext in their own test without re-declaring the options shape by hand. intentionallyNotExported makes docs:check green without fixing what it was reporting.

Either export them (AsyncQueue via common.ts, NodeContextOptions/ScheduleDynamicNode via the workflow barrel) and drop this block, or, if scheduler really is internal wiring, make it non-public instead of hiding it from the docs. This is the one thing in the PR that is genuinely hard to change after release.

Comment thread core/src/index.ts
export * from './tools/mcp/mcp_session_manager.js';
export * from './tools/mcp/mcp_tool.js';
export * from './tools/mcp/mcp_toolset.js';
export * from './workflow/index.js';

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. Star re-export of a ~45-symbol barrel, where the rest of this file is deliberate named exports.

export * from './workflow/index.js';

Only ./common.js and the three tools/mcp/* lines use export *; everything else is export {X} / export type {Y}. With a star, the top-level @google/adk surface changes silently whenever workflow/index.ts changes, and if a name ever collides with an existing export ESM drops it from both sides with no error. I checked the current names (Node, Graph, Edge, START, RetryConfig, ErrorClass, RequestInput, NodeState, ...) against common.ts and index.ts — no collision today, so this is only about keeping the surface intentional.

Separately: the workflow barrel is added here but not to core/src/common.ts, which is what core/src/index_web.ts re-exports — so the workflow API is absent from the web entry point. Intentional? Nothing in workflow/ looks node-only.

// --- Nodes ---
export {BaseNode, START} from './base_node.js';
export type {BaseNodeConfig} from './base_node.js';
export {Node, node} from './node.js';

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a nit. Public naming, and this is the PR that fixes it in place.

export {Node, node} from './node.js';

Node and node both land in the flat @google/adk namespace differing only in case, and Node shadows the DOM / @types/node global — import {Node} from '@google/adk' in a file that also touches DOM types is a quiet footgun, and Node/node next to each other in an import list is easy to mistype. The same, more mildly, for Edge, Graph, START and DEFAULT_ROUTE (lines 21-22, 40): generic words in a root barrel.

Python gets namespacing for free (workflow.Node via the module); a flat root export does not, so parity does not force the name here. WorkflowNode for the subclassing base (keeping node() as the factory) would read consistently with WorkflowAgent/WorkflowConfig in this same file.

Comment on lines +41 to +54
export interface WorkflowConfig extends BaseNodeConfig {
/** Edge definitions used to build the workflow graph. */
edges?: EdgeItem[];
/**
* An imperative entry function driving execution via `ctx.runNode(...)`.
* Mutually exclusive with {@link edges}.
*/
dynamicEntry?: DynamicEntry;
/**
* Maximum number of graph-scheduled nodes running in parallel. `undefined`
* means unlimited. Does not throttle dynamic (`ctx.runNode`) children.
*/
maxConcurrency?: number;
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a nit. Exactly one of edges / dynamicEntry is required, but the type marks both optional.

export interface WorkflowConfig extends BaseNodeConfig {
  edges?: EdgeItem[];
  dynamicEntry?: DynamicEntry;
  maxConcurrency?: number;
}

So new Workflow({name: 'x'}) typechecks and then throws at :101-105, and passing both typechecks and throws at :96-100. The repo guideline is to reflect strictly-required options in the type; a union does it at compile time:

export type WorkflowConfig = BaseNodeConfig & {maxConcurrency?: number} & (
  | {edges: EdgeItem[]; dynamicEntry?: never}
  | {dynamicEntry: DynamicEntry; edges?: never}
);

That also retires the config.edges! non-null assertion at :109. Keep the runtime throws for JS callers. Caveat I could not fully check: this turns an interface into a type alias, so if Parts 6-8 do interface X extends WorkflowConfig they would need type X = WorkflowConfig & {...} — I only looked at this part's diff.

Comment on lines +158 to +167
export function makeFastForwardContext(
parent: NodeContext,
prior: RehydratedNode,
): NodeContext {
return {
output: prior.output,
route: prior.route,
branch: prior.branch ?? parent.branch,
interruptIds: [],
} as unknown as NodeContext;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a nit. A four-field object is cast to a full NodeContext and then handed to user code on the resume path.

return {
  output: prior.output,
  route: prior.route,
  branch: prior.branch ?? parent.branch,
  interruptIds: [],
} as unknown as NodeContext;

Inside the workflow loop this is safe — handleCompletion only reads those four fields. But DynamicNodeScheduler.schedule returns it directly (dynamic_node_scheduler.ts:73), and NodeContext.runNode returns the scheduler's result verbatim (node_context.ts:140-148). So in a dynamicEntry workflow that is resumed, this is what the author's own code receives:

const r = await ctx.runNode(child, input);
r.emit(evt);      // TypeError: r.emit is not a function
r.state;          // undefined

The double cast is exactly what stops the compiler from reporting it. Same cast at workflow.ts:337 for the static-graph resume path. Suggest a real shared result type that both paths return honestly:

export interface NodeResult {
  output: unknown;
  route?: RouteValue | RouteValue[];
  branch?: string;
  interruptIds: string[];
}

with runNode() typed Promise<NodeContext | NodeResult> — or build a genuine child NodeContext for the fast-forward case so the public contract still holds. Two sites either way.

Comment on lines +86 to +89
for await (const event of channel) {
yield event;
}
await settle;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a nit, optional. Nothing cleans up if the consumer stops reading.

for await (const event of channel) {
  yield event;
}
await settle;

If the caller breaks out of its for await (or the Runner cancels the invocation), this generator's return() runs, the loop exits and await settle never executes — the workflow keeps going, nodes keep making model/tool calls, and they keep pushing into an AsyncQueue nobody drains. A try { ... } finally { channel.close(); await settle; } gives it one exit path.

Related, and the reason I'd fix them together: workflow.ts:534-539 awaits in-flight siblings with Promise.allSettled after a node fails but never cancels them, so a workflow that is already throwing still pays for every sibling to run to completion. node_runner.ts:185-193 already builds each child's abortSignal from an AbortController chained to invocationContext.abortSignal, so a workflow-level controller aborted in cleanupPending (and in the finally above) would be picked up by cooperative nodes for free.

Comment on lines +272 to +273
private atConcurrencyLimit(loop: LoopState): boolean {
return !!this.maxConcurrency && loop.pending.size >= this.maxConcurrency;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. maxConcurrency: 0 silently means unlimited.

private atConcurrencyLimit(loop: LoopState): boolean {
  return !!this.maxConcurrency && loop.pending.size >= this.maxConcurrency;
}

0 is falsy, so the one value a caller might reach for to mean "stop scheduling" turns the cap off entirely — the opposite of the intent. Also -1 / 1.5 are accepted.

return this.maxConcurrency !== undefined && loop.pending.size >= this.maxConcurrency;

plus a constructor check rejecting anything < 1. The doc comment at :50-52 is honest that dynamic ctx.runNode() children are not throttled — but that does mean a dynamicEntry workflow has no cap at all on in-flight children. Is bounding those planned for a later part?

* Replay/checkpointing, dynamic scheduling, and task/chat isolation scopes are
* added in later phases; hook points are marked with TODO(phase-N).
*/
export class Workflow extends BaseNode {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a nit. This PR turns the whole workflow subsystem into public @google/adk API, and none of the new entry points are marked experimental.

export class Workflow extends BaseNode {

The repo has the decorator for exactly this case (core/src/utils/experimental.ts), used at core/src/tools/openapi_tool/openapi_toolset.ts:17 and core/src/skills/gcp_skill_registry.ts:22:

@experimental
export class Workflow extends BaseNode {

Same for WorkflowAgent (workflow_agent.ts:32) and Node (node.ts:44). Parts 6-8 still change this surface (the barrel itself says LLM nodes land in Part 7), so marking it now is what buys the freedom to change it without a semver break — and it is much easier to add before the first release than to explain afterwards.

}
}

// eslint-disable-next-line require-yield

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. Bare suppression, no reason on the directive line.

// eslint-disable-next-line require-yield

Both uses here are legitimate — the AsyncGenerator signature is mandated by the base class and there is genuinely nothing to yield — but the repo standard is to name the cause inline so the next reader does not have to work it out:

// eslint-disable-next-line require-yield -- child events stream via ctx.channel; this generator yields nothing

Also at workflow_agent.ts:92 (runLiveImpl throws).

Comment on lines +27 to +34
} as unknown as Session;
return new InvocationContext({
invocationId: 'inv-1',
session,
agent: {
name: 'wf',
runAsync: async function* () {},
} as unknown as BaseAgent,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. The same fixture, with the same two double-casts, is copy-pasted across five test files.

  } as unknown as Session;
  ...
      } as unknown as BaseAgent,

createIc() + driveWorkflow() appear verbatim in workflow_test.ts:22-34, parallel_test.ts:28-40, hitl_test.ts:26-38, dynamic_workflow_test.ts:21-33 and auth_gate_test.ts:147-159. Two things:

  • createSession() (core/src/sessions/session.ts:63) returns a real Session, so the first cast can just go — that is the factory the rest of core/test uses.
  • For the agent, a three-line local class extending BaseAgent removes the second cast and typechecks for real.

Then lift the pair into core/test/workflow/testing_utils.ts and import it. That is ~60 duplicated lines gone, and Parts 6-8 edit one fixture instead of six. Nothing else in the diff reaches for any or @ts-expect-error, which is why these stood out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants