AWS Bedrock AgentCore

• Session isolation — each session runs in a separate microVM, providing strong security isolation between concurrent rollouts
• Auto-scaling — new runtime sessions spin up instantly on demand, enabling massive parallel rollouts without contending for local CPU resources
• Sandboxed execution — each session runs in a secure microVM with resource controls, so agents can safely execute tools (code, shell commands, API calls). Sessions can run for up to 8 hours
• Decoupled dependencies — your agent runs in its own container with its own dependencies, completely separate from the training library

​

Architecture

• AWS Bedrock AgentCore Runtime (ACR) — Serverless runtime for deploying agents with auto-scaling and session isolation. Hosted agents call the standard OpenAI chat completions API.
• rllm-model-gateway — HTTP proxy that requests and intercepts training-related data such as token IDs and logprobs from inference servers, and groups them under corresponding sessions.

​

Prerequisites

• rLLM installed with a training backend
• AWS account with ACR access, an ECR repository, and an S3 bucket
• AWS credentials configured (aws configure with permissions for Bedrock AgentCore, ECR, and S3)

​

Setup

uv pip install -e ".[agentcore]"

from reward import GSM8KReward
from strands import Agent
from strands.models.openai import OpenAIModel
from strands_tools import calculator

from agentcore_rl_toolkit import AgentCoreRLApp

app = AgentCoreRLApp()

system_prompt = (
    "Your task is to solve the math problem. "
    + "Use the calculator tool to compute all mathematical expressions. "
    + 'Let\'s think step by step and output the final answer after "####".'
)

reward_fn = GSM8KReward()

@app.rollout_entrypoint
def invoke_agent(payload: dict):
    base_url = payload["_rollout"]["base_url"]
    model_id = payload["_rollout"]["model_id"]
    params = payload["_rollout"].get("sampling_params", {})

    model = OpenAIModel(
        client_args={"api_key": "EMPTY", "base_url": base_url},
        model_id=model_id,
        params=params,
    )

    agent = Agent(
        model=model,
        tools=[calculator],
        system_prompt=system_prompt,
    )

    user_input = payload.get("prompt")
    answer = payload.get("answer")

    response = agent(user_input)

    rewards = reward_fn(
        response_text=response.message["content"][0]["text"],
        ground_truth=answer,
    )

    return {"rewards": rewards}

if __name__ == "__main__":
    app.run()

uv run python -m examples.agentcore_math.prepare_gsm8k_data

AGENTCORE_AGENT_ARN=arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/your-agent
AGENTCORE_S3_BUCKET=your-s3-bucket

bash examples/agentcore_math/train_agentcore_math_tinker.sh

bash examples/agentcore_math/train_agentcore_math_verl.sh

​

What happens during training

1. rLLM loads a batch of prompts from the dataset and submits them to ACR, each as a separate agent session
2. ACR auto-scales containers. Each agent runs rl_app.py, calling the model via base_url (routed through rllm-model-gateway)
3. The gateway captures token IDs, logprobs, routing replays, etc. from inference server responses
4. Each agent computes a reward and returns {"rewards": ...}. The @rollout_entrypoint decorator saves results to S3
5. rLLM collects rewards from S3 and combines them with token data from the gateway to compute advantages and update the policy

​

Troubleshooting