AmineDiro

# Copyright 2020-2026 The HuggingFace Team. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""TRL AsyncGRPO: proposed API and all examples in one file.

DESIGN SKETCH, not runnable against today's TRL. Hook names/imports require the
proposed implementation. There are no training side effects on import.
After implementation: build_plain(model_id, dataset, reward_funcs).train(), etc.
Supply a conversational dataset, importable reward functions, and a vllm server.
The harness example has one explicitly marked application adapter to implement.

AmineDiro

"""PROPOSED API: ordinary reward shaping plus explicit post-scoring filtering.

Use build_trainer(model_id, dataset, [reward_correct]).train().
The length penalty is a single-turn example. Keeping every zero-advantage group
is a demonstration of filter control, not a recommendation for efficient GRPO.
"""

from trl import AsyncGRPOConfig, AsyncGRPOTrainer
from trl.trainer.async_grpo_trainer import AsyncRolloutLoop

AmineDiro

# Copyright 2020-2026 The HuggingFace Team. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""PROPOSED API: full ScaleRL-style recipe, independently copyable.

Use build_trainer(model_id, dataset, [binary_correctness_reward]).train().
Requires a reasoning model with </think>, no tools, and sufficient server context.
Set think_budget=None for a model without that delimiter. Aggregate reward > 0
is treated as success for the curriculum. Restore requires the coordinated
pipeline checkpoint proposal; state alone is not enough.

AmineDiro

"""PROPOSED API: resource-owning harness integration.

Use build_trainer(model_id, dataset, reward_funcs, harness_url).train().
Implement the application-specific open_harness_pool adapter below. Its returned
pool must provide async run(prompt, sample) and cancel_and_close(). run returns a
trace with turns, completion and completion_ids; it calls the supplied sampler
for every model turn, preserving its TurnRecords unchanged. No OpenEnv SDK method
names are assumed here. This file intentionally has one integration placeholder.
"""

AmineDiro

"""PROPOSED API: replace reward calculation with a child-owned HTTP judge.

Use build_trainer(model_id, dataset, judge_url).train(). This example defines its
own service contract: POST judge_url with {prompts, completions}; receive
{"rewards": [number_or_null, ...]} in conversation order. It is not an existing
TRL judge endpoint. The inherited generator and trainer loss are unchanged.
"""

from dataclasses import dataclass

AmineDiro

"""PROPOSED API: adaptive prompt filtering with constructor-injected state.

Use build_trainer(model_id, dataset, [binary_correctness_reward]).train().
The criterion assumes aggregate reward > 0 means success. The runtime calls the
inherited __init__(args, processing_class, state={...}); setup preserves restored
keys. Exact resume still requires the coordinated pipeline checkpoint design.
"""

from dataclasses import dataclass

AmineDiro

"""PROPOSED API: forced thinking budget, using the base scoring and loss.

Use build_trainer(model_id, dataset, [reward_correct]).train().
For a model with a </think> delimiter; single-turn, no tools. Ensure prompt plus
thinking/interruption/answer fits the server context. Turn records returned by
the sampling helper retain probabilities, support and provenance automatically.
"""

from dataclasses import dataclass

AmineDiro

# Copyright 2020-2026 The HuggingFace Team. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""PROPOSED API: CISPO through compute_loss only; no new learner hooks.

Use build_trainer(model_id, dataset, reward_funcs).train(). The forward/unpacking
and accounting follow TRL fe422f7a79c8613e98504cd4d0d45e3a1ba96909, Apache-2.0.
Source: trl/experimental/async_grpo/async_grpo_trainer.py at that revision.

This recipe replaces the objective with upper-cap CISPO, with score centering
disabled and no additional importance-ratio rejection mask. It retains the

AmineDiro

"""PROPOSED API: change only the baseline; retain the default trainer loss.

Use build_trainer(model_id, dataset, [reward_correct]).train().
This is a leave-one-out advantage estimator, not a complete RLOO paper recipe.
Unscorable conversations are excluded from the baseline and receive advantage 0.
"""

import numpy as np

from trl import AsyncGRPOConfig, AsyncGRPOTrainer

AmineDiro

"""PROPOSED API: existing environment/tools interface, no rollout subclass.

Use build_trainer(model_id, dataset).train(). Each row may provide target=1..10
and a conversational prompt; the default target is 7. The model needs a supported
tool-calling chat template. Environment state belongs to one active rollout and
is reset on reuse; it is distinct from persistent curriculum state in loop.state.
"""

from trl import AsyncGRPOConfig, AsyncGRPOTrainer