> ## Documentation Index
> Fetch the complete documentation index at: https://docs.outerproduct.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Retries and durability

> Configure retries and understand how completed work is reused.

An OuterProduct Run forms a computation graph. The top-level `client.run()` creates a new Run and its root Call. Inside that Call, the scoped client can create child Calls within the same Run.

This hierarchy allows OuterProduct to track each Call independently. For example, in the following code, the `pipeline` call orchestrates a 2-step workflow (first, `prepare_data` , then `train_model` ). Every function receives an OuterProduct client as its first argument. A function can ignore it, as `_client` does in the example below, or use it to create more Calls.

Breaking an application into Calls gives OuterProduct durable boundaries where work can be retried, tracked, and checkpointed.

```python theme={null}
import outerproduct as op

image: op.Image = op.Image.from_registry("python:3.12-slim")


def prepare_data(_client, records: list[str]) -> list[str]:
    # ...
    return prepared_records


def train_model(_client, records: list[str]) -> object:
    # ...
    return trained_model


def pipeline(client, records: list[str]) -> object:
    prepared: list[str] = client.run(
        fn=prepare_data,
        args=(records,),
        image=image,
    ).wait()

    return client.run(
        fn=train_model,
        args=(prepared,),
        image=image,
    ).wait()


if __name__ == "__main__":
    client = op.init()

    run: op.Run[object] | op.Call[object] = client.run(
        fn=pipeline,
        args=(["To be", "or not to be"],),
        image=image,
    )

    model: object = run.wait()
```

## Retrying a Call

Use `RetryStrategy` to specify the maximum number of Attempts for a Call:

```python theme={null}
run: op.Run[object] | op.Call[object] = client.run(
    fn=train_model,
    args=(training_data,),
    image=training_image,
    gpu_type="L4",
    retry_strategy=op.RetryStrategy(max_attempts=4),
)
```

`max_attempts` includes the first Attempt. In this example, OuterProduct can try the Call up to three times. If no retry strategy is provided, the Call receives three Attempt by default. Pass `RetryStrategy(max_attempts=1)` to disable retries. Each Call defines its own retry strategy; a parent and its children can therefore use different limits.

## Durable execution

OuterProduct checkpoints work at Call boundaries. Once a Call succeeds, its result is stored as part of the Run. If a parent Call fails and is retried, it begins executing again. When it reaches a child Call that already succeeded, OuterProduct returns the stored result instead of executing that child again.

A long workflow can therefore recover from a failure without repeating all of its successful work. For example:

```text theme={null}
Attempt 1
├── prepare_data   ✓ completed
├── train_model    ✓ completed
└── publish_model  ✗ failed

Attempt 2
├── prepare_data   ↪ reuse result
├── train_model    ↪ reuse result
└── publish_model  ✓ completed
```

> [!WARNING]
>
> If a Call fails halfway through its function, its next Attempt starts that function again from the beginning. Split expensive or important stages into separate Calls when you want them checkpointed independently.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.