> ## Documentation Index
> Fetch the complete documentation index at: https://docs.outerproduct.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Volumes

> Store and mount data using managed or external object storage.

A volume is an OuterProduct resource that maps a name to a location in object storage. The volume stores metadata about that location; the data and objects remain in the backing object store.

Volumes have two storage types:

* **OuterProduct-managed storage**: OuterProduct allocates and manages the backing object-store location. This is the default when creating a volume.
* **External storage**: The volume references an existing S3 bucket or Google Cloud Storage bucket. OuterProduct does not copy the existing objects.

Both types use the same Python API and can be mounted as files in a function’s runtime environment.

## Creating a managed volume

`client.create_volume()` creates a volume and returns a `VolumeHandle`. OuterProduct manages the underlying object storage by default when the `provider` argument is omitted. Each volume has a name consisting of `volumes/` followed by the string provided at creation time.

```python theme={null}
import outerproduct as op

client = op.init()
volume_handle: op.VolumeHandle = client.create_volume("datasets")
print(volume_handle.name)  # volumes/datasets
```

Volume names are case-insensitive and must be unique within a workspace. Creating a volume with an existing name raises an `already_exists` error.

## Creating a volume based on an external object store

OuterProduct Volumes can also be created from existing S3 buckets and Google Cloud Storage buckets without copying their contents.

To connect to an external store, cloud credentials should first be stored as OuterProduct [Secrets](https://docs.outerproduct.com/secrets.md).

> [tab] Amazon S3
>
> An S3 volume specifies a bucket and a stored AWS credential:
>
> ```python theme={null}
> credential: op.Secret = client.get_secret("aws-storage")
>
> volume_handle: op.VolumeHandle = client.create_volume(
>     "external-datasets",
>     provider=op.VolumeProvider.S3(
>         bucket="my-bucket",
>         prefix="datasets/",
>         region="us-east-1",
>         credential=credential.name,
>     ),
> )
> ```
>
> With this prefix, `training.csv` in the volume corresponds to `s3://my-bucket/datasets/training.csv`. If no prefix is specified, paths will be relative to the bucket’s root.

> [tab] Google Cloud Storage
>
> A Google Cloud Storage volume specifies a bucket and a stored GCP credential:
>
> ```python theme={null}
> credential: op.Secret = client.get_secret("gcp-storage")
>
> volume_handle: op.VolumeHandle = client.create_volume(
>     "external-datasets",
>     provider=op.VolumeProvider.Gcs(
>         bucket="my-bucket",
>         prefix="datasets/",
>         credential=credential.name,
>     ),
> )
> ```
>
> With this prefix, `training.csv` in the volume corresponds to `gs://my-bucket/datasets/training.csv`. If no prefix is specified, paths will be relative to the bucket’s root.

## Managing volume resources

`client.list_volumes()` returns accessible `Volume` metadata, while `client.get_volume()` takes a volume name (with or without the `volumes/` prefix) and returns a handle to an existing volume. Use `update_volume()` to rename a volume or change its description without changing its identity.

`client.delete_volume()` removes the volume resource and its metadata. Deleting an OuterProduct-managed volume also deletes its stored files. For a volume backed by your own object store, the files are retained.

```python theme={null}
for volume in client.list_volumes():
    print(volume.name)

volume_handle: op.VolumeHandle = client.get_volume("datasets")

client.update_volume(
    "datasets",
    new_name="training-data",
    description="Training inputs",
)

client.delete_volume("old-datasets")
```

## Reading and writing data

`VolumeHandle` provides methods for reading and writing objects through [obstore](https://developmentseed.org/obstore/).

```python theme={null}
from pathlib import Path

from obstore import GetResult

volume_handle.put("training.csv", Path("training.csv"))

result: GetResult = volume_handle.get("training.csv")
Path("downloaded.csv").write_bytes(bytes(result.bytes()))
```

`put()` accepts bytes, local files, and streams. `get()` returns a result whose contents can be read into memory with `.bytes()` or consumed as a stream. When using these methods, object paths should be specified relative to the volume's root.

The handle also supports other [standard methods](https://developmentseed.org/obstore/latest/getting-started/#configuration) for interacting with an object store, including listing objects, inspecting metadata, copying, renaming, and deleting data.

```python theme={null}
from obstore import ListResult, ObjectMeta

listing: ListResult = volume_handle.list_with_delimiter()
metadata: ObjectMeta = volume_handle.head("training.csv")

volume_handle.copy("training.csv", "archive/training.csv")
```

`volume_handle.list(prefix)` is an async iterator that yields batches of object metadata under the prefix. It is used with `async for` and has no synchronous form. See [Async usage](https://docs.outerproduct.com/async-usage.md).

## Mounting a volume

The `volumes` argument to `client.run()` is a dictionary mapping filesystem paths to volume handles, volume names, or UUIDs. At runtime, the volume’s contents are mounted at the specified path:

```python theme={null}
def count_rows(client) -> int:
    import csv

    with open("/data/training.csv") as file:
        return sum(1 for _ in csv.DictReader(file))


if __name__ == "__main__":
    result: op.Run[int] | op.Call[int] = client.run(
        fn=count_rows,
        image=image,
        volumes={"/data": volume_handle},  # count_rows can access the volume at /data
	   timeout=3600,  # mounts stop working an hour after the Attempt starts
    )

    print(result.wait())
```

Volume mounts are *not* inherited by child calls, so each child call must declare its own mounts.

Mounting requires both read and write access to the volume. See [Policies](https://docs.outerproduct.com/policies.md) for more information on permissions in OuterProduct.

Mounted volumes support reading files and sequential writes to new files. They do not support in-place edits, appending, or atomic renames.

> [!WARNING]
>
> Volume mounts use temporary credentials and stop working an hour after an Attempt starts. Set `timeout=3600` or less on `client.run()` for Calls that mount volumes; `client.run()` warns otherwise. Calls that need longer should read and write the volume through `VolumeHandle` methods such as `get()` and `put()`, which refresh their credentials as needed.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.