Storage Configuration

Configure the storage backend for Iceberg data files.

Overview

Bingsan supports three storage backends:

S3 - Amazon S3 and compatible services (MinIO, R2)
GCS - Google Cloud Storage
Local - Local filesystem (development only)

Options

storage:
  type: s3
  warehouse: s3://bucket/warehouse

  s3:
    endpoint: ""
    region: us-east-1
    access_key_id: ""
    secret_access_key: ""
    bucket: warehouse
    use_path_style: false

  gcs:
    project: ""
    credentials_file: ""
    bucket: ""

  local:
    root_path: /tmp/iceberg/data

S3 Options

Option	Type	Default	Description
`endpoint`	string	`""`	Custom endpoint (for MinIO, R2)
`region`	string	`us-east-1`	AWS region
`access_key_id`	string	`""`	AWS access key
`secret_access_key`	string	`""`	AWS secret key
`bucket`	string	`warehouse`	S3 bucket name
`use_path_style`	boolean	`false`	Use path-style URLs

Amazon S3

With IAM Role (Recommended)

When running on EC2, ECS, EKS, or Lambda:

storage:
  type: s3
  warehouse: s3://my-bucket/warehouse

  s3:
    region: us-east-1
    bucket: my-bucket
    # No credentials needed - uses instance role

Required IAM Permissions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ]
    }
  ]
}

MinIO

storage:
  type: s3
  warehouse: s3://warehouse/data

  s3:
    endpoint: "http://minio:9000"
    region: us-east-1
    access_key_id: "minioadmin"
    secret_access_key: "minioadmin"
    bucket: warehouse
    use_path_style: true

Google Cloud Storage

storage:
  type: gcs
  warehouse: gs://my-bucket/warehouse

  gcs:
    project: my-gcp-project
    bucket: my-bucket
    credentials_file: /path/to/service-account.json

Local Storage

Local storage is intended for development and testing only.

storage:
  type: local
  warehouse: file:///data/warehouse

  local:
    root_path: /data/warehouse

Storage Configuration

On this page