SeaweedFS

SeaweedFS is an open-source distributed storage system written in Go. It stores billions of small files efficiently and serves them with O(1) disk reads, while also exposing S3-compatible object storage, a POSIX-like filesystem via FUSE, WebDAV, Hadoop compatibility, and a built-in Iceberg REST catalog. It's aimed at teams that need self-hosted, scalable storage without paying cloud egress and API costs, or that want a lightweight alternative to HDFS, Ceph, or GlusterFS.

The architecture splits metadata from data: a lightweight master tracks volumes, while volume servers handle files and their own metadata, keeping per-file overhead around 40 bytes. This design traces back to Facebook's Haystack paper and borrows ideas from Facebook's f4 warm storage and Tectonic filesystem, plus Google's Colossus. An optional Filer layer adds directories, POSIX attributes, and pluggable metadata stores (MySQL, Postgres, Redis, Cassandra, Etcd, and others).

Key features

• S3-compatible API: Access buckets and objects with standard S3 tooling and credentials.
• POSIX filesystem mount: Mount storage as a local directory via FUSE for regular file access.
• Erasure coding: Rack-aware 10.4 erasure coding for warm data reduces storage cost and boosts availability.
• Cloud tiering: Transparently tier hot data locally and warm data to S3, GCS, Azure, or Backblaze, cutting cloud API costs.
• Filer metadata stores: Plug in MySQL, Postgres, Redis, Cassandra, RocksDB, Etcd, TiDB, and more for filer metadata.
• Active-active replication: Asynchronous one-way or two-way replication across clusters.
• Iceberg REST catalog: Built-in catalog lets Spark, Trino, Dremio, DuckDB, and RisingWave query Iceberg tables without a separate metastore.
• Kubernetes support: CSI driver and an operator for running SeaweedFS on Kubernetes.
• Hadoop compatibility: Drop-in Hadoop-compatible filesystem for Spark, Flink, and HBase workloads.
• Encryption and TTL: AES256-GCM encrypted storage and automatic file/entry TTL expiration.

Ideal use cases

• Storing billions of small files (images, thumbnails, logs, user uploads) where per-file metadata overhead matters.
• Running a self-hosted S3-compatible endpoint for internal apps, backups, or CI artifacts without cloud storage bills.
• Mounting cloud or cluster storage locally via FUSE for legacy apps that expect a filesystem.
• Small analytics teams wanting a lakehouse (Iceberg tables queryable by Spark/Trino/DuckDB) without standing up a Hive Metastore or Glue.
• Hybrid setups where hot data stays local for speed and cold/warm data tiers to cloud storage automatically.
• Not a great fit if you need a fully managed service with SLAs and support out of the box, or if your workload is a handful of huge files rather than many small ones, since SeaweedFS's core design optimizes for small-file efficiency. Teams unwilling to operate distributed infrastructure (masters, volume servers, filers) may prefer a managed object store instead.

Installation

The fastest way to try it is the all-in-one weed mini command, which starts master, volume server, filer, S3 API, WebDAV, and admin UI together.

Download a binary from the releases page, or build from source:

go install github.com/seaweedfs/seaweedfs/weed@latest

Start a single-node S3 store with a pre-created bucket and credentials:

AWS_ACCESS_KEY_ID=admin \
AWS_SECRET_ACCESS_KEY=secret \
S3_BUCKET=my-bucket \
./weed mini -dir=/data

This exposes the S3 endpoint at localhost:8333, the Master UI at :9333, the Volume Server at :9340, the Filer UI at :8888, WebDAV at :7333, and the Admin UI at :23646. Omit the AWS env vars to run unauthenticated for local development.

Alternatively, run it via Docker:

docker run -p 8333:8333 \
  -e AWS_ACCESS_KEY_ID=admin \
  -e AWS_SECRET_ACCESS_KEY=secret \
  -e S3_BUCKET=my-bucket \
  chrislusf/seaweedfs

To scale beyond a single node, start a dedicated master and add volume servers pointing at it:

./weed master
weed volume -dir="/tmp/data1" -max=5 -master="localhost:9333" -port=8080 &
weed volume -dir="/tmp/data2" -max=10 -master="localhost:9333" -port=8081 &

Additional volume servers can run on the same machine or be distributed across thousands of machines by pointing them at the master's address.