Core Operational Concepts

• 1: Deployment Architecture
• 2: Availability and Resiliency
• 3: Erasure Coding
• 4: Object Healing
• 5: Object Scanner
• 6: Thresholds and Limits

minio server https://minio{1...4}.example.net/mnt/disk{1...4}

             |                    Server Pool                |

minio server https://minio{1...4}.example.net/mnt/disk{1...4} \
             https://minio{5...8}.example.net/mnt/disk{1...4}

             |                    Server Pool                |

https://minio-{1...4}.example.net/mnt/drive-{1...4}
https://minio-{5...8}.example.net/mnt/drive-{1...4}
https://minio-{9...12}.example.net/mnt/drive-{1...4}

https://minio-{1...4}.example.net/mnt/drive-{1...4}
https://minio-{9...12}.example.net/mnt/drive-{1...4}
https://minio-{13...16}.example.net/mnt/drive-{1...4}

1 - Deployment Architecture

This page provides an overview of MinIO deployment architectures from a production perspective. For information on specific hardware or software configurations, see:

• Hardware Checklist
• Security Checklist
• Software Checklist
• Thresholds and Limits

Distributed MinIO Deployments

A production MinIO deployment consists of at least 4 MinIO hosts with homogeneous storage and compute resources.

MinIO aggregates these resources together as a pool and presents itself as a single object storage service.

MinIO provides best performance when using locally-attached storage, such as NVMe or SSD drives attached to a PCI-E controller board on the host machine.

Storage controllers should present XFS-formatted drives in “Just a Bunch of Drives” (JBOD) configurations with no RAID, pooling, or other hardware/software resiliency layers. MinIO recommends against caching, either at the drive or the controller layer. Either type of caching can cause I/O spikes as the cache fills and clears, resulting in unpredictable performance.

MinIO automatically groups drives in the pool into erasure sets.

Erasure sets are the foundational component of MinIO availability and resiliency. MinIO stripes erasure sets symmetrically across the nodes in the pool to maintain even distribution of erasure set drives. MinIO then partitions objects into data and parity shards based on the deployment parity and distributes them across an erasure set.

For a more complete discussion of MinIO redundancy and healing, see Erasure Coding and Object Healing.

MinIO uses a deterministic hashing algorithm based on object name and path to select the erasure set for a given object.

For each unique object namespace BUCKET/PREFIX/[PREFIX/...]/OBJECT.EXTENSION, MinIO always selects the same erasure set for read/write operations. MinIO handles all routing within pools and erasure sets, making the select/read/write process entirely transparent to applications.

Each MinIO server has a complete picture of the distributed topology, such that an application can connect and direct operations against any node in the deployment.

The MinIO responding node automatically handles routing internal requests to other nodes in the deployment and returning the final response to the client.

Applications typically should not manage those connections, as any changes to the deployment topology would require application updates. Production environments should instead deploy a load balancer or similar network control plane component to manage connections to the MinIO deployment. For example, you can deploy an NGINX load balancer to perform “least connections” or “round robin” load balancing against the available nodes in the deployment.

You can expand a MinIO deployment’s available storage through pool expansion.

Each pool consists of an independent group of nodes with their own erasure sets. MinIO must query each pool to determine the correct erasure set to which it directs read and write operations, such that each additional pool adds increased internode traffic per call. The pool which contains the correct erasure set then responds to the operation, remaining entirely transparent to the application.

If you modify the MinIO topology through pool expansion, you can update your applications by modifying the load balancer to include the new pool’s nodes. Applications can continue using the load balancer address for the MinIO deployment without any updates or modifications. This ensures even distribution of requests across all pools, while applications continue using the single load balancer URL for MinIO operations.

Client applications can use any S3-compatible SDK or library to interact with the MinIO deployment.

MinIO publishes its own SDK specifically intended for use with S3-compatible deployments.

MinIO uses a strict implementation of the S3 API, including requiring clients to sign all operations using AWS Signature V4 or the legacy Signature V2. AWS signature calculation uses the client-provided headers, such that any modification to those headers by load balancers, proxies, security programs, or other components will result in signature mismatch errors and request failure. Ensure any such intermediate components support pass-through of unaltered headers from client to server.

While the S3 API uses HTTP methods like GET and POST for all operations, applications typically use an SDK for S3 operations. In particular, the complexity of signature calculation typically makes interfacing via curl or similar REST clients impractical. MinIO recommends using S3-compatible SDKs or libraries which perform the signature calculation automatically as part of operations.

Replicated MinIO Deployments

MinIO site replication provides support for synchronizing distinct independent deployments.

You can deploy peer sites in different racks, datacenters, or geographic regions to support functions like BC/DR or geo-local read/write performance in a globally distributed MinIO object store.

Replication performance primarily depends on the network latency between each peer site.

With geographically distributed peer sites, high latency between sites can result in significant replication lag. This can compound with workloads that are near or at the deployment’s overall performance capacity, as the replication process itself requires sufficient free I/O to synchronize objects.

Deploying a global load balancer or similar network appliance with support for site-to-site failover protocols is critical to the functionality of multi-site deployments.

The load balancer should support a health probe/check setting to detect the failure of one site and automatically redirect applications to any remaining healthy peer.

The load balancer should meet the same requirements as single-site deployments regarding connection balancing and header preservation. MinIO replication handles transient failures by queuing objects for replication.

2 - Availability and Resiliency

This page provides an overview of MinIO’s availability and resiliency design and features from a production perspective.

Distributed MinIO Deployments

MinIO implements erasure coding as the core component in providing availability and resiliency during drive or node-level failure events.

MinIO partitions each object into data and parity shards and distributes those shards across a single erasure set.

MinIO uses a deterministic algorithm to select the erasure set for a given object.

For each unique object namespace BUCKET/PREFIX/[PREFIX/...]/OBJECT.EXTENSION, MinIO always selects the same erasure set for read/write operations. This includes all versions of that same object.

MinIO requires read and write quorum to perform read and write operations against an erasure set.

The quorum depends on the configured parity for the deployment. Read quorum always equals the configured parity, such that MinIO can perform read operations against any erasure set that has not lost more drives than parity.

With the default parity of EC:4, the deployment can tolerate the loss of 4 (four) drives per erasure set and still serve read operations.

Write quorum depends on the configured parity and the size of the erasure set.

If parity is less than 1/2 (half) the number of erasure set drives, write quorum equals parity and functions similarly to read quorum.

MinIO automatically increases the parity of objects written to a degraded erasure set to ensure that object can meet the same SLA as objects in healthy erasure sets. The parity upgrade behavior provides an additional layer of risk mitigation, but cannot replace the long-term solution of repairing or replacing damaged drives to bring the erasure set back to full healthy status.

With the default parity of EC:4, the deployment can tolerate the loss of 4 drives per erasure set and still serve write operations.

If parity equals 1/2 (half) the number of erasure set drives, write quorum equals parity + 1 (one) to avoid data inconsistency due to “split brain” scenarios.

For example, if exactly half the drives in the erasure set become isolated due to a network fault, MinIO would consider quorum lost as it cannot establish a N+1 group of drives for the write operation.

An erasure set which permanently loses more drives than the configured parity has suffered data loss.

For maximum parity configurations, the erasure set goes into “read only” mode if drive loss equals parity. For the maximum erasure set size of 16 and maximum parity of 8, this would require the loss of 9 drives for data loss to occur.

Transient or temporary drive failures, such as due to a failed storage controller or connecting hardware, may recover back to normal operational status within the erasure set.

MinIO further mitigates the risk of erasure set failure by “striping” erasure set drives symmetrically across each node in the pool.

MinIO automatically calculates the optimal erasure set size based on the number of nodes and drives, where the maximum set size is 16 (sixteen). It then selects one drive per node going across the pool for each erasure set, circling around if the erasure set stripe size is greater than the number of nodes. This topology provides resiliency to the loss of a single node, or even a storage controller on that node.

In the above topology, the pool has 8 erasure sets of 16 drives each striped across 16 nodes. Each node would have one drive allocated per erasure set. While losing one node would technically result in the loss of 8 drives, each erasure set would only lose one drive each. This maintains quorum despite the node downtime.

Each erasure set is independent of all others in the same pool.

If one erasure set becomes completely degraded, MinIO can still perform read/write operations on other erasure sets.

However, the lost data may still impact workloads which rely on the assumption of 100% data availability. Furthermore, each erasure set is fully independent of the other such that you cannot restore data to a completely degraded erasure set using other erasure sets. You must use Site or Bucket replication to create a BC/DR-ready remote deployment for restoring lost data.

For multi-pool MinIO deployments, each pool requires at least one erasure set maintaining read/write quorum to continue performing operations.

If one pool loses all erasure sets, MinIO can no longer determine whether a given read/write operation would have routed to that pool. MinIO therefore stops all I/O to the deployment, even if other pools remain operational.

To restore access to the deployment, administrators must restore the pool to normal operations. This may require formatting disks, replacing hardware, or replacing nodes depending on the severity of the failure. See Recover after Hardware Failure for more complete documentation.

Use replicated remotes to restore the lost data to the deployment. All data stored on the healthy pools remain safe on disk.

Replicated MinIO Deployments

MinIO implements site replication as the primary measure for ensuring Business Continuity and Disaster Recovery (BC/DR) in the case of both small and large scale data loss in a MinIO deployment.

MinIO replication can automatically heal a site that has partial or total data loss due to transient or sustained downtime.

Once all data synchronizes, you can restore normal connectivity to that site. Depending on the amount of replication lag, latency between sites and overall workload I/O, you may need to temporarily stop write operations to allow the sites to completely catch up.

If a peer site completely fails, you can remove that site from the configuration entirely. The load balancer configuration should also remove that site to avoid routing client requests to the offline site.

You can then restore the peer site, either after repairing the original hardware or replacing it entirely, by adding it back to the site replication configuration. MinIO automatically begins resynchronizing existing data while continuously replicating new data.

Sites can continue processing operations during resynchronization by proxying GET/HEAD requests to healthy peer sites

The client receives the results from first peer site to return any version of the requested object.

PUT and DELETE operations synchronize using the regular replication process. LIST operations do not proxy and require clients to issue them exclusively against healthy peers.

3 - Erasure Coding

• Overview of MinIO Erasure Coding

MinIO implements Erasure Coding as a core component in providing data redundancy and availability. This page provides an introduction to MinIO Erasure Coding.

See Availability and Resiliency and Deployment Architecture for more information on how MinIO uses erasure coding in production deployments.

Erasure Coding Basics

MinIO groups drives in each server pool into one or more Erasure Sets of the same size.

MinIO determines the optimal number and size of erasure sets when initializing a server pool. You cannot modify these settings after this initial setup.

For each write operation, MinIO partitions the object into data and parity shards.

Erasure set stripe size dictates the maximum possible parity of the deployment. The formula for determining the number of data and parity shards to generate is:

N (ERASURE SET SIZE) = K (DATA) + M (PARITY)

You can set the parity value between 0 and 1/2 the Erasure Set size.

Objects written with a given parity settings do not automatically update if you change the parity values later.

MinIO requires a minimum of K shards of any type to read an object.

The value K here constitutes the read quorum for the deployment. The erasure set must therefore have at least K healthy drives in the erasure set to support read operations.

MinIO cannot reconstruct an object that has lost read quorum. Such objects may be recovered through other means such as replication resynchronization.

MinIO requires a minimum of K erasure set drives to write an object.

The value K here constitutes the write quorum for the deployment. The erasure set must therefore have at least K available drives online to support write operations.

If Parity EC:M is exactly 1/2 the erasure set size, write quorum is K+1

This prevents a split-brain type scenario, such as one where a network issue isolates exactly half the erasure set drives from the other.

The K+1 logic ensures that a client could not potentially write the same object twice - once to each “half” of the erasure set.

For an object maintaining read quorum, MinIO can use any data or parity shard to heal damaged shards.

Use the MinIO Erasure Coding Calculator to explore the possible erasure set size and distributions for your planned topology. Where possible, use an even number of nodes and drives per node to simplify topology planning and conceptualization of drive/erasure-set distribution.

Erasure Parity and Storage Efficiency

Setting the parity for a deployment is a balance between availability and total usable storage. Higher parity values increase resiliency to drive or node failure at the cost of usable storage, while lower parity provides maximum storage with reduced tolerance for drive/node failures. Use the MinIO Erasure Code Calculator to explore the effect of parity on your planned cluster deployment.

The following table lists the outcome of varying erasure code parity levels on a MinIO deployment consisting of 1 node and 16 1TB drives:

Bit Rot Protection

Bit rot is silent data corruption from random changes at the storage media level. For data drives, it is typically the result of decay of the electrical charge or magnetic orientation that represents the data. These sources can range from the small current spike during a power outage to a random cosmic ray resulting in flipped bits. The resulting “bit rot” can cause subtle errors or corruption on the data medium without triggering monitoring tools or hardware.

MinIO’s optimized implementation of the HighwayHash algorithm ensures that it captures and heals corrupted objects on the fly. Integrity is ensured from end to end by computing a hash on READ and verifying it on WRITE from the application, across the network, and to the memory or drive. The implementation is designed for speed and can achieve hashing speeds over 10 GB/sec on a single core on Intel CPUs.

4 - Object Healing

What is healing?

Healing is MinIO’s ability to restore an object that has been damaged, corrupted, or partially lost. The loss can come from multiple types of corruptions or loss, such as but not limited to:

• drive-level errors or failure
• OS or filesystem errors or failure
• bit rot

Healing and Erasure Coding

The ability of MinIO to restore a damaged object relates directly to the following:

• total number of drives in the erasure set where the object exists

• number of drives available with intact parts of the object

• parity setting for the erasure set

  Parity refers to the number of dedicated recovery shards MinIO creates when writing the object. For example, an erasure set may have eight total drives and use three drives during a write for parity. In this scenario, MinIO splits an object into 5 data shards and create 3 parity shards. MinIO distributes these eight shards across the drives in the erasure set. No one drive contains only parity shards or only data shards. Instead, MinIO writes shards for each object in a randomized way to distribute reads evenly across drives.

  When MinIO needs to provide the object, it looks for the data shards for the object. If any of the data shards are missing or damaged, MinIO uses one or more of the parity shards to restore the object. When looking for the parity shards, if any of the parity shards are missing or damaged, MinIO restores those as well, provided there are sufficient other shards to serve the object. For this scenario, up to three of data shard parts can be lost or damaged and MinIO can still successfully restore and serve the object.

  The number of drives available with intact data or parity shards of the object must meet or exceed the number of drives used for data shards in the erasure set. In the scenario above, five drives with intact shards must be online and available for MinIO to successfully serve the object.

When does MinIO heal an object?

MinIO has a robust system for healing objects.

Healing during GET requests

MinIO automatically checks the consistency of an object’s data shards each time you request an object with a GET or HEAD operation. For versioned buckets, MinIO also checks for consistency during PUT operation.

If all of the data shards are found intact, MinIO serves the object from the data shards without inspecting the corresponding parity shards.

If the object has missing or damaged data shards, MinIO uses the available parity shards to heal the object before serving it as part of the operation. There must be an intact parity shard available for each lost or damaged data shard, otherwise the object cannot be recovered. If any parity shards are lost or damaged, MinIO restores the parity shard, provided there are sufficient other parity shards to serve the object.

Healing with the object scanner

MinIO uses an object scanner to perform a number of tasks related to objects. One of these tasks checks the integrity of objects and, if found damaged or corrupted, heals them.

On each scanning pass, MinIO uses a hash of the object name to select one out of every 1,024 objects to check.

If any object is found to have lost shards, MinIO heals the object from available shards. By default, MinIO does not check for bit rot corruption using the scanner. This can be an expensive operation to perform and the risk of bit rot across multiple disks is low.

Healing by manual request

Administrators can use mc admin heal to initiate a full system healing. The procedure is very resource intensive and not typically needed.

Consult with MinIO Engineers before manually starting a healing process on a deployment.

Healing metrics

MinIO provides several healing metrics to monitor the status of healing processes on a deployment.

Refer to the Metrics and alerts for more information on available endpoints and configuration.

5 - Object Scanner

Overview

MinIO uses the built-in scanner to check objects for healing and to take any scheduled object actions. Such actions may include:

• calculate data usage on drives
• evaluate and apply configured lifecycle management or object retention rules
• perform bucket or site replication
• check objects for missing or corrupted data or parity shards and perform healing

The scanner performs these functions at two levels: cluster and bucket. At the cluster level, the scanner splits all buckets into groups and scans one group of buckets at a time. The scanner starts with any new buckets added since the last scan, then randomizes the scanning of other buckets. The scanner completes checks on all bucket groups before starting over with a new set of scans.

At the bucket level, the scanner groups items in buckets and scans selected items from that bucket. The scanner selects objects for a scan based on a hash of the object name. Over a span of 16 scans, MinIO checks every object in the namespace. MinIO fully scans any prefixes known to be new since the last scan.

Scan Length

Multiple factors impact the time it takes for a scan to complete.

Some of these factors include:

• Type of drives provided to MinIO
• Throughput and iops available
• Number and size of objects
• Other activity on the MinIO Server

For example, by default, MinIO pauses the scanner to make I/O operations available for read and write requests. This can lengthen the time it takes for a scan to complete.

MinIO waits between each scan by a factor multiplication of the time it takes each scan operation to complete. By default, the value of this factor is 10.0, meaning MinIO waits 10x the length of an operation after one scan completes before starting the next scan. The value of this factor changes depending on the configured scanner speed setting.

Scanner Performance

Many factors impact the scanner performance. Some of these factors include:

• available node resources
• size of the cluster
• number of erasure sets compared to the number of drives
• complexity of bucket hierarchy (objects and prefixes).

For example, a cluster that starts with 100TB of data and then grows to 200TB of data may require more time to scan the entire namespace of buckets and objects given the same hardware and workload. Likewise, a single erasure set of 16 drives takes longer to scan than the same number of drives split into two erasure sets of 8 drives each.

MinIO treats the scanner as a background task and pauses it in favor of completing read and write requests on the cluster. As the cluster or workload increases, scanner performance decreases as it yields more frequently to ensure priority of normal S3 operations.

You can adjust how MinIO balances the scanner performance with read/write operations using either the MINIO_SCANNER_SPEED environment variable or the scanner speed configuration setting.

Scanner Metrics

MinIO provides a number of metrics related to the scanner.

Use mc admin scanner info to see the current status of the scanner and the time since the last full scan. This can help in understanding the metrics provided by the scanner operation.

Scanner metrics, including usage metrics, reflect the last completed scan. PUT or DELETE operations since the last scan do not update in the usage until the next scan of the affected bucket(s).

The output resembles the following:

Overall Statistics
------------------
Last full scan time:   0d0h14m; Estimated 2885.28/month
Current cycle:         70464; Started: 2024-04-19 20:02:34.568479139 +0000 UTC
Active drives:         2

Last Minute Statistics
----------------------
Objects Scanned:       620 objects; Avg: 124.929µs; Rate: 892800/day
Versions Scanned:      620 versions; Avg: 2.801µs; Rate: 892800/day
Versions Heal Checked: 0 versions; Avg: 0ms
Read Metadata:         621 objects; Avg: 88.416µs, Size:
ILM checks:            656 versions; Avg: 663ns
Check Replication:     656 versions; Avg: 1.061µs
Verify Deleted:        0 folders; Avg: 0ms
Yield:                 3.086s total; Avg: 4.705ms/obj

6 - Thresholds and Limits

This page reflects limits and thresholds that apply to MinIO.

Refer to the hardware and software for related recommendations and requirements.

S3 API Limits

Erasure Code Limits

Object Name Limitations

Filesystem and Operating System Restrictions

Object Names in MinIO are restricted primarily by the local operating system and filesystem. Windows and some other operating systems restrict file systems with certain special characters, such as ^, *, |, \, /, &, ", or ;.

This list is not exhaustive and may not apply to your operating system and filesystem combination.

On Unix-like operating systems, objects with a path name of ., .., or / return an error of file access denied.

Consult your operating system vendor or filesystem documentation for a comprehensive list for your situation.

MinIO recommends using a Linux operating system with an XFS based filesystem for production workloads.

Conflicting Objects

Applications must assign non-conflicting, unique keys for all objects. This includes avoiding creating objects where the name can collide with that of a parent or sibling object. MinIO returns an empty set for LIST operations at the location of the collision.

For example, the following operations create a namespace conflicts

PUT data/invoices/2024/january/vendors.csv
PUT data/invoices/2024/january <- collides with existing object prefix

PUT data/invoices/2024/january
PUT data/invoices/2024/january/vendors.csv <- collides with existing object

While you can perform GET or HEAD operations against these objects, the name collision causes LIST operations to return an empty result set at the /invoices/2024/january path.