This is the multi-page printable view of this section. .
Compatibility
- 1: Migrate from MinIO to Silo
- 2: Silo vs. MinIO Server Compatibility
- 3: MCLI Client Compatibility Notes
- 4: Native Package Migration
- 5: Feature Notes
Silo is a community fork of MinIO. This section records what Silo keeps from MinIO, where it deliberately differs, and what either means when moving between them.
Contents, in reading order: the migration guide (scope and Docker), native package migration (RPM/DEB), the server compatibility audit, and the mcli client notes. The Feature Notes subsection covers the opposite direction: design records for capabilities Silo adds beyond upstream.
1 - Migrate from MinIO to Silo
Migrating from MinIO to Silo is an in-place binary replacement, not a data migration. Nothing is exported or re-imported. In a container deployment the only required change is the image name. For RPM/DEB installations, see Native Package Migration.
What changes
In order of importance:
- Container image:
minio/minio,quay.io/minio/minio, andpgsty/minioare all replaced bydocker.io/pgsty/silo. - Package, systemd service, and server executable:
minio→silo. - Upstream services: the in-place updater and MinIO-operated callhome/SUBNET are disabled; upgrades go through packages, images, or your orchestrator.
- Default OS service account:
silo— fresh installations only; migrations keep running as the existing data owner. - Branding: banners, Console appearance, log wording, and product links say Silo.
What stays
- Object data and the
.minio.sysmetadata directory — the on-disk format is unchanged and remains interoperable with MinIO in both directions. - Buckets, versions, users, access keys, policies, lifecycle rules, replication state, encryption metadata.
- S3 API, SigV4 signing, SDKs,
mc/mcli, presigned URL behavior. - Endpoint hostname, API port
9000, Console port, volume mounts. MINIO_*environment variables and existing server options./minio/*routes,x-minio-*headers,minio_*metrics.
There is no data-conversion step. If your MinIO build is years old, validate the version distance itself in staging; it is a large software upgrade, not a format change.
Docker migration
Whichever image you run today, replace it with:
Tags: immutable RELEASE.YYYY-MM-DDTHH-MM-SSZ (pin these), rolling latest, and the -distroless variants below. The old pgsty/minio repository stays published, frozen at its final tag.
In Compose, change only the image line:
The entrypoint translates the legacy first argument, so an inherited command: minio server /data keeps working. A hard-coded entrypoint: /usr/bin/minio must change to /usr/bin/silo. Existing mc ready local healthchecks keep working; the native replacement is test: ["CMD", "silo", "healthcheck", "ready"] (reference). Do not run docker compose down -v — -v deletes the data volume.
Distroless variant
pgsty/silo:<RELEASE-tag>-distroless ships the silo binary only: no shell, no mc, no curl. It has a built-in HEALTHCHECK (the native probe) and works under any --user:
The same deployment as a Compose file:
depends_on: condition: service_healthy works against it with no healthcheck: block. The volume format is the same as the classic image and MinIO — the variants are interchangeable over the same data. TLS certificates mount at /tmp/.silo/certs. If command-line flags move the listen address, point the built-in probe with MINIO_HEALTHCHECK_URL. There is no shell inside; debug with docker debug / kubectl debug.
Kubernetes
Kubelet probes are httpGet requests in the pod spec; Docker HEALTHCHECK is ignored, so both image variants are probed identically and existing probe configs keep working. For Helm releases, keep the release identity with nameOverride/fullnameOverride and compare helm template output before applying (details).
Rollback
The disk format is unchanged and works with both servers: set image: back to the recorded MinIO tag and docker compose up -d. The same volume stays attached, and data written by Silo remains readable by MinIO.
One cluster, one binary
Distributed nodes verify each other’s binary at bootstrap. A node started among peers running a different binary does not fail — it waits indefinitely in activating, logging:
This applies to any pair of different binaries: MinIO next to Silo, and one Silo version next to another. So do not migrate — or later upgrade — a cluster node by node. Switch all nodes in one pass: stop the old binary everywhere, start the new one everywhere (in Compose: change the image for all nodes in one edit, docker compose up -d once). Single-node deployments are unaffected. The same applies to rollback. Rolling restarts of the same binary work normally; gate them with silo healthcheck --maintenance cluster (exit 0 = safe to stop this node).
Verification
Then download a known object and compare its checksum, exercise one application through its existing SDK, restart the service once, and re-check.
2 - Silo vs. MinIO Server Compatibility
Silo is a maintained fork of the MinIO server. It preserves MinIO’s S3-facing and on-disk compatibility, but it is not a byte-for-byte, operationally invisible rename. This page is the compatibility contract for moving from the upstream baseline to the Silo source prepared on 2026-08-06.
Read this before replacing a MinIO deployment. The binary, package, service account, systemd unit, default local configuration directory, container path, Helm resource names, embedded Console, update behavior, several authorization decisions, and some error responses changed. Data disks and the MINIO_* configuration namespace did not receive a matching rename.
Audit scope and method
This is a source audit, not a compilation of release-note claims.
| Boundary | Audited value |
|---|---|
| Upstream baseline | minio/minio@7aac2a2c5b7c882e68c1ce017d8256be2feea27f, 2026-02-11 |
| Silo snapshot | pgsty/silo@219670d3176a5b27ded60914390d5ee7e763cf58, 2026-08-06 |
| Commit set | 7aac2a2c..219670d3: 96 reachable commits; 93 were on origin/main and the final three were committed locally at audit time |
| Net source diff | 523 files, 36,715 insertions, 21,450 deletions |
| Interpretation | The behavior of the final snapshot. Intermediate changes later replaced or removed are not presented as current behavior |
Every commit in the range was inspected. Release and security posts were used as an index of intended changes, then checked against the final implementation, tests, dependency graph, build recipes, package payloads, container entrypoint, and rendered Helm manifests. The complete commit ledger prevents a documentation-only or CI-only commit from silently falling out of scope.
The range is defined by Git reachability, not author-date sorting. It therefore contains d4cd4b433, authored in December 2025 but joined into the post-baseline graph later; this is not an extra undocumented baseline.
The tagged RELEASE.2026-08-04T00-00-00Z ends at d88f46cce, 18 commits before this audit head. Accordingly, this page records the 2026-08-06 prepared source state; it does not claim that the last 18 changes were already present in a public package, image, tag, or deployed website.
Executive compatibility matrix
| Surface | Status | Practical result |
|---|---|---|
| S3 wire API | Compatible with documented exceptions | Routes, XML/JSON schemas, SigV4, S3 headers, ports, and ordinary error codes retain the MinIO contract. Security and correctness fixes below deliberately reject some requests previously accepted |
| Data disks | Compatible | .minio.sys, erasure metadata, bucket/object layout, healing, replication, and encryption formats keep their names and schemas. Poisoned or unusable metadata is now rejected earlier |
| Configuration | Mostly compatible | Existing MINIO_* variables, config keys, KMS/KES, IAM, notification, and storage settings remain. The default per-user directory becomes ~/.silo, with a deterministic ~/.minio fallback |
| Metrics and automation APIs | Compatible | minio_* Prometheus metrics, /minio/* routes, x-minio-* headers, admin/S3 error identifiers, and release tag syntax stay unchanged |
| Binary and distribution | Renamed | minio becomes silo; package, unit, image, chart, archives, checksums, and paths move to the Silo identity. There is no installed server-binary alias |
| Runtime identity | Changed | CLI text, banners, HTTP Server, User-Agent application names, FTP banner, log names, support links, and some human-readable errors say Silo |
| Upstream network services | Disabled | In-place update, update polling, callhome, SUBNET registration, and diagnostic uploads do not contact MinIO services |
| Authorization/security | Intentionally stricter | OIDC HMAC tokens, unsafe LDAP failures, forged replication metadata, object-only grants for protected bucket writes, shadowed policy inputs, ambiguous version IDs, and several malformed internode requests change behavior |
| Embedded UI and Go dependencies | Forked behind compatible import paths | Silo Console, MCLI, and Silo Pkg are selected with replace directives while github.com/minio/... module/import paths remain |
| Mixed-version cluster | Not supported for this transition | The private ReadMultiple storage-REST operation was removed without bumping storage REST v63. Upgrade all nodes as one build |
What deliberately stays compatible
Protocol, storage, and configuration names
The following MinIO identifiers are compatibility identifiers, not unfinished branding work, and must remain visible:
- the Go module path
github.com/minio/minioand the inheritedgithub.com/minio/...imports; - the
MINIO_*environment namespace, includingMINIO_ROOT_USER,MINIO_ROOT_PASSWORD,MINIO_VOLUMES,MINIO_OPTS, and notification variables; - S3 and admin routes under
/minio/*,x-minio-*headers, MinIO-specific S3 extensions, and established API error codes; - Prometheus metric names under
minio_*; - the
.minio.sysinternal volume and all existing disk metadata names; - the default S3 port
9000, existing--address/--console-addressflags, and release tags of the formRELEASE.YYYY-MM-DDTHH-MM-SSZ; - configuration KV formats, IAM data, KMS/KES configuration, encryption metadata, bucket metadata, replication state, and healing state.
The automated rebrand baseline records 137 compatible imports, 436 environment names, 19 metric namespaces, 84 headers, 330 routes, one internal root, three Grid namespaces, 15 storage-REST identifiers, 58 policy identifiers, and 9,014 exported symbols. The guard treats an unreviewed change to that manifest as a compatibility failure.
No data copy or metadata rewrite is required when the same disks move from MinIO to Silo. This does not mean every malformed historical object is accepted: the storage hardening described below rejects unsafe paths, invalid erasure geometry, negative part sizes, and poisoned metadata that older code could carry farther into the stack.
Source compatibility
The server module remains github.com/minio/minio. Silo selects maintained forks without forcing callers to rewrite imports:
This preserves most source compatibility, but it is not an assertion that every private or exported Go symbol is frozen. The internal ReadMultiple storage interface was removed, and the selected silo-pkg release has several developer-visible fixes described in Dependencies.
The maintained source remote is github.com/pgsty/silo on branch main; the former minio branch is archived. go install github.com/minio/minio@... still resolves the upstream project, not Silo, so clone the Silo repository or use an explicit module replace. Contributions no longer require MinIO’s CLA, but commits require DCO sign-off (git commit -s).
Identity, binary, and outbound-service changes
| Surface | Upstream baseline | Silo snapshot | Compatibility consequence |
|---|---|---|---|
| Server executable | minio / minio.exe |
silo / silo.exe |
Scripts and absolute paths must change. Archives, packages, and images do not install a /usr/bin/minio server alias |
| Version output | MinIO identity | Silo release/commit/runtime, AGPL, upstream copyright, PGSTY modification copyright, and MinIO technology lineage | Parsers should rely on stable fields, not grep for MinIO prose |
| Local config home | ~/.minio |
~/.silo for a new home |
See the fallback rules below; data disks are unrelated |
| HTTP identity | Server: MinIO and MinIO application UAs |
Server: Silo; internal batch/fan-out/perf UAs use silo-* / Silo names |
Protocol headers such as x-minio-* remain unchanged; identity-sensitive monitoring may need an update |
| Human text | MinIO banner, help, errors, examples, FTP greeting, support links | Silo identity; examples prefer mysilo |
Exact-string log parsers and snapshots may change, not status/error codes unless listed elsewhere |
| Integration-visible labels | MinIO NATS/Redis connection names and Veeam model | NATS name Silo Notification, Redis CLIENT SETNAME Silo, Veeam model "Silo <release>" |
Broker dashboards, connection-name filters, and Veeam inventory display can change |
| KMS validation prose | MinIO-branded conflict messages | Brand-neutral “both KMS/KES/static-key configuration” messages | Configuration rules are the same; exact-text automation can change |
| Updater | Release polling and in-place update paths | Permanently disabled | MINIO_UPDATE is parsed but cannot re-enable it; admin update routes remain and fail stably instead of disappearing |
| Callhome/SUBNET | Registration, callhome, support uploads, embedded MinIO support key | Configuration is accepted for migration but forced off; no registration/upload/post; no fallback encryption key | Remove automation that expects MinIO-operated services. Requester-key inspect encryption remains |
| Embedded Console | Upstream snapshot had the Console stripped | Silo Console v2.1.1, English/Chinese UI, Metrics V3, no SUBNET UI | Browser behavior and assets change; the S3/Admin API boundary remains the server contract |
| Bundled client in OCI | No maintained fork contract | /usr/bin/mcli plus /usr/bin/mc -> mcli |
This mc is the client compatibility alias, never a server alias |
| Warm-tier probe | Temporary object contains MinIO |
Same-length probe contains Silo! |
Only observable through backend inspection or a failed cleanup; protocol semantics do not change |
| Log rotation default | minio-*.log |
silo-*.log |
Log collectors matching filenames must change |
The deleted /api/health/upload reference was an outbound SUBNET URL path, not a local Silo HTTP endpoint. The compatibility change is that Silo no longer issues that POST; it is incorrect to describe this as removal of a server route.
Default configuration-directory selection
Unless --config-dir is explicit, Silo makes one decision at startup:
| Home-directory state | Selected directory | Message |
|---|---|---|
| Neither directory exists | ~/.silo |
none |
Only ~/.silo exists |
~/.silo |
none |
Only ~/.minio exists |
~/.minio |
informational legacy notice; no files are moved |
| Both exist | ~/.silo |
ambiguity warning |
--config-dir always wins. Unless --certs-dir is also supplied, the certificate directory follows the selected configuration directory. For deterministic automation, set --config-dir instead of depending on filesystem discovery.
The fork adds only three server configuration controls that materially change compatibility behavior; there is no parallel SILO_* replacement namespace:
| Setting | Purpose | Default |
|---|---|---|
MINIO_API_TRUSTED_PROXIES |
General source-address trust boundary | unset: exact historical trust-any behavior |
MINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIES / LDAP key sts_trusted_proxies |
Source buckets used by LDAP STS failure limiting | no trusted proxy; use the socket peer |
MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH |
Temporary rollback for the protected bucket/object IAM boundary | off |
Notification KV registration fixes do not rename their existing environment variables. MINIO_UPDATE, SUBNET, and callhome inputs are retained only as ignored/migration-compatible inputs as described next. SILO_OPTS appears only inside the generated inspect helper, not as a general replacement for MINIO_OPTS.
Updater, callhome, SUBNET, and inspect
- Startup never polls
dl.min.io; the release URL and upstream minisign root are absent. MINIO_UPDATEvalues that request updates produce a warning and are ignored. Public and peer admin update handlers remain registered, returningMethodNotAllowedor the stable “in-place updates are disabled” error.- Legacy
subnetandcallhomekeys remain parseable so an old config can start. Registration state is always false, Console SUBNET variables are unset, callhome is forced off, and no diagnostic or license payload is posted. - Inspect output is encrypted only for a requester-provided public key. It no longer falls back to MinIO’s built-in support key. The helper is
start-silo.sh, invokessilo, and includescluster.infoonly in the requester-key flow. - Upgrade through a package manager, an image rollout, or an orchestrator. Do not call
mc admin update/mcli admin updateagainst Silo as an upgrade mechanism.
Installation and deployment compatibility
RPM, DEB, and APK
The package is silo for Linux amd64/arm64. Its relevant payload is:
The package creates a system silo:silo account without a home. It does not chown existing data, migrate ownership, stop a running MinIO service during installation, or declare package-manager Provides, Obsoletes, Replaces, or Conflicts against the minio package. Both packages can therefore be installed, but their services cannot run together through the shipped units.
silo.service has Conflicts=minio.service, runs as silo:silo, and reads /etc/default/minio first and /etc/default/silo second. The shipped Silo file has no active assignments, so a legacy file continues to work until an administrator overrides it in the later file. It starts:
Before switching the unit, make every data, certificate, KMS credential, and environment-file path readable by silo and every writable path writable by it. An old deployment owned by minio:minio will otherwise fail at startup. Package removal stops/disables silo.service; package upgrades do not deliberately stop the running service in the pre-remove hook.
Container image
The source repository is github.com/pgsty/silo; the intended image name is docker.io/pgsty/silo. There is no registry-level promise that pgsty/minio redirects to it.
- The server exists only at
/usr/bin/silo; an explicit/usr/bin/minio ...command breaks. - The entrypoint translates a first argv word of
miniotosilo, and prependssilowhen argv starts withserver,fmt-gen, or an option. Consequently the commoncommand: minio server /dataform continues to work. - An explicitly requested shell or utility is left alone.
- Every privilege path uses
exec, so the server becomes PID 1 and receivesSIGTERMfor graceful shutdown instead of timing out behind the entrypoint. HOME=/tmpis the image default and is normalized to a writable directory for arbitrary-UID and legacyMINIO_USERNAMEdrop-user execution.- Port
9000,/data, and theMINIO_*interface remain. The amd64/arm64 image manifest also contains checksummed MCLIRELEASE.2026-08-04T00-00-00Zand the client-onlymcsymlink. - OCI license material is under
/licenses/{LICENSE,NOTICE,CREDITS}.
Helm chart
The inherited helm/minio chart, helm-releases, root chart index, and reindex helper were removed. The maintained chart is helm/silo, chart version 7.0.0.
Most values deliberately keep their established names, including minioAPIPort, minioConsolePort, and all MINIO_* environment settings. The changes that matter during migration are:
- image repositories become
pgsty/silo; - generated resource names and labels follow chart name
silo; - default service account becomes
silo-sa; - certificate/client mount paths move from
/etc/minio/{certs,mc}to/etc/silo/{certs,mc}; - no insecure
console/console123user is created by default (users: []); - post-job examples prefer alias
mysilo, whilemyminiois also registered so inheritedcustomCommandscan still resolve it; - the new chart executes
silo, so an image-only rollback to an old MinIO image is not safe.
To preserve the old Kubernetes object identities while adopting the new chart, start from the exact old values and set at least:
Render both charts and compare Services, selectors, StatefulSets/Deployments, PVC templates, Secrets, service account, storage mounts, environment, and ports before applying. Roll back chart and image together.
Archives, provenance, and legal files
Release archives are named silo_<version>_<os>_<arch> and contain the executable, README, LICENSE, and NOTICE. The checksum manifest is silo_<version>_checksums.txt; each archive receives an SPDX JSON SBOM, and the checksum set is accompanied by a keyless Sigstore bundle. Static, CGO-disabled, kqueue-tagged binaries are published only for Linux, macOS, and Windows on amd64/arm64; formerly compile-checked but unshipped architectures are no longer release gates. RPM/DEB/APK are built for Linux amd64/arm64, with timestamp package versions such as YYYYMMDDHHMMSS.0.0 (RPM release 1) and a separate package checksum manifest. Normal builds no longer stamp the build host’s GOPATH/GOROOT, improving reproducibility and removing path leakage.
Unlike the upstream baseline Docker recipe, which downloaded and verified a prebuilt dl.min.io server, the Silo release image consumes the exact source-built, attested release archive at the selected tag. Image publication is a separate, explicitly dispatched workflow after a GitHub release; build success alone does not publish it.
CREDITS is regenerated from the modules actually linked into the server and guarded in CI. It is included in the OCI image; it is intentionally omitted from packages and archives because of its size. Upstream AGPL and copyright notices remain, alongside PGSTY’s modification notice.
Runtime and security behavior changes
These are user-visible changes even when they close a vulnerability. “Stricter” means a request, policy, token, configuration, or corrupted internal message that formerly succeeded or failed differently can now be rejected.
Authentication, IAM, and request identity
| Change | Final behavior | Who must act |
|---|---|---|
OIDC JWT verification (d24f449e0) |
The client secret is no longer a verification key. Only asymmetric JWKS algorithms RS256/384/512, ES256/384/512, RS3256/3384/3512, and ES3256/3384/3512 are accepted. HS256/384/512 tokens fail; unknown kid still triggers the established JWKS refresh/retry path |
An IdP signing Silo tokens with HMAC must migrate to an asymmetric JWKS key |
LDAP STS errors (3b950f8fa) |
Unknown user and bad password share one external InvalidParameterValue authentication failure. LDAP infrastructure failures remain server errors and are logged |
Clients must not distinguish account existence from response text |
LDAP STS rate limiting (18b712d49, 9e10f6d9a, f44110890, 5e40665ac) |
Per-source, per-node in-memory bucket: burst 10, refill one per 6 seconds, idle TTL 15 minutes. Only authentication failures consume tokens; success and infrastructure failures refund them. Exhaustion returns HTTP 429, ThrottlingException, Retry-After: 6 |
Proxies should configure MINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIES; it is separate from the general source-address setting |
| LDAP trusted-proxy source | For an allow-listed socket peer, clean X-Real-IP is preferred; otherwise XFF is walked right-to-left past trusted hops. RFC 7239 Forwarded is ignored. A proxy must overwrite X-Real-IP |
Review ingress header sanitation; the limiter is not a distributed account lockout |
| LDAP service-account lookup | “User DN not found” matching is case-insensitive, preserving the intended Admin no-such-user / login-name error classification across dependency message capitalization | Only brittle clients that depended on the accidental misclassification see a difference |
Bucket/object IAM boundary (97b7d2804) |
Twelve protected bucket-write actions no longer inherit an Allow from only arn:aws:s3:::bucket/*; the bare bucket ARN is required. Deny/NotResource and built-in * policies retain their semantics |
Add arn:aws:s3:::bucket to custom policies that legitimately perform protected bucket writes, or use the temporary global escape hatch MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on |
| Protected actions | Delete/ForceDelete bucket; put/delete bucket policy; put replication, lifecycle, object-lock, versioning, or CORS; delete CORS; put bucket QoS or inventory configuration | Read/list, create bucket, tags, encryption, and notifications keep the inherited matching behavior |
Effective policy inputs (2f55347f7) |
Server-derived conditions cannot be shadowed by a same-named header/query parameter. Exact keys win in the policy package. Existing/request tags, storage class, content/copy/checksum, object-lock, signature age, and list parameters are sourced from the value actually consumed by the operation | Policies that accidentally depended on attacker-controlled shadow values stop matching |
| Request tags | PutObject, CreateMultipartUpload, and PutObjectTagging bind s3:RequestObjectTag/* to the parsed input. ExistingObjectTag comes only from stored metadata. Other operation paths keep the inherited header fallback |
Re-test tag-conditioned write policies |
s3:signatureAge |
Present only for verified presigned SigV4 requests | Raw x-amz-signature-age injection no longer creates the condition |
s3:versionid (744a9dcd7) |
Absent means absent, whitespace is normalized, and DeleteObject/MultiDelete uses the effective per-object version. A URL version cannot decoy a different XML version | Re-test version-conditioned delete policies and any policy relying on Null |
Replication metadata (56fa63bfd) |
Ordinary PUT/COPY cannot inject internal replication status/time metadata. An authenticated replica request must have ReplicateObjectAction; multipart and Snowball replication flows retain their legitimate metadata |
Custom replication callers must use the authorized replication path |
The bucket-boundary escape hatch is startup-global and all-or-nothing. It exists for migration, not as a permanent mixed-policy mode. An empty or unparseable LDAP source is deliberately not placed into one shared limiter bucket; it is not throttled until a usable source can be derived.
General source-address trust
fe6dc4780 adds MINIO_API_TRUSTED_PROXIES, because the chosen client address feeds aws:SourceIp, audit remotehost, event Host, admin trace, and node-to-node forwarding.
| Value | Result |
|---|---|
| unset | Exact inherited behavior: trust source headers from any peer; left-most XFF, then X-Real-IP, then Forwarded |
none or off |
Ignore all three source-address headers and use the TCP peer |
| IP/CIDR list | Trust headers only when the TCP peer is listed; walk XFF right-to-left past trusted hops (maximum 100), then use the last X-Real-IP line, then walk Forwarded right-to-left |
Malformed entries, a non-empty list naming no proxy, or a remote env:// read failure fail closed and stop startup rather than reverting to trust-any. Invalid values inside a received chain are skipped; no usable address falls back to the peer. Loopback is implicitly trusted only for the FTP/SFTP peer bridge, not skipped as an arbitrary hop inside a client chain.
The inherited _MINIO_API_XFF_HEADER=off retains its exact old semantics and initialization timing: it disables only XFF parsing, not X-Real-IP or Forwarded, and therefore is not a security boundary. Configure every cluster peer that legitimately forwards requests and ensure edge proxies overwrite or strip client-provided source headers.
S3 request and response behavior
| Area | Change and compatibility effect |
|---|---|
| Presigned streaming auth | A query/presigned SigV4 request declaring STREAMING-UNSIGNED-PAYLOAD-TRAILER is rejected with SignatureVersionNotSupported; it cannot fall through to anonymous authorization. Header SigV4 is verified before body processing |
| Snowball | Authorization happens before tar extraction; the streaming-trailer bypass cannot write objects before a later failure |
| S3 Select record limits | CSV input, JSON Lines input, and output records over 1 MiB return an OverMaxRecordSize event. JSON Lines always uses the bounded reader (possibly slower on SIMD-capable CPUs), JSON parse errors use JSONParsingError, and already completed records can precede the terminal error |
| Streaming responses | The tracking writer implements Flush; Write/Flush records an implicit HTTP 200. ListenBucketNotification/watch streams and S3 Select keepalives reach clients, while audit/status metrics record the committed status correctly |
| Multipart full-object checksum | FULL_OBJECT CRC32/CRC32C/CRC64NVME completion may omit every per-part checksum. If any are supplied they are still checked; COMPOSITE still requires every part. A zero-byte multipart object’s checksum is retained correctly |
| Multipart part ordering | Duplicate or non-increasing part numbers fail with InvalidPartOrder before assembly. Gaps and a first part other than 1 remain legal; the upload stays available for retry |
| Erasure read pooling | Correct shard-buffer ownership is restored, avoiding loss of pooling and a wrong-buffer association that could cause hangs, corruption, or severe performance loss |
| Update buffers | Returned download buffers are owned correctly; the public updater was subsequently disabled, so no current supported upgrade path exercises this code |
Distributed storage and private APIs
These changes are normally invisible to an S3 client but are compatibility changes for mixed clusters, custom internal callers, corrupted disks, and adversarial peers.
ReadMultipleand its private storage-REST/rmplendpoint, client method, exported Go types, and metric were removed. External S3 List/Get operations are unchanged.storageRESTVersionremains v63, so do not infer mixed-node compatibility from the version number.- Every remote StorageAPI path field, nested metadata name, and raw-volume sink is validated at the storage boundary, including peer-S3 Grid messages. Lexical traversal, volume-root aliases, and Windows separator/drive forms are rejected.
- Erasure geometry, non-positive blocks, negative part sizes, and unusable stored erasure metadata are rejected at all decoded sinks and during CheckParts/VerifyFile.
- Internode allocation declarations are bounded: AppendFile caps preallocation at 1 MiB while still accepting the body; DeleteVersions grows as it decodes and rejects negative declarations; legacy ReadFile is capped at 5 GiB.
- Deadline-bounded work converts worker panics into errors and logs a bounded stack instead of taking down the process.
ReadPartskeeps the actual backend error across keepalive frames. An empty part list returns a successful empty result without a trace panic or leaked goroutine.- HTTP stream helpers left orphaned by
ReadMultiplewere deleted later; that cleanup creates no additional public behavior change.
This containment is lexical. It does not resolve filesystem symlinks, and native Windows server CI was not available for the audit. Upgrade every node together and keep untrusted clients away from the internode port even though root credentials and validation protect it.
Notification configuration and audit output
- NATS now registers parser-consumed
user_credentials,nkey_seed, andtls_handshake_first; AMQP registersimmediate. Existing environment variable names stay unchanged. - The old literal
MINIO_NOTIFY_NATS_USER_CREDENTIALSremains accepted for NATS only. Precedence is environment, new key, then old migration key. - AMQP legacy migration maps
immediatecorrectly. Invalid notification errors identify names but no longer echo credential values. - Generated PostgreSQL notification DSNs now quote/escape every libpq value and use the correct
userkeyword. An explicitconnection_stringis passed through unchanged. - Dangling-object deletion audit events once again include per-drive errors under
merrs.
One inherited migration risk is intentionally documented, not disguised as fixed: PostgreSQL/MySQL legacy notification migration can write host/port/user/password/database keys that are not registered by the current target schema, potentially disabling all targets on the next load; historical passwords may be stored in plaintext. Review legacy notification KV state before restarting into Silo.
Toolchain, dependencies, and embedded components
The build declaration moved from Go 1.24 plus a 1.24.8 toolchain to go 1.26.5. That can change TLS, HTTP, DNS, scheduler, garbage-collector, and standard-library edge behavior even where no Silo source line changed. Security-sensitive dependencies were also advanced, including Go-Jose, OpenTelemetry, Go crypto/network modules, cloud SDKs, etcd, NATS, and compression libraries.
Observable edge corrections inherited through those updates include Go TLS/X.509/URL/archive fixes; MQTT oversized UTF-8 packet encoding; malformed Azure NTLM challenge handling; Thrift framed transport and 32-bit compilation; NATS authentication, authorization, identity, and denial-of-service fixes; and Prometheus remote-read/write and UI hardening. They are dependency behavior changes, not a promise that every advisory path is reachable from Silo. The jsonparser CVE-2026-32285 investigation produced no patch: the resolved v1.1.2 already contained the fix and no vulnerable reachable symbol was found, so it creates no compatibility delta in this range.
Important deliberate dependency decisions are:
| Component | Final selection | Compatibility rationale / effect |
|---|---|---|
| Console | pgsty/silo-console v2.1.1 behind github.com/minio/console |
Restores the embedded UI, applies Silo branding and bilingual text, adds Metrics V3, removes SUBNET flows, and fixes untranslated metric legends |
| Client library | pgsty/mc behind github.com/minio/mc |
Keeps Console’s import path while consuming the maintained MCLI fork |
| Shared package | pgsty/silo-pkg/v3 v3.11.0 behind github.com/minio/pkg/v3 |
Supplies the IAM exact-match half, LDAP TLS/StartTLS/deadline/close fixes, certificate-watcher cleanup, and RNG fixes |
| Kafka | Sarama 1.45.1 | Pinned to avoid a breaking broker-negotiation drift |
| PostgreSQL | lib/pq 1.10.9 | Pinned to avoid a nil-[]byte / PostgreSQL-before-14 behavior regression; generated DSN quoting is fixed in server code |
| Compression | klauspost/compress 1.18.7 | Explicit security/correctness upgrade |
| Thrift | 0.24.0 | Fixes 32-bit builds |
| systemd library | require 22.7, replace with 22.6 | Retains NetBSD compilation until the monotonic-clock regression is fixed upstream |
The LDAP package now honors TLS fields for ldaps://, keeps StartTLS active even with server_insecure, avoids a nil-TLS panic, applies a StartTLS deadline, and closes a connection after failed StartTLS. Certificate-watcher shutdown no longer leaks; on Windows, polling can delay reload by up to about ten seconds. RNG subkey entropy/reset behavior is corrected, although the server does not exercise the reset path.
For external Go consumers of silo-pkg, two changes are broader than this server’s own call paths: xtime.Duration JSON moves from integer nanoseconds to duration strings, and some AIStor action vocabulary / protected-action helpers differ from upstream. In particular, Policy.IsAllowedActions can disagree on protected actions; the server does not call it. The server stores the relevant state through YAML/msgp and does not call the differing AIStor/action helper paths, so no server data migration or authorization delta was found from those library changes.
Generated String() files were regenerated under the new toolchain. Valid enum output remains the same; the diff is generator provenance and invalid-value formatting machinery, not a separately claimed S3 behavior change.
Known residual risks and non-fixes
This audit does not turn inherited limitations into claims of compatibility:
- Source IP is still forgeable by default. Unset
MINIO_API_TRUSTED_PROXIESdeliberately retains upstream trust-any behavior. Setnonefor direct deployments or an exact proxy allowlist for proxied deployments. - Some version-condition gaps remain. MultiDelete governance-bypass reauthorization still consults the query/absent version rather than each XML entry, and Snowball reads the PAX
minio.versionIdafter per-file authorization. Emptyusername,userid,signatureversion, andauthTypecondition keys are also still inserted, soNullon them has present-empty semantics. - Multipart parser defense is not complete. Handler-level ordering is fixed, but the object layer has no independent uniqueness defense; XML-root validation and the inherited nonnumeric-part error mapping were not changed.
- Legacy notification migration remains risky. Review it as described above.
- Storage path validation is lexical. Symlinks are not resolved; native Windows execution was not independently covered.
- Private APIs are not a stable compatibility promise.
ReadMultipleproves that a same-numbered storage REST protocol can still lose an operation. Do not run a rolling mixed build across this boundary. - A source result is not a released artifact. This page does not assert that GitHub tags, packages, OCI manifests, signatures, or the public site contain the three audit-head-only commits until each channel is verified separately.
- Informational HTTP responses remain imperfectly tracked. The response-tracking layer treats a 1xx response as final. The Flush/implicit-200 change did not introduce this behavior and does not claim to fix it.
Migration checklist
For a MinIO-to-Silo move, use this order:
- Record the exact MinIO binary/tag, chart and values, image digest, package payload, service unit, environment files, config directory, data ownership, IAM policies, OIDC/LDAP settings, notification targets, and proxy topology.
- Back up configuration and IAM metadata. Silo reads existing disks in place, but a rollback still needs the old executable/config/chart and unchanged data ownership available.
- Replace invocations of the server binary with
silo; do not assume/usr/bin/minioexists. In containers, argv-levelminio serveris translated, but the absolute path is not. - Decide the config directory explicitly. Reuse
--config-dir ~/.minioor let the legacy-only fallback select it; do not create an empty~/.siloaccidentally and then wonder why the old configuration is ignored. - For packages, grant
silo:siloaccess to data, certificates, secrets, and logs. Move deliberate overrides into/etc/default/silo; understand that it overrides/etc/default/minio. - For Helm, render the old and new charts with the complete old values, preserve names with
nameOverride/fullnameOverride/serviceAccount.namewhere required, and change chart plus image atomically. - Remove updater, callhome, SUBNET-registration, and support-upload automation. Replace it with package/image/orchestrator rollout and your own diagnostic transfer path.
- Change HMAC-signed OIDC tokens to asymmetric JWKS. Exercise success, bad-password, unknown-user, backend-failure, and rate-limit LDAP paths.
- Add bare bucket ARNs for the twelve protected actions. Test effective tag, signature-age, source-IP, and per-version delete conditions. Use the legacy bucket switch only as a temporary rollback lever.
- Set
MINIO_API_TRUSTED_PROXIES=noneor an exact allowlist, sanitize all three source-address headers, and include cluster peers that forward authenticated requests. - Test oversized S3 Select records, streaming notifications, unsigned-trailer rejection, multipart full-object checksums, duplicate parts, replication, healing, KMS, every notification target, audit ingestion, and graceful container shutdown.
- Upgrade all distributed nodes as one build. Keep the old chart and image paired for rollback; never roll back only one of them.
Verification evidence
The audit used the final source, not prose alone. At the recorded snapshot:
| Check | Result / boundary |
|---|---|
| Commit enumeration | 96/96 commits classified in the ledger below; origin/main accounted for 93 and the local prepared head for three more |
| Net diff review | All 523 changed paths classified across server runtime, internal protocol, dependency, delivery, documentation, tests, or superseded changes |
| Rebrand compatibility guard | Passed; compatibility manifest and delivery/runtime assertions unchanged, including Docker argv tests |
| Go test suite | Full go test ./... passed against 219670d31, including cmd, OIDC, LDAP, notify, event targets, Grid, handlers, hash, and all S3 Select packages |
| Helm migration | buildscripts/verify-helm-migration.sh passed: lint, render, legacy upgrade, archive, and identity comparison across seven rendered resources |
| Package lifecycle | buildscripts/package/lifecycle_test.sh passed; package payload/provenance assertions cover empty DEB conflict metadata, unit/default paths, and legal files |
| Site | Strict make check passed: module verification, warning-fatal Hugo render (617 EN / 615 ZH pages), and 388,962 internal references across 1,084 HTML files; git diff --check, bilingual anchors, and 96/96 commit coverage also passed |
Security articles contain deeper threat models and test vectors, but their historical “released/unreleased” labels describe their publication date. Where they conflict with the final snapshot, this page’s audited boundary is authoritative.
Complete commit coverage ledger
The hashes are in graph order. Merge, documentation, test, and CI commits are included because delivery behavior and the strength of a compatibility claim are themselves user-relevant; “no independent runtime delta” means exactly that, not “not reviewed.”
| Class | Commits | Verified net effect |
|---|---|---|
| Initial fork, Console, CI, dependency base | d4cd4b433, 8630937e7, 68521b37f, 00f3cf74f, 5abd9a80f, 377fc616d, f2f9a40dc, ee55e5391, ce1c537eb, 68e0ba997, 1869bd30b, ff58df949, e4fa06394 |
Go/SDK evolution; embedded Console restoration/fork; MCLI in OCI; replacement CI; LDAP TLS regression fix; security dependency upgrades. The two merge commits add no delta beyond their parents |
| April security series | d24f449e0, 3b950f8fa, 56fa63bfd, 3252d5b7f, f444b6f37, efb6e5b00, db4c0fd5e, 18b712d49, 9e10f6d9a, f44110890, f48dbe777 |
OIDC, LDAP STS, replication metadata, S3 Select, unsigned-trailer/Snowball, Go 1.26.2, limiter accounting/source hardening, and security documentation |
| May–June reliability and private API | 65795ee1f, 5e40665ac, fd69c89d0, 73ac52472, df627ff89, 3e61b1d3a, d495d30d5 |
HTTP Flush, final LDAP bucketing, full S3 Select bound, ReadMultiple removal, Go 1.26.4/dependency update, and documentation-link change |
| Pre-August component integration | ce01ccbdc, 4dfc27ce3, b7f52ca43, 7babc0c39, c1aec0518, 15fcc3c8a, 3f192f3f0 |
Historical chart image switch, security dependency upgrade, notification-stream merge, portable dependency pins, compression, MCLI replacement, Console v2.0 |
| August runtime correctness/security | c8590413f, 3e14733f1, 924717926, 89d346bf5, 8069a32ac, a36fd8fff, ca7baa670, 80e8eaa42, b6f70ab08, 1af351a70, 38366f654, 22c1e41fd, 97b7d2804, 2f55347f7, 744a9dcd7, fe6dc4780, 162ded343, 0c14d8151, 9dd1dc172, 2602177ef |
Multipart, erasure buffers, response commits, panic containment, path/metadata/allocation/ReadParts containment, orphan cleanup, IAM/effective values/version ID/source trust, notification/libpq, and audit details |
| Chart hardening and audit documentation | dfe669862, 5f4513fd4, b42ee4e8a, 8eae745ab |
Secure chart user default, portal/doc routing, maintainer ignore rules, and advisories; only the chart default changes runtime delivery behavior |
| Release engineering through the 20260804 tag | 9c799f42d, 10c7670b8, cf7df097b, 32863c852, 632ade111, 1814ae52f, 475236c79, 11d79fddc, 3b8a55dee, ca674a696, 4c185d5a6, 2ca4971d9, e064b5555, aa5139369, 021110b45, d88f46cce |
RPM/DEB/APK, provenance, OCI publishing gates, pinned lint/generation, S3 Select test-race fix, broad CI, PID-1 signal fix, published-target cross-builds, safe release dispatch, reproducibility, stale-config removal, systemd location, honest gates, and runtime-image shutdown assertion |
| Silo cutover and 2026-08-06 prepared head | 15def34dc, 77bdc4c0c, 15ab10833, 30749911b, e071bb77e, bd8df5166, 6613c2a3c, fd2ca1c6d, c46b16ec6, c47733abc, f1c77d5a2, 62717d7bf, 6740e6978, b57275be3, 05be686b8, a6d6d9b02, 6bd9cf77e, 219670d31 |
Removes unpublished MinIO delivery residue; Silo runtime identity/offline boundary; renamed packages, OCI and Helm with migration guards; pinned fixtures; docs/repository cutover; Console 2.1.0 then 2.1.1; Node 24 actions; DCO/legal/docs polish; regenerated CREDITS; LICENSE/NOTICE delivery |
The ledger totals 96 unique commits. Changes replaced inside the range—such as Console 2.0 → 2.1.0 → 2.1.1, the historical pgsty/minio image/chart state, and updater-buffer code after the updater was disabled—are described only where they leave a final compatibility consequence.
See also
- MCLI — client artifact, config-directory, update, SUBNET, and command compatibility
- Silo 20260804 release notes — the earlier tagged release boundary, not the full 2026-08-06 audit head
- Silo Pkg 3.11.0 — shared IAM, LDAP, watcher, RNG, and developer-facing changes
- OIDC JWT hardening, LDAP STS hardening, replication metadata, S3 Select limits, and ReadMultiple removal
- Unsigned-trailer query auth, Snowball authentication, and the jsonparser no-change finding
- Internode path containment, duplicate multipart parts, bucket/object IAM boundary, version-ID conditions, source-address trust, and notification key registration
3 - MCLI Client Compatibility Notes
mcli is Silo’s build of the MinIO Client (mc). This page records where the two are interchangeable and where they differ.
pgsty/mc forked from the upstream minio/mc at its final commit, 77f82e18 (2025-11-06). The upstream repository was archived in July 2026 without ever cutting a release that contains that commit — so every mcli release is strictly newer than any official mc binary ever published. Fork releases to date: 20260313, 20260321, 20260417, 20260804, and 20260806.
Principles
The fork follows one rule: the shipped artifact and its channels are renamed; the tool you use is not.
- Renamed / replaced — the artifact name on disk (
mcli), the product identity in--versionand--help, the distribution channels (GitHubpgsty/mc, the Pigsty repository,docker.io/pgsty/mc), and the signing keys. Not the command syntax, and — depending on how you install it — not even the name you type. - Unchanged — every command, subcommand, and flag; S3 and admin API behavior, request signing, and protocol headers (
x-minio-*); JSON output schemas; exit codes of normal operations; the configuration file format and alias semantics;MC_*environment variables (includingMC_HOST_<alias>); the.part.minioresume suffix; and the Go module pathgithub.com/minio/mc. - Severed — every connection to MinIO-operated services: the release/update feed, the SUBNET support and licensing portal, telemetry, and the pre-seeded
playdemo alias. Affected commands remain in the CLI for script compatibility and fail with a stable error rather than disappearing. - Preserved — upstream copyright and the AGPL-3.0 license. Runtime output credits both MinIO, Inc. and PGSTY.
A configuration written by upstream mc is readable by mcli unchanged, and vice versa; both clients can talk to MinIO servers, Silo servers, and any other S3-compatible endpoint.
What changed
Ordered by how likely each change is to affect you, most likely first.
1. The name — what you type, and where the config lives
For many users nothing changes here: the container image keeps mc as its entrypoint, and a binary installed under the name mc behaves identically to upstream. What changed is what we ship — archives and Linux packages install the binary as /usr/local/bin/mcli (package name mcli).
Neither name is hardcoded anywhere. Since 2016 the upstream client has derived its runtime identity from the name it is invoked as, and mcli is the exact rename upstream’s own CONFLICT.md recommended (issue #873) for the Midnight Commander clash — this fork merely promoted that suggestion to the official shipping name, with zero code changes. What that mechanism means in practice:
| Follows the invoked name | Fixed, regardless of the name |
|---|---|
Configuration directory: ~/.mc vs ~/.mcli (Windows: %USERPROFILE%\mc\ vs …\mcli\) |
Environment variables: always MC_* — there is no MCLI_CONFIG_DIR |
| Program name shown in help and usage text | config.json format — identical and interchangeable in both directions |
| Shell-completion registration | All commands, flags, JSON output, exit codes |
User-Agent application suffix (mc/… vs mcli/…) |
--config-dir and MC_CONFIG_DIR overrides |
The one real trap: run mcli for the first time and your existing mc aliases are not there — it starts from an empty ~/.mcli. Either keep invoking it as mc (a symlink suffices — argv[0] is what counts), or copy the state once with cp -a ~/.mc ~/.mcli. For automation and configuration templates, set MC_CONFIG_DIR explicitly: the environment prefix does not follow the name, so one template serves both. Details in Migration.
Get it from GitHub Releases (SHA-256 mcli_<version>_checksums.txt), the Pigsty repository (RPMs GPG-signed, key fingerprint 9592A7BC7A682E7333376E09E7935D8DB9BD8B20), or docker.io/pgsty/mc. Upstream’s minisign key does not sign these artifacts, and dl.min.io is never contacted. Release tags (RELEASE.YYYY-MM-DDTHH-MM-SSZ) and package versions (YYYYMMDDHHMMSS.0.0) keep their upstream schemes.
2. mcli update always fails — on purpose
Self-update is removed. mcli update never contacts the network and never replaces its binary; it prints an explicit notice and always exits 1. Upstream mc update exited 0 when already current, so any cron job or script that calls it and treats a non-zero exit as failure will start failing — drop the call and upgrade through your package manager or GitHub Releases instead. The per-invocation version probe against upstream release feeds is also gone, and MC_UPDATE / MINIO_UPDATE are no longer consulted.
(mcli admin update ALIAS — updating the server — still exists, but Silo servers reject in-place updates server-side.)
3. SUBNET, licensing, and telemetry commands
Everything that reached MinIO SUBNET is disabled at build time. Affected commands keep their names and flags, print a stable notice — “MinIO SUBNET services (registration, licensing, uploads) are disabled in this Silo build of mc; diagnostics remain available locally.” — and exit 1:
| Command | Behavior now | Use instead |
|---|---|---|
mcli license register |
notice, exit 1 |
— |
mcli license update ALIAS (online renewal) |
notice, exit 1 |
mcli license update ALIAS license.key (offline, still works) |
mcli support upload |
notice, exit 1 |
share files through your own channels |
mcli support proxy set |
notice, exit 1 |
proxy remove still clears a legacy setting |
mcli support callhome enable |
notice, exit 1 |
disable / status still work |
The diagnostics themselves stay: mcli support diag / perf / profile / inspect always run in local (airgap) mode — results are written to local files, nothing is uploaded, and SUBNET registration is no longer a prerequisite. Two related hardening changes: inspect no longer falls back to encrypting output with an embedded MinIO public key (your archives stay decryptable by you), and since 20260804 --debug output redacts SUBNET credentials — if you ever shared debug logs from older builds, rotate the keys in them. mcli license info and unregister work locally.
4. The play demo alias is no longer pre-seeded
Fresh configurations seed local, s3, and gcs — not play. Tutorials and smoke scripts that assume the demo alias need it added explicitly: mcli alias set play https://play.min.io <access-key> <secret-key> restores the old behavior, since nothing blocks deliberate access to any S3 endpoint. Existing configuration files are never modified.
5. Output text carries the Silo identity
mcli --version keeps its machine-readable first line and adds an identity line plus dual copyright; --help says “Silo client” and examples use mysilo. Command syntax is untouched — only scripts that grep for upstream identity strings (e.g. “MinIO Client”) need adjusting.
6. For developers
The module path stays github.com/minio/mc, so imports compile unchanged — but go install github.com/minio/mc@latest installs the archived upstream, not this fork. Build from source (git clone https://github.com/pgsty/mc && cd mc && make) or consume it via a replace directive. Contributions need no CLA but require a DCO sign-off (git commit -s). Upstream being archived also means inherited defects are only ever fixed here — most notably minio/mc#5139 (mirror --remove --watch on versioned buckets).
Migration
Moving from an official mc binary to mcli:
- Install
mclifrom one of the fork’s channels (see §1 for verification): GitHub Releases archive,yum install mcli/apt install mclifrom the Pigsty repository, ordocker pull pgsty/mc. - Decide what to call it — this determines which configuration it reads:
- Keep the
mcname (least friction): after confirming no upstream binary remains (command -v mc), install it asmc— e.g.ln -s /usr/local/bin/mcli /usr/local/bin/mc. Invoked asmc, it reads your existing~/.mcuntouched; nothing else to migrate. - Adopt the
mcliname: carry your state over once withcp -a ~/.mc ~/.mcli, or setMC_CONFIG_DIR=~/.mc. Both clients can also coexist side by side, each with its own directory.
- Keep the
- Clean up automation:
- remove
mc updatecalls — they now always exit1; - remove
license register,support upload,support callhome enable, andsupport proxy set— same stable failure; support diag/perf/profile/inspectkeep working and write local files; drop any step that expected a SUBNET upload;- review anything that greps
--versionoutput beyond the first line.
- remove
- Re-check
playusage in tutorials and smoke scripts (§4). - Verify:
mcli --version,mcli alias ls, thenmcli ls <alias>andmcli ping <alias>against your servers. - Rollback stays trivial: the configuration format is identical in both directions, so keeping the old
mcbinary around lets you switch back at any time.
See also
- Silo vs. MinIO — how the
siloserver compares tominio
4 - Native Package Migration
Silo publishes silo packages for RPM, DEB, and APK on amd64/arm64 via GitHub Releases, with SHA-256 sums and build-provenance attestations. This page records what changes relative to a minio package installation: the file layout, the service account, and the caveats. General migration scope is in the migration guide.
File layout
| MinIO installation | Silo package |
|---|---|
/usr/bin/minio |
/usr/bin/silo (also provides silo healthcheck) |
minio.service |
/usr/lib/systemd/system/silo.service |
/etc/default/minio |
Still read, first; /etc/default/silo overrides per variable (noreplace/conffile — upgrades never overwrite edits) |
service account minio-user (upstream) / minio (Pigsty) |
silo:silo, declared in /usr/lib/sysusers.d/silo.conf, created on install |
| — | /usr/share/doc/silo/LICENSE, NOTICE (AGPL-3.0-or-later) |
Two package properties:
- Installation never starts or enables the service;
postinstallonly creates thesiloaccount and reloads systemd. - The package installs alongside the
miniopackage — no file conflicts, so the old package stays available for rollback.
Service account
The unit defaults to User=silo, but existing data, TLS keys, and KMS credentials belong to the old MinIO user. Do not chown the data. Run Silo as the current owner via a drop-in:
This also keeps TLS working: Silo resolves certificates from the runtime user’s home (~/.silo/certs, falling back to the legacy ~/.minio/certs), so the existing public.crt/private.key/CAs/ are found without copying. Without the drop-in, a TLS deployment fails to start:
Adopting the silo account is an optional later change: move the certificates to a silo-readable path, set --certs-dir in MINIO_OPTS, and transfer data ownership outside the migration window.
Takeover and rollback
The unit is a takeover unit:
Conflicts=minio.service: systemd never runs both; starting one stops the other. This implements takeover and rollback in both directions.- The
EnvironmentFilechain meansMINIO_VOLUMES,MINIO_OPTS, credentials, and KMS settings from/etc/default/minioapply unchanged. Type=notify:systemctl startreturns success only after the server is actually ready.
Switch over:
Roll back (nothing to restore — data ownership, certificates, and the old unit were never touched):
Caveats
- Clusters switch all nodes together. Two different binaries do not form a cluster — MinIO next to Silo, or one Silo version next to another; a mixed node waits indefinitely in
activating(details). Prepare every node first (install package, create drop-in), then flip all nodes in quick succession:systemctl disable --now minio && systemctl enable --now --no-block silo. Rollback and later upgrades likewise: all nodes together. - Non-packaged installations work the same way. A
/usr/local/bin/miniowith a custom unit is taken over identically, as long as its configuration lives in/etc/default/minio. - Crash loops rate-limit. A misconfigured start (for example, missing certificates) repeats under
Restart=alwaysuntil systemd’s start limit trips (Start request repeated too quickly). Fix the cause, thensystemctl reset-failed silo && systemctl start silo. - Keep the rollback window. Leave the
miniopackage, unit, and binary installed until validation completes; a disabled unit costs nothing. Remove the old package afterwards if desired. - Rolling restarts after migration: gate each with
silo healthcheck --maintenance cluster; exit0means stopping this node keeps write quorum, HTTP412means it does not.
5 - Feature Notes
The component pages in this section record where Silo matches MinIO. This subsection records the places where Silo deliberately goes beyond it: each page is a design note, written before the implementation lands, and kept afterwards as the authoritative record of what was decided and why.
A note here describes intent, not necessarily shipped behavior — every page carries a status line saying which it is.
5.1 - Native Health Checks and the Distroless Image
Status: P1 (subcommand,
2ff594f4b) and P2 (distroless image + CI gate,4c34d2309) implemented inpgsty/silo; P3 (Helm probes) and P4 (docs) pending · Decided: 2026-08-06 · Owner:pgsty/silo(command, images, Helm chart), this site (docs)
Silo is getting a native silo healthcheck subcommand and, alongside the existing container image, a new distroless image variant that contains exactly one file that matters: the silo binary. This note records the reasoning and the design decisions before implementation, so the code has a specification to be checked against — and so that “why is it built this way?” has a permanent answer.
Background
Today’s release image (docker.io/pgsty/silo) is built on ubi-micro and ships four moving parts: the silo server, the mcli client (with an mc alias), a statically linked curl, and a POSIX-shell entrypoint script. The compose examples check container health with the bundled client:
That arrangement is inherited from upstream MinIO, and its fragility is a matter of record: when mc was briefly missing from the image, users found their health checks failing “with no option but to disable” (#9). Upstream’s own history rhymes — when MinIO moved to ubi-micro in 2023 and lost curl, the maintainers’ answer was to lean harder on mc ready local (minio/minio#18373, #18389), and upstream minio/minio has since been archived with server as the only subcommand its binary ever had. Nobody upstream is going to fix this.
A distroless image forces the question. There is no shell, no curl, no mc — by design. The only program guaranteed to exist inside the container is the server binary itself. If Docker-level health checking is to exist at all in that image, the binary has to provide it.
Why mc ready had to be retired as a probe
Reading the actual mc implementation (cmd/ready-main.go) shows the current health check works by accident, not by design. Four independent defects:
- It never fails on its own.
mc readyis a wait-until-ready loop: it retries every 5 seconds forever and only ever exits zero, on success. Connection refused does not break the loop. As a Docker healthcheck, the “unhealthy” verdict is produced entirely by Docker’stimeoutkilling the process — the probe semantics are a side effect of SIGKILL. - It checks the wrong scope.
mc readyrequests/minio/health/cluster— cluster-wide write quorum. Every container’s “health” therefore reflects the state of the whole cluster, which is precisely the cascading-failure anti-pattern the Kubernetes documentation warns about: lose quorum, and every node is marked unhealthy simultaneously. - It has hidden failure modes. It requires a writable
~/.mcconfig directory (on a read-only rootfs or an arbitrary OpenShift UID, the probe fails while the server is perfectly healthy), it prints config-creation noise on first run, and its built-inlocalalias is hardcoded tohttp://localhost:9000— wrong the moment TLS is enabled or the port changes, a limitation users complained about upstream. - It is the last functional reason to bundle a second binary. Both
mcliand the pinned staticcurlcarry ongoing supply-chain and maintenance cost (the curl pin is stuck on v8.11.0 because a later release dropped the aarch64 build) for what a subcommand of the existing binary can do in ~150 lines.
The decisions
Three tracks, deliberately decoupled:
| # | Decision |
|---|---|
| D1 | The silo binary gains a healthcheck subcommand — a thin, anonymous HTTP client for the server’s existing /minio/health/* endpoints. It ships in every build, so every image and bare-metal install gains the capability. |
| D2 | The existing image does not change. It keeps mcli, curl, the shell entrypoint, and the mc ready local examples. Users of the current image who want the new probe can opt in by overriding their healthcheck.test — nothing is taken away and no default behavior moves. |
| D3 | A new distroless variant is published alongside it, as a pilot: single binary, no shell, native HEALTHCHECK baked in. If the pilot proves out, it becomes the recommended default later and the switch completes; the classic image remains for compatibility either way. |
D2 and D3 answer the obvious “why not just slim the main image?” — because the main image’s contents are a compatibility surface. #9 exists because that surface was changed underneath people once before. The distroless image is a new name with a new contract, so nobody’s existing healthcheck, docker exec mc habit, or entrypoint assumption breaks while it is evaluated.
The silo healthcheck command
The design principles behind that shape:
- Thin client, single source of truth. The command is only ever an HTTP client of the canonical health API. It never re-implements a check in-process, so the semantics of “healthy” live in exactly one place: the server handlers.
- CLI vocabulary = API vocabulary. The check names are the endpoint paths. No new concepts to learn, nothing to keep in sync.
- Share the server’s own configuration. The port comes from the same
--address/MINIO_ADDRESScontract the server uses, and http-vs-https is decided by the same certificate check the server itself performs at startup (public.crt+private.keyin the certs directory). This is the Traefikhealthcheckpattern — the closest prior art, which resolves its ping endpoint from the same static configuration as its server — with the TLS handling Traefik left as a// TODOactually implemented. It is also the direct fix formc ready’s port-guessing defect. - The default check is node-local.
liveanswers “is this process serving,” which is the only question a per-container health status should answer. Cluster-scope checks exist, but only behind explicit arguments, mirroringmc ready’s--cluster-read/--maintenanceso the operational vocabulary carries over. - Exit codes are 0 and 1, nothing else. The Dockerfile reference explicitly reserves exit code 2 (
vault status, which uses 2 for “sealed”, is the cautionary tale). Rich diagnostics belong in the single output line instead — Docker stores the first 4096 bytes of probe output indocker inspect, and the command decodes the server’s diagnostic headers (x-minio-server-status,x-minio-write-quorum,x-minio-healing-drives) into it, which is exactly the detail a barecurl -fthrows away. - Skip TLS certificate verification, with no opt-out in v1. This is a loopback self-probe of an anonymous endpoint carrying no data — and the kubelet’s documented behavior for HTTPS
httpGetprobes is precisely the same. Matching it means one TLS deployment produces one verdict across Docker and Kubernetes; verifying by default would only manufacture false negatives, since self-signed server certs rarely carry a127.0.0.1SAN.
Two implementation constraints, discovered in the source, that are load-bearing rather than stylistic:
- The request must be strictly anonymous. The health routes are exempted from the reserved-path guard only for requests the server classifies as anonymous; attaching an
Authorizationheader reclassifies the request and gets it rejected (ErrAllAccessDisabled) instead of answered. - The HTTP transport must set
Proxy: nil. Containers routinely inheritHTTP_PROXYwithout aNO_PROXYentry for127.0.0.1; a loopback probe must never route through a corporate proxy. (Traefik’s healthcheck does this deliberately, for the same reason.)
And one number that looks arbitrary but is not: the 15-second default timeout for cluster checks exists because the server evaluates cluster health under its own 10-second cluster_deadline — a client that gives up at 5s abandons the request before the server delivers its considered 503, losing every diagnostic header with it. Two details were added after adversarial review: --url is environment-backed (MINIO_HEALTHCHECK_URL) because a probe process cannot see the server’s command line — it is the documented way to point a baked-in HEALTHCHECK at a server whose address or TLS setup comes from CLI arguments; and any outer (Docker) timeout must exceed the probe’s own deadline, or the probe is SIGKILLed before it can print its diagnostic line.
What the endpoints really do
The table below is verified against the handler source, not quoted from documentation — and it corrects a common misreading:
| Endpoint | Returns 200 when… | Fails with… | Notes |
|---|---|---|---|
/minio/health/live |
almost always — even before the object layer is initialized (that state is only signaled via the x-minio-server-status: offline header) |
503 when the request queue is saturated | touches no external system; the only endpoint quiet enough for high-frequency probing |
/minio/health/ready |
as live, plus KMS can generate a key and etcd answers a read — each only if configured |
KMS/etcd failure; queue saturation | without KMS or etcd, ready and live are the same code path |
/minio/health/cluster |
object layer, bucket metadata and IAM are initialized, and every erasure set has write quorum | 503 with quorum diagnostic headers; with ?maintenance=true, failure is 412 |
each failed evaluation writes a server-side log line — do not poll it tightly |
/minio/health/cluster/read |
the read-quorum version of the above | as above |
Consequences worth spelling out: live and ready are liveness-grade signals — they do not tell you the node can serve objects; only the cluster pair does. That is exactly why the cluster pair must stay out of per-container probes (scope, log noise, cascading restarts) and why it is the right tool for operational questions like “may I take this node down?” (--maintenance, where 200 means safe and 412 means you would lose HA).
The distroless variant
Base: gcr.io/distroless/static-debian12 — which is sufficient because silo builds with CGO_ENABLED=0. The base ships the four things the server actually needs from a rootfs: CA certificates (for KMS/webhook/STS egress), tzdata, /tmp, and an /etc/passwd with root/nonroot entries. It ships no shell, no package manager, no libc.
Sketch of the contract:
The decisions folded into that sketch:
ENTRYPOINTis the binary itself.docker run pgsty/silo:distroless server /data— no argv-translation script, because there is no shell to run one. The classic image’sMINIO_USERNAME/MINIO_GROUPNAMEprivilege-drop path (which needs GNUchrootand a writable/etc/passwd) is not supported in this variant; the supported mechanism is--user/ KubernetesrunAsUser./datais created in the layer, mode 0777, and the default user stays root for the pilot. Issue #55 demonstrated that declaringVOLUME ["/data"]without creating it breaks every non-root invocation, and that no entrypoint can repair it after the fact — in distroless there is no entrypoint at all. Creating it world-writable in the layer is the one option that makes all privilege modes work (--userincluded), keeps drop-in parity with the classic image, and its exposure is bounded by the image running a single process. A nonroot-by-default posture (uid 65532) was considered and deferred: it would break the documented bind-mount workflow on UID mismatch, and the pilot’s job is to measure friction, not maximize it. Revisit at promotion time, possibly as a-nonroottag.- The health check is baked in, exec-form. Shell-form
HEALTHCHECKstrings need/bin/shand are impossible here; the JSON-array form is mandatory. Compose inherits an image’sHEALTHCHECKautomatically (withdisable: trueas the escape hatch), so compose users of this variant get workingdepends_on: condition: service_healthywith zero configuration.readyrather thanlivebecause Docker’s health status feeds gating (start ordering), which is readiness semantics — and the two are identical anyway unless KMS/etcd are configured. - One release-blocking verification:
HEALTHCHECKis a Docker extension, absent from the OCI image spec (opencontainers/image-spec#749 is still open), and OCI-media-type builds drop it silently. The publish pipeline must assertdocker inspectshows theHealthconfig on the pushed manifest, or adjust the build’s media types until it does. - Naming:
docker.io/pgsty/silo:<RELEASE>-distroless, plus a rollingdistrolesstag. A newDockerfile.distrolessin the server repo — which, having no download stages, is fully offline and can therefore be built and asserted in CI on every release, closing the “the gate tests a synthetic image, not the shipped one” coverage gap that #55 documented for the classic Dockerfile.
What a user gives up in the variant, stated honestly in its docs: no docker exec <container> sh debugging (use docker debug / kubectl debug ephemeral containers), no in-image mc (use the pgsty/mc image or a host-installed mcli), no MINIO_USERNAME path (use --user).
Kubernetes needs no image support at all
Worth stating explicitly, because it bounds the problem: Kubernetes ignores Dockerfile HEALTHCHECK entirely — kubelet probes are configured in the pod spec and executed from outside the container as httpGet requests. Both image variants are therefore probed identically:
Three cautions that belong next to any such config: the cluster endpoints must never appear in probes (liveness would restart the whole fleet on quorum loss; readiness would tangle with bootstrap — the chart’s headless service correctly sets publishNotReadyAddresses: true for exactly that reason); live deliberately returns 503 under sustained request-queue saturation, so a saturated node restarts after ~90s by design; and with scheme: HTTPS the kubelet skips certificate verification, so self-signed deployments need nothing extra.
The Silo Helm chart currently ships no probes at all (neither does upstream’s, despite its docs). Adding the three probes above to the chart is planned as an independent follow-up — it depends on neither image track, and it is a differentiator over upstream rather than a compatibility risk.
Rollout
| Phase | Scope | Repo |
|---|---|---|
| P1 | silo healthcheck subcommand + tests; ships in the next release binary (all images inherit the capability, no image behavior changes) |
pgsty/silo |
| P2 | Dockerfile.distroless + CI build/health gate + publish -distroless tags as a pilot |
pgsty/silo |
| P3 | Helm chart: add the three probes; refresh the stale default image tag | pgsty/silo |
| P4 | Docs: command reference, probe guide, distroless migration notes; pilot feedback → decide on promoting distroless to the recommended default | this site |
Compatibility promises across all phases: the classic image’s contents and examples do not change; mc ready local keeps working everywhere it works today; the health HTTP API is untouched (the subcommand is purely additive); and the /minio/health/* paths remain frozen as compatibility surface, same as every other /minio/* route in the fork.
Deferred decisions
Recorded so they are not re-litigated from scratch:
--waitmode (block-until-healthy, the one thingmc ready’s loop is genuinely for): deferred — no in-repo consumer needs it yet, and adding it later is backward-compatible. The flag namespace is reserved.- Nonroot-by-default for the distroless image: deferred to promotion time, as above.
- JSON output schema for
--json: follows the global-flag convention; the exact schema is fixed at implementation time and documented in the command reference.