Use the keyring vault component¶

The keyring_vault component extends the server capabilities and provides an interface for the database with a HashiCorp Vault server to store key and secure encryption keys.

Percona Server relies on a keyring to safeguard the master keys that encrypt data at rest. A manifest file loads the keyring component, which reads a separate configuration file during initialization. Avoid --early-plugin-load and INSTALL COMPONENT: neither mechanism can load a keyring early enough in startup.

Why the manifest is the only supported load path¶

The keyring must be live before InnoDB opens an encrypted page, which rules out any mechanism that depends on a running SQL layer. A typical startup proceeds in this order:

mysqld parses startup configuration and reads the manifest file next to the binary.
The server loads components named in the manifest.
InnoDB initializes, replays the redo log, and opens tablespaces.
The SQL layer accepts connections.

The keyring must be ready between steps 1 and 3. Both alternative mechanisms miss that window:

INSTALL COMPONENT runs as SQL, so the statement cannot execute until step 4. The registration lives in mysql.component, an InnoDB table the server reads only after InnoDB initializes — a circular dependency when the system tablespace is encrypted. Crash recovery also runs before SQL, so an encrypted redo log must be readable without any SQL layer.
--early-plugin-load applies to legacy keyring plugins, not components. Plugins and components load through separate subsystems; the flag cannot locate component entry points. The manifest is the only early-load channel for components.

One practical consequence: a component registered through INSTALL COMPONENT on a running server disappears on the next restart, so InnoDB fails to unwrap tablespace keys without a manifest file on disk. A missing or malformed mysqld.my is therefore a startup failure for any instance with encrypted tablespaces.

Place a global manifest named mysqld.my in the server installation directory. For per-instance overrides, add a local manifest — also named mysqld.my — in the data directory.

To install a keyring component:

Write the manifest file in valid JSON.
Write the component’s configuration file in valid JSON.

The manifest names the component to load. Without a matching manifest file, the server quietly skips the component. On startup, the server reads the global manifest from the installation directory; the global manifest either holds the component entries directly or delegates to a local manifest in the data directory. When instances on the same host require different keyring components, place a local manifest in each data directory so every instance loads the correct component.

Warning

Run exactly one keyring per server instance. Percona Server does not support multiple keyring plugins, multiple keyring components, or any mix of plugin and component — such configurations risk data loss.

The following example is a global manifest file that does not use local manifests:

{
 "read_local_manifest": false,
 "components": "file://component_keyring_vault"
}

The following is an example of a global manifest file that points to a local manifest file:

{
 "read_local_manifest": true
}

The following is an example of a local manifest file:

{
 "components": "file://component_keyring_vault"
}

The configuration settings are either in a global configuration file or a local configuration file.

The component communicates with the Hashicorp Vault server . Prepare the certificate and key files for a secure HTTPS connection to the server. You must have an organizational Certificate Authority (CA), a private vault key, and certificate for the Hashicorp Vault server instance.

You can use OpenSSL to generate these files or use existing files. The key files contain sensitive information. Store these key files and the password used to create each key in a secure location.

You can use the Hashicorp Vault to build your own CA , if needed, and then create a Hashicorp Vault server certificate.

The component_keyring_vault.cnf file contains the following information:

read_local_config [optional] - this option can be used only in the global configuration file. This option indicates whether the component should read configuration information from the local configuration file. The allowed values are true or false. If you do not use this option, the component uses only the global configuration file.

If you use the read_local_config option in the global configuration file along with other items, the component checks the read_local_config item value first:

false - the component processes other items in the global configuration file and ignores the local configuration file.

true - the component ignores other items in the global configuration file and attempts to read the local configuration file.
timeout - the duration in seconds for the Vault server connection timeout. The default value is 15. The allowed range is from 0 to 86400. The timeout can be also disabled to wait an infinite amount of time by setting this variable to 0.
vault_url - the Vault server address.
secret_mount_point - the mount point name where the keyring_vault stores the keys.
token - a token generated by the Vault server.
secret_mount_point_version [optional] - the KV Secrets Engine version (kv or kv-v2) used. The allowed values are AUTO, 1, and 2. The default value is AUTO.
vault_ca [optional] - if the machine does not trust the Vault’s CA certificate, this variable points to the CA certificate used to sign the Vault’s certificates.

Example of a configuration file in JSON format

{
 "timeout": 15,
 "vault_url": "https://vault.public.com:8202",
 "secret_mount_point": "secret",
 "secret_mount_point_version": "AUTO",
 "token": "{randomly-generated-alphanumeric-string}",
 "vault_ca": "/data/keyring_vault_confs/vault_ca.crt"
}

Warning

Each secret_mount_point must be used by only one server. The behavior is unpredictable if multiple servers use the same secret_mount_point.

The first time a key is fetched from a keyring, the keyring_vault communicates with the Vault server to retrieve the key type and data.

secret_mount_point_version information¶

The secret_mount_point_version can be either a 1, 2, AUTO, or the secret_mount_point_version parameter is not listed in the configuration file.

Value	Description
1	Works with `KV Secrets Engine - Version 1 (kv)`. When forming key operation URLs, the `secret_mount_point` is always used without any transformations. For example, to return a key named `skey`, the URL is /v1//skey
2	Works with `KV Secrets Engine - Version 2 (kv)` The initialization logic splits the `secret_mount_point` parameter into two parts: The `mount_point_path` - the mount path under which the Vault Server secret was created The `directory_path` - a virtual directory suffix that can be used to create virtual namespaces with the same real mount point For example, both the `mount_point_path` and the `directory_path` are needed to form key access URLs: /v1/<mount_point_path/data//skey
AUTO	An autodetection mechanism probes and determines if the secrets engine version is `kv` or `kv-v2` and based on the outcome will either use the `secret_mount_point` as is, or split the `secret_mount_point` into two parts.
Not listed	If the `secret_mount_point_version` is not listed in the configuration file, the behavior is the same as `AUTO`.

If you set the secret_mount_point_version to 2 but the path pointed by secret_mount_point is based on KV Secrets Engine - Version 1 (kv), an error is reported, and the component fails to initialize.

If you set the secret_mount_point_version to 1 but the path pointed by secret_mount_point is based on KV Secrets Engine - Version 2 (kv-v2), the component initialization succeeds but any server keyring-related operations fail.

Upgrade from Vault Secrets Engine Version 1 to Version 2¶

You can upgrade from the Vault Secrets Engine Version 1 to Version 2.

Use either of the following methods:

Set the secret_mount_point_version to AUTO or the variable is not set in the keyring_vault component configuration files in all Percona Servers. The AUTO value ensures the autodetection mechanism is invoked during the component initialization.
Set the secret_mount_point_version to 2 to ensure that components do not initialize unless the kv to kv-v2 upgrade completes.

Note

The keyring_vault component that works with kv-v2 secret engines does not use the built-in key versioning capabilities. The keyring key versions are encoded into key names.

Maintain the Vault connection¶

Installation is only the first step. In production, the most common cause of keyring_vault failures is not a bad configuration but a lost connection to Vault — either because the token the component uses has expired, or because the Vault server itself has sealed. Both leave Percona Server unable to fetch master keys.

The keyring_vault component reads the token value from the configuration file at startup and uses the stored token for the lifetime of the running server process. The component does not renew the token automatically and does not reread the configuration file while the server is running. Any renewal or rotation must happen outside MySQL, on the Vault side or through a helper process.

Token expiration and renewal¶

Every Vault token has a TTL. When the TTL elapses, the token is revoked and any API call made with the revoked token — including the keyring_vault component’s fetch of a master key — fails with a 403 from Vault.

Symptoms of an expired or revoked token include:

Startup failure of mysqld after a restart, with keyring component errors about initialization or key fetch.
Failure of ALTER INSTANCE ROTATE INNODB MASTER KEY.
Failure to open an encrypted table after a restart, returning a keyring error.
A row in performance_schema.keyring_component_status showing the component loaded but unable to reach Vault.

Because the component does not renew tokens, pick one of the following patterns so the token the component reads on startup is always valid:

Option 1 (recommended): Periodic tokens¶

A periodic token has no maximum TTL. As long as the token is renewed within each period, the token continues to work indefinitely. This renewal model matches how a long-running database server uses Vault.

Create a role that issues periodic tokens and use the role to mint the token placed in component_keyring_vault.cnf:

vault write auth/token/roles/percona-keyring \
    allowed_policies="percona-keyring-policy" \
    period="24h" \
    renewable=true

vault token create -role=percona-keyring -format=json

Store the resulting token in the token field of your keyring_vault configuration file. Pair the periodic token with a renewer (see Option 2) so the token is renewed at least once per period.

Option 2: Sidecar renewer¶

Regardless of whether the token is periodic or has a fixed max TTL, a sidecar process must call vault token renew before each expiry. Two common implementations:

vault agent configured with an auto_auth block. The agent authenticates, writes the resulting token to a sink, and renews the token automatically. Point component_keyring_vault.cnf at a token minted from the same auth method, or generate the config with agent templating.
A systemd timer or cron job that runs vault token renew on a schedule shorter than the token TTL (for example, every 1 hour for a 24-hour token). Run the job as a dedicated service account, not as root, and log renewal failures to your alerting system.

Whichever you pick, alert on renewal failure. A silently failing renewer is indistinguishable from no renewer at all until the next MySQL restart — at which point the server will not come back up.

Operational checklist¶

Treat the token value as a secret on par with the Vault unseal keys. Restrict filesystem permissions on component_keyring_vault.cnf to the mysql user.
Monitor token TTL with vault token lookup and alert when remaining TTL drops below a safe threshold.
Rotate the token only during a maintenance window: update the configuration file, then restart mysqld. The component reads the file only at startup.
Share one token with only one server. Pair one-token-per-server with the existing secret_mount_point warning earlier on this page.

Vault seal status¶

On startup, and every time the component performs a key operation, keyring_vault reaches the Vault HTTP API. If Vault is sealed, that API returns a 503 for any request against the secrets engine, and the component cannot read or write keys — even with a perfectly valid token.

Vault becomes sealed in several ordinary situations:

A host reboot or package upgrade restarts the Vault process.
An operator manually runs vault operator seal.
Vault seals itself in response to a detected integrity issue.

While Vault is sealed, Percona Server behavior depends on timing:

If mysqld is already running and keys for currently-open tables are cached in memory, reads and writes against those tables continue to work.
If an operation needs a new key fetch — opening an encrypted table that was not already open, rotating the master key, or creating an encrypted tablespace — the operation fails.
If mysqld is restarted while Vault is sealed, InnoDB cannot unwrap tablespace keys, and startup fails or encrypted tables remain inaccessible until Vault is unsealed.

Recommended practice¶

Unseal Vault before starting or restarting Percona Server. Confirm with vault status that Sealed is false.
Automate unsealing with auto-unseal. Use auto-unseal backed by a cloud KMS, HSM, or Transit secret engine so that a Vault restart does not require manual intervention. Without auto-unseal, a Vault reboot that happens outside business hours can block every Percona Server that depends on the sealed Vault.
Order your startup dependencies. If Vault and MySQL run on the same host or are managed by the same orchestrator, make mysqld depend on Vault being unsealed, not merely on Vault being reachable.
Monitor sys/health. Vault’s /v1/sys/health endpoint reports seal status. Alert on sealed=true independently of whether any database has noticed yet, so operators can act before the next MySQL restart.
Keep unseal keys and recovery keys offline and distributed. This key custody practice is a Vault concern rather than a Percona one, but a sealed Vault whose unseal keys are lost is equivalent to permanent data loss for every Percona Server that depended on that Vault.