security Secure PVAccess

Secure PVAccess (SPVA) enhances the existing PVAccess protocol by integrating Transport Layer Security (TLS) with comprehensive Cert Management, enabling encrypted communication channels and authenticated connections between EPICS clients and servers (EPICS agents) - see AuthN & AuthZ.

For a glossary of terms see: SPVA Glossary

Key Features:

• Encrypted communication using TLS 1.3

• Certificate-based authentication

• Comprehensive certificate lifecycle management

• Backward compatibility with existing PVAccess deployments

• Integration with site authentication systems

Note: This release requires specific unmerged changes to epics-base.

See Quick Start PVXS to get started.

Transport Layer Security

SPVA uses TLS 1.3 to establish secure connections between EPICS agents. Both client and server can authenticate their peer using X.509 certificates. Key features of the TLS implementation:

• Mutual authentication when both peers present valid certificates

• Server-only authentication when only the server presents a certificate

• Fallback to TCP when TLS is not configured or certificates are invalid

• Certificate status verification during connection establishment

Supported Keychain-File Formats, Encodings and File Types

File Type	Extension	Encoding	Includes Private Key?	Includes Certificate Chain?	Common Usage
PKCS#12	.p12, .pfx	Binary	Optional (password)	Yes	Distributing cert key

To use any of these formats just use the appropriate file extension when specifying the keychain file.

Unsupported Certificate Formats, Encodings and File Types

File Type	Extension	Encoding	Includes Private Key?	Includes Certificate Chain?	Common Usage
PEM	.pem, .crt, .cer, .key	Base64	Optional	Optional (concatenated)	Web servers, OpenSSL
JKS	.jks	Binary	Optional	Yes	Java applications

TLS encapsulation of the PVAccess protocol

In network protocols, encapsulation is used to transport a higher layer protocol over a lower layer protocol, e.g., TCP over IP. In the context of TLS, PVAccess messages are encapsulated within TLS records for secure transport.

Encapsulation involves wrapping the higher-layer protocol’s data within the lower-layer protocol’s format. TLS is so named because it wraps all data above the Transport Layer in an impermeable Security layer. For SPVA, this means PVAccess messages are wrapped in TLS records that include headers specifying content type, protocol version, and length, followed by the encrypted PVAccess data as the payload.

Note: We use TLS version 1.3 for Secure PVAccess. This version deprecates support for connection renegotiation which is a security risk. So any connections that are established using Secure PVAccess will not be renegotiated but will be closed if a certificate is revoked or needs to be renewed

Environment Variables

The following environment variables control SPVA behavior:

Note

There is an implied hierarchy to the applicability of the environment variables such that the PVAS version supersedes a PVA version. So, if an EPICS server agent wants to specify its keychain file location it can simply provide the EPICS_PVA_TLS_KEYCHAIN environment variable as long as EPICS_PVAS_TLS_KEYCHAIN is not configured.

Name	Key	Value	Description
EPICS_PVA_CERT_PV_PREFIX	{string prefix for certificate management PVs} e.g. ORNL_CERTS		This replaces the default CERT prefix. Followed by :STATUS:..., :ROOT, or :CREATE to form PV names
EPICS_PVA_TLS_KEYCHAIN	{fully qualified path to keychain file} e.g. ~/.config/client.p12, ~/.config/server.p12		This is the string that determines the fully qualified path to the keychain file that contains the certificate, and private keys used in the TLS handshake. Note: If not specified then TLS is disabled
EPICS_PVAS_TLS_KEYCHAIN
EPICS_PVA_TLS_KEYCHAIN _PWD_FILE	{fully qualified path to keychain password file} e.g. ~/.config/client.pass, ~/.config/server.pass		This is the string that determines the fully qualified path to a file that contains the password that unlocks the keychain file. This is optional. If not specified, the keychain file contents will not be encrypted. It is not recommended to not specify a password file.
EPICS_PVAS_TLS_KEYCHAIN _PWD_FILE
EPICS_PVA_TLS_OPTIONS Sets the TLS options for clients and servers. A string containing key/value pairs separated by commas, tabs or newlines	client_cert Determines whether client certificates are required	optional (default)	During TLS handshake require client certificate to be presented
		require	Don’t require client certificate to be presented.
	on_expiration Determines what to do when an EPICS agent’s certificate has expired, and a new one can’t be automatically provisioned	fallback-to-tcp (default)	For servers only tcp search requests will be responded to. For clients then no client certificate will be presented in the TLS handshake (but searches will still offer both tls and tcp as supported protocols)
		shutdown	The process will exit gracefully.
		standby	Servers will not respond to any requests until a new certificate is successfully provisioned. It will keep retrying the keychain file periodically. When a valid certificate is available it will continue as normal. For a client standby has the same effect as shutdown.
	on_no_cms Determines what to do when CMS is unavailable	fallback-to-tcp (default)	If revocation status check is required but CMS cannot degrade connections to tcp mode
		throw	Otherwise throw an exception
	no_revocation_check Determines whether cert status is monitored		This flag, if present, disables certificate revocation status monitoring meaning that this certificate will not be able to be revoked. Default: revocation status monitoring is not disabled
	no_stapling Determines whether stapling is enabled	yes, true, 1	Servers won’t staple certificate status, clients won’t request stapling information during TLS handshake
		no, false, 0 (default)	Don’t disable stapling
EPICS_PVA_TLS_PORT	{port number} default 5076 e.g. 8076		This is a number that determines the port used for the Secure PVAccess, either as the port on the Secure PVAccess server for clients to connect to - PVA, or as the local port number for Secure PVAccess servers to listen on - PVAS.
EPICS_PVAS_TLS_PORT
EPICS_PVAS_TLS_STOP _IF_NO_CERT.	yes, true, 1		For servers only. Stop if no certificate is provided.
	no, false, 0 (default)
SSLKEYLOGFILE	{fully qualified path to key log file} e.g. ~/.config/keylog		This is the path to the SSL key log file that, in conjunction with the build-time macro PVXS_ENABLE_SSLKEYLOGFILE, controls where and whether we store the session key for TLS sessions in a file. If it is defined, then the code will contain the calls to save the keys in the file specified by this variable.

API Configuration Options

The following are new configuration options now available in both the pvxs::server::Config and pvxs::client::Config classes, via their public base pvxs::impl::ConfigCommon class:

• pvxs::impl::ConfigCommon::expiration_behaviour - Set certificate expiration behavior

• pvxs::impl::ConfigCommon::tls_keychain_file - Set keychain file path

• pvxs::impl::ConfigCommon::tls_keychain_pwd - Set keychain file password

• pvxs::impl::ConfigCommon::tls_client_cert_required - Control client certificate requirements

• pvxs::impl::ConfigCommon::tls_disable_stapling - Disable certificate status stapling

• pvxs::impl::ConfigCommon::tls_disable_status_check - Disable certificate status checking

• pvxs::impl::ConfigCommon::tls_disabled - Disable TLS

• pvxs::impl::ConfigCommon::tls_port - Set TLS port number

• pvxs::impl::ConfigCommon::tls_throw_if_cant_verify - Control verification failure behavior

Here are server-specific configuration options:

• pvxs::server::Config::tls_stop_if_no_cert - Stop server if certificate unavailable

• pvxs::server::Config::tls_throw_if_no_cert - Throw exception if certificate unavailable

API Additions for Secure PVAccess

Runtime Reconfiguration

Allows runtime reconfiguration of a TLS connection. It does this by dropping all TLS connections and then re-initialising them using the given configuration. This means checking if the certificates and keys exist, loading and verifying them, checking for status and status of peers, etc.

• pvxs::client::Context::reconfigure and

• pvxs::server::Server::reconfigure

Example of TLS configuration reconfiguration:

// Initial client setup with certificate
auto cli_conf(serv.clientConfig());
cli_conf.tls_keychain_file = "client1.p12";
auto cli(cli_conf.build());

// Later reconfiguration with new certificate
cli_conf = cli.config();
cli_conf.tls_keychain_file = "client2.p12";
cli_conf.tls_keychain_pwd = "pwd";
cli.reconfigure(cli_conf);

Wildcard PV Support

This addition is based on the Wildcard PV support included in epics-base since version 3. It extends this support to pvxs allowing PVs to be specified as wildcard patterns. We use this to provide individualised PVs for each certificate’s status management.

• pvxs::server::SharedWildcardPV

Example of support for pattern-matched PV names:

// Define a server that responds to any SEARCH request with WILDCARD:PV:<4-characters>:<any-string>
// It will extract the 4-character part of the PV name as the `id` and
// the last string as the `name`

SharedWildcardPV wildcard_pv(SharedWildcardPV::buildMailbox());
wildcard_pv.onFirstConnect([](SharedWildcardPV &pv, const std::string &pv_name,
                            const std::list<std::string> &parameters) {
    // Extract id and name from parameters
    auto it = parameters.begin();
    const std::string &id = *it;
    const std::string &name = *++it;

    // Process and post value
    if (pv.isOpen(pv_name)) {
        pv.post(pv_name, value);
    } else {
        pv.open(pv_name, value);
    }
});
wildcard_pv.onLastDisconnect([](SharedWildcardPV &pv, const std::string &pv_name,
                            const std::list<std::string> &parameters) {
    pv.close(pv_name);
});

// Add wildcard PV to server
serv.addPV("WILDCARD:PV:????:*", wildcard_pv);

Protocol Operation

Connection Establishment

Connections are established using TLS if at least the server side is configured for TLS.

Prior to the TLS handshake:

• Certificates are loaded and validated

• Certificate authority trust is verified all the way down the chain

• Both sides subscribe to certificate status where configured for their own certificate and all those in the chain

• All certificate statues are cached

During the TLS handshake:

• Certificates are exchanged

• Servers staple cached certificate status in handshake

• Both sides validate and verify their peer certificate against trusted root certificates

After the TLS handshake:

• Both sides subscribe to peer certificate status where configured

• Clients may use stapled server certificate status

Connection Types

There are three types of possible connections

• tcp / tcp : TCP only connection, legacy ca

• tcp / tls : server-only authenticated TLS

• tls / tls : mutually authenticated TLS

If an EPICS agent finds a certificate, trust anchor, and private key at the location specified by EPICS_PVA_TLS_KEYCHAIN then it will use that certificate for the handshake. This includes the default location that the variable points to even if its is not set.

• ~/.config/pva/1.3/client.p12 - for clients

• ~/.config/pva/1.3/server.p12 - for servers

For a server-only authenticated TLS connection two things are required:

• The server must be configured to allow server-only authenticated TLS connections by setting the EPICS_PVA_TLS_OPTIONS option client_cert to optional

• The client must have a keychain file containing a trust anchor alone, at the location specified by EPICS_PVA_TLS_KEYCHAIN. - The p12 file can be created using authnstd with the -t option.

State Machines

Server TLS Context State:

The state transitions based on:

• Certificate validity and configuration

• Certificate status monitoring results

• API Configuration Options options (e.g., EPICS_PVAS_TLS_STOP_IF_NO_CERT)

States:

• Init: Initial state, loads and validates certificates

• TcpReady: Responds to TCP protocol requests when certificates are valid

• TlsReady: Responds to both TCP and TLS protocol requests

• DegradedMode: Fallback state for invalid certificates, missing TLS configuration, or revoked certificates

Client TLS Context State:

Similar to server but with key differences:

• Never exits on TLS configuration issues

• Trust Anchor validation affects initial state transitions

States:

• Init: Initial state, loads and validates certificates

• TcpReady: Responds to TCP protocol requests when certificates are valid

• TlsReady: Responds to both TCP and TLS protocol requests

• DegradedMode: Fallback state for invalid certificates, missing TLS configuration, or revoked certificates

Peer Certificate Status State Machine:

Shows the possible states and transitions for a peer’s certificate status. Applies equally to EPICS Clients, and Servers.

States:

• UNKNOWN: Initial state before status is determined

• GOOD: Certificate is valid and trusted

• NOT GOOD: Certificate is not currently trusted

Pseudo States:

• STALE: Status information is outdated. State transitions immediately to UNKNOWN

• REVOKED: Certificate has been permanently revoked. No way back from this status

• EXPIRED: Certificate has passed its validity period. No way back from this status

State transitions occur based on:

• Subscribed-to status updates from PVACMS

  • Certificate expiration

  • Certificate revocation

• PVACMS availability

PVAccess Sequence Diagram

The following diagram shows the PVAccess connection establishment sequence:

Secure PVAccess Sequence Diagram

The following diagram shows how the Secure PVAccess protocol establishes a secure connection between an EPICS client and server:

Click for a detailed diagram with or without certificate status monitoring

1. Each agent uses an X.509 certificate for peer authentication

   • Certificate is verified against own trust anchor (stored in same keychain file as certificate)

   • Multiple certificates may be verified in the chain to trust anchor

   • Verification checks the signature, expiration, and usage flags

   • If certificate is configured for status monitoring, subscribes to status updates from PVACMS

   • Certificate status is verified and cached

2. During handshake:

   • Certificates are exchanged

   • Both sides verify peer certificates against their own trust anchor

   • Multiple peer certificates may be verified in the chain to own trust anchor

   • Verification checks the signature, expiration, and usage flags

   • If peer certificate is configured for status monitoring, subscribes to status updates from PVACMS

   • Peer certificate status is verified and cached

   • Server may staple its own certificate status in handshake

   • Client may use stapled status immediately, instead of waiting for status monitoring results

3. SPVA certificates may include status monitoring extension requiring:

   • Subscription to certificate status from issuing Certificate Authority’s service (PVACMS)

   • Receipt of GOOD status before trust

4. Agents subscribe to:

   • Own certificate status

   • Peer’s certificate status

5. Servers cache and staple certificate status in handshake

Certificate Status Verification

Certificate Status received from the PVACMS for a certificate returns a GOOD status if, and only if,

• the certificate status is GOOD and

• so is that of its trust anchor certificate chain

• all the way back to the root certificate

In this way agents need monitor only their own entity certificate and that of their peer.

Certificate status verification occurs at several points:

1. Initial Connection

   • Certificates are verified during TLS handshake

   • Both peers verify against a trusted root certificate that they have loaded from their own keychain file

   • Basic checks include:

     • Signature validation

     • Expiration dates

     • Usage flags

2. Runtime Monitoring

   • EPICS agents subscribe to:

     • Their own certificate status

     • Peer entity certificate status

3. Status Response Handling

   • If status not received:

     • Search requests are ignored

     • Client retries later

   • If status not GOOD:

     • Server offers only TCP protocol

     • Client fails connection validation

   • If status GOOD:

     • Server offers both TCP and TLS

     • Connection proceeds normally

4. Optimization

   • Servers cache status for stapling

   • Clients can use stapled status

   • Reduces initial PVACMS requests

Status Caching

• Agents subscribe to peer certificate status

• Status transitions trigger connection status re-evaluation

• Cached status are reused within validity period to reduce PVACMS requests

• Servers staple cached status in handshake

• Clients may skip initial PVACMS request using stapled status

Beacons

PVAccess Beacon Messages have not been upgraded to TLS support. Important considerations:

1. Historical Use: - Previously used to trigger resend of unanswered Search Messages - This practice is now discouraged - Other methods should be used to determine server status

2. Current Behavior: - Servers broadcast on any configured port - Clients should not use ports directly - Use only as server availability indicator

3. Security Implications: - Beacons remain unencrypted - Do not contain sensitive information - Cannot be used for secure discovery

Protocol Debugging

TLS Packet Inspection

For detailed TLS traffic analysis:

1. Enable key logging at build time:

   • Set PVXS_ENABLE_SSLKEYLOGFILE during compilation

2. Configure runtime logging:

export SSLKEYLOGFILE=/tmp/sslkeylog.log

3. Configure Wireshark:

   • Edit > Preferences > Protocols > TLS

   • Set “(Pre)-Master-Secret log filename” to match SSLKEYLOGFILE path

   • TLS traffic will now be decrypted in Wireshark

Debug Logging

Enable detailed PVXS debug logging:

1. Environment variable method:

export PVXS_LOG="pvxs.stapling*=DEBUG"

New Debug Categories:

• pvxs.certs.auth - Authenticators

• pvxs.auth.cfg - Authn configuration

• pvxs.auth.cms - CMS

• pvxs.auth.jwt - JWT Authenticator

• pvxs.auth.krb - Kerberos Authenticator

• pvxs.auth.mon - Certificate Status Monitoring

• pvxs.auth.stat - Certificate Status

• pvxs.auth.std - Standard Authenticator

• pvxs.auth.tool - Certificate Management Tools (pvacert)

• pvxs.certs.status - Certificate Status Management

• pvxs.ossl.init - TLS initialization

• pvxs.ossl.io - TLS I/O

• pvxs.stapling - OCSP stapling

Network Deployment

Deployment Patterns

1. Standard Network Deployment

   • Agents run on networked hosts with local storage

   • Certificates stored in local protected directories

   • Standard TLS configuration applies

2. Diskless Network Deployment

   • Agents run on hosts without local storage

   • Certificates stored on network-mounted storage

   • Special considerations for certificate protection

3. Hybrid Deployment

   • Mix of standard and diskless nodes

   • Common trust anchor required

   • Consistent Cert Management across node types

Keychain-File Storage

Standard Nodes:

• XDG_CONFIG_HOME standard is used by default to locate keychain files for clients and servers

  • default, if not set, is ~/.config

  • we append /pva/1.3/ to make the full path default to ~/.config/pva/1.3/

  • client keychain files are, by default, client.p12

  • server keychain files are, by default, server.p12

• Store certificates in local protected directory

  • keychain files contain - the certificate - the private key - the certificate authority chain including the root certificate

  • the files are protected with 400 so that only the owner can read

• Automatic reconfiguration on certificate updates

Diskless Nodes:

• Use network-mounted storage (NFS, SMB/CIFS, AFP)

• Protected certificate storage location

• Optional password protection via diskless server

Trust Establishment

1. Trust Anchor Distribution:

   • Administrators must distribute PKCS#12 files containing the Root Certificate Authority certificate to all clients

   • These files must be stored at the location pointed to by EPICS_PVA_TLS_KEYCHAIN or equivalent

   • These files are replaced with any new certificates that are generated for the user but the trust anchor certificate is preserved

   • Use of publicly signed trust anchor certificates is not supported at present

2. Trust Anchor Distribution with Authenticators:

   • The trust anchor certificate is delivered with the entity certificate.

   • Users must verify that the issuer of the certificate matches the Root Certificate Authority they are expecting.

   • To control the selection of PVACMS service and thus the trust anchor certificate, users verify their PVAccess configuration.

     • EPICS_PVA_ADDR_LIST

     • EPICS_PVA_AUTO_ADDR_LIST

   • Consistent across all deployment types

3. Getting Trust Anchor from PVACMS

   • Use authnstd to get the trust anchor certificate from PVACMS

   • The p12 file created can be used by a client to create a server-only authenticated TLS connection

# Get the trust anchor certificate from PVACMS and save to the location specified by ``EPICS_PVA_TLS_KEYCHAIN``
authnstd --trust-anchor

# Get the trust anchor certificate from PVACMS and save to the location specified by ``EPICS_PVAS_TLS_KEYCHAIN``
authnstd -u server -a

4. Certificate Authority = Trust Anchor

   • PVACMS serves as site Certificate Authority

   • Common trust anchor for all nodes

   • Handles certificate lifecycle management