Encryption and Key Management for Regulated Workloads: Questions for Your Cloud Service Provider.

Background and Context

Much has been written by cloud service providers lately in over-simplified glossy marketing and website articles about their encryption and data protection services. Encryption in the cloud is a hot topic, and it is crucial for cloud service providers to design and implement secure services with broad applicability across their user-base.

However, regulated clients need to give specific and focused attention to aspects that pertain to the compliant and secure handling of regulated data. The intent of this note is to outline key considerations for clients and cloud service providers serving regulated clients.

Important Note: Each of the many global and local regulators are taking a different position on the topics and questions below. It is appropriate to assume that regulators will take a cautious approach during early adoption of cloud infrastructure. Also, regulators may view IaaS and SaaS services as presenting different levels of risk. So far, SaaS services have provided easier adoption paths. However, that may be a point in time factor that may not hold in future.

Introduction

As soon as clients think about placing sensitive, private or regulated data into a cloud hosted service, the question arises about how it is protected from unauthorized access. While traditional identification management and access control solutions are useful, they are not enough as there are multiple potential points of circumvention. Therefore, additional layers of protection are required, and the most obvious of these is encryption.

Encryption is a way of encoding and protecting data using an algorithm seeded by keys. Data that has been encoded by an encryption algorithm is said to be encrypted. To decrypt data from encoded back to the original form requires knowledge and access to a key.

Encryption algorithms are typically incredibly sophisticated and to prevent keys from being guessed, they are long sequences of bits. Even if a user tried to guess a key using multiple attempts at a brute-force, it could take many hundreds of years of super-computing power to gain access to decrypted data.

Therefore, encryption is typically considered the most secure way to protect data stored in a cloud service.

Note: there are other approaches such as tokenization (a one-way encryption mechanism) and obfuscation.

Points of Potential Encryption Layers

The consumer of a cloud service needs to be extremely mindful that performing encryption is possible in multiple places within the ecosystem. Typically, a client may wish to encrypt in multiple layers.

Layers include:

• Individual storage drives (both spinning and solid-state) attached to servers: current-generation storage drives can typically self-encrypt and decrypt data at the I/O operation level. The manufacturer usually sets default encryption keys, but these can be changed using a bios utility or other drive firmware configuration software. Self-encrypting drives were primarily designed to protect the “fall off the back of a truck” transport situations which might are more prevalent in traditional data centers, but are highly unlikely given the business controls and practices adopted by cloud service providers. Therefore, self-encrypting drives are not considered to provide an acceptable level of protection as data was likely in unencrypted form through multiple layers of infrastructure before reaching storage.
• A storage sub-system such as a NAS or SAN device providing file, block or object service: All cloud service providers provide pre-configured storage services that can be mounted to servers using standard protocols, for example, NFS. Additionally, some cloud service providers enable their clients to deploy software-defined storage products and frameworks onto bare metal servers. These can include IBM Spectrum Accelerate, IBM Spectrum Scale, and EMC ScaleIO, as well as many others. Storage subsystems typically provide encryption mechanisms. While encryption at this layer can be useful, typically it is unacceptable, as data typically moved in clear from the operating system to the storage service.
• The Operating System (OS), for example, Windows or Linux, is often an appropriate layer in which to include encryption capabilities. Encryption libraries, from the OS vendor or 3rd parties, can be installed and configured to ensure that the data contained within all IO operations, whether to local drives, or to a mounted file system device, are automatically encrypted on write, and decrypted on read. Encryption and decryption align with the operating system file access permissions.
• A Hypervisor
• Middleware capabilities such as a database or message queuing service can encrypt data above the operating system. This approach is typically built into middleware products to remove any dependency on an operating system framework or capability.
• Applications: This method is used to remove the dependency on any lower level service such as middleware, OS or storage. Given that some clients are concerned that application data could is accessible in clear, from memory, network communication or storage, they take the very guarded approach of encrypting data within application operations.
• At the Customer Premises, before transmission to a Cloud Service. A cautious client might determine that no clear or unencrypted data transfer to a cloud service. Client-side encryption is a highly cautious approach, but it certainly ensures that there is little chance or possible access or decryption from within the cloud service. Some cloud services add value through their ability to access and perform analytic or semantic operations on data. Such services would most unlikely be unable to process encrypted input data.

Keys

Central to the function of encryption and decryption algorithms are the keys uses to drive the algorithms. For the current standard of AES encryption algorithms, keys are typically 128, 192 or 256 bits in length. While the purpose of this article is not to go into detail about the algorithms or keys, multiple types of key might be used to drive an algorithm, such as an overall master key, key encrypting keys which are used to protect encryption and decryption keys, and the encryption and decryption keys.

Key Storage

While it may seem obvious, let’s be careful to note that keys of any kind need to be subject to safe handling such that their access is protected such that a malicious actor could not gain access to be able to decrypt protected data.

Firstly, some regulated clients question whether keys can ever storage is possible inside a cloud service. Regulated customers may choose to insist on storing keys on their premises. We’ll touch on the reasons for this in the key ownership section below. The client restricting key storage to their premises could have a severe impact on their ability to leverage services built by a cloud service provider or 3rd party such as a SaaS provider. Either of these providers is likely to make assumptions about key storage which would not include access to a key repository outside the cloud service.

Note that placing a key-store outside the premises of a cloud service provider, will introduce network latency to key-access operations requested from inside the cloud service provider. Depending upon the number of key-access operations, this may become a deployment architecture consideration.

 Regardless of where to place a key-store, it is likely to accessed and operated using one of the following mechanisms:

• Unsecured flat file or database – never a recommended approach!
• Within an application’s source code or configuration – not recommended!
• Using a secure software key store (e.g., macOS Keychain)
• Using a key management and encryption framework, eg., HyTrust or IBM Cloud Data Encryption Services
• Using a Hardware Security Module (HSM), a trusted attack resistant security appliance that resides on the client premises or the cloud service provider facilities. Most cloud service providers offer a form of HSM device. Some offer a virtual software version, others such as IBM, offer a full hardware appliance which is resident on a client’s VLAN. Key users, such as an application or storage encryption framework, use standard secure APIs to access the HSM which can also service encrypt/decrypt operations.

The Regulatory Consideration: Key Ownership and Attestation

For a regulated client to be compliant with anticipated regulatory requirements, the current thinking is that they will find it necessary to be able to assert and attest (meaning to provide documentary evidence) that the client, and only the client, owns and has access to keys.

If this regulatory position fully plays out as currently anticipated, it will mean some if not all of the following:

• A client cannot use keys owned, created, shared, known or manipulated by a cloud service infrastructure provider or any other cloud ecosystem partner, such as a SaaS provider
• A client must generate, own and not share encryption keys
• A cloud service provider can have no access to a client key store, including those provided by the cloud, such as an HSM. Only a client may have secure access to a key store.

The key question for regulated clients is whether they can assert and attest regulatory compliant key ownership when using the key management framework and encryption solutions provided by cloud service providers. The Line of business developers, based upon assurances or marketing by cloud service providers, have implemented regulated business capabilities within cloud service providers' embedded services. The associated certification staff will need to determine whether the security approach of embedded services is compliant with regulatory standards.

As a closing thought on key-ownership, I should note that some cloud providers do enable their clients to assert and operate security and encryption frameworks that will allow regulatory key ownership attestation. IBM Cloud enables clients to construct uniquely compliant solutions with highly granular control of encryption and key management solutions, in harmony with its customers' current on-premises frameworks and approaches.

Key Re-Key - Changing Keys

Another important consideration of key management is the frequency with which to change keys. To improve the security of key management, a client will typically take the decision to change keys at a periodic rate, e.g., monthly.

When changing keys, it is paramount to be mindful of the lost access to previously encrypted data unless:

• Old keys are retained and associated with a date range period
• Data encrypted with the old key is decrypted and re-encrypted with the new key. Given the quantity of stored data, this may create a time consideration, or a period during which data is temporarily unavailable.
• If neither of the above, the old key is destroyed access to any data encrypted with that key is no longer available. Given key destruction, encrypted is effectively deleted and destroyed. Decryption of previous data is impossible after key deletion.

Key Deletion and its Relationship to Data Deletion.

In a prior point, I noted that rigorously deleting a key is akin to data deletion. While this may seem like an obscure point, maybe in a cloud environment, it is a more appropriate way to address data deletion and destruction, than delayed overwrite algorithms or demagnetization. Key deletion immediately prevents unauthorized access to data.

CONCLUSIONS and NEXT STEPS

In conclusion, regulated clients need to consider:

• Where in an ecosystem to perform encryption and decryption
• How and where to store and manage keys
• How to assert and attest key ownership
• Whether frameworks like IBM Cloud Data Encryption Service, HyTrust or others can assist
• Under which circumstances customers may use embedded services that use keys managed or provided by a cloud service provider or 3rd party SaaS provider.

Attestation… Does Your Configuration Drift?

Happy New Year!

Now that we've relaxed, celebrated the holidays, and rung in a new year with optimism and hope, it is time to get right back to work! :-)


To begin 2017, let’s look at attestation and why it is necessary for regulated cloud environments.

Within the context of regulated industries, attestation is the ability to provide documented evidence to prove an assertion.  Such evidence might be needed to establish the “good” or “correct” configuration of a server hardware, server BIOS, embedded firmware, hypervisor, container, operating system, device drivers, middleware, and applications. 

As an example, a good configuration of an application environment means that every element of its executable code, libraries and configuration files, has traceable lineage back through build control and source control systems.  Typically, all components are appropriately licensed and have gone through appropriate vulnerability assessment processes.  Any change could only be made to code in a regulated application by following change-control processes updating attestable evidence.

While change control processes within the software development lifecycle, and DevOps mechanisms are well understood, perhaps there has been less ability to attest configurations in a regulated cloud environment.  This inability is partly because certain infrastructure responsibilities may shift from the regulated entity to a cloud service provider.

For example, can your cloud service provider attest any evidence about the safe configuration of their server, firmware or hypervisor?  If they do, what safeguards and change control mechanisms are in place to cover changes to the configuration, whether planned, unintentional or malicious, a situation known as “configuration drift.”

Unless a cloud service provider can both attest and ensure a known configuration on an ongoing basis, the most cautious regulated firms will typically require bare metal capabilities of their cloud service provider.  Bare metal uniquely enables the regulated entity to control the configuration, detection, and attestation.

Make no mistake, attestation of configuration across many tiers of infrastructure is necessary but can be a burdensome and expensive challenge.  Some frameworks and tools simplify the operationalization of attestation.  For example, take a look at Cloud Raxak, a cloud compliance firm founded by former IBMer and friend, Sesh Murthy. Cloud Raxak documents configuration, and detects/addresses drift from boot time onwards through multiple stack tiers. https://www.cloudraxak.com

Attestation will be a recurring theme for regulated cloud in 2017, a year in which we can finally expect to see a widespread and accelerated adoption of Cloud Service Providers by regulated firms.

Incident Management Planning with a Tabletop Exercise

Within an operational IT ecosystem, issues occur continuously.  Mostly, incidents relate to application defects, resource consumption or the outage of a core infrastructure component such as a server, hypervisor, storage subsystem or network.

There are a whole series of other failures or issues that can occur because of an operational error, a configuration issue, or any number of malicious attacks.

In the course of managing an IT ecosystem, regulated firms need to determine which of the plethora of events are to be regarded, classified and handled as incidents. 

Typically, a regulated company will work closely with its cloud service provider to understand incident management roles and responsibilities in detail.  Before the deployment of production workload, the stakeholders conduct a Table Top Exercise, a subset of techniques derived from planning military operations.

The table top exercise is a meeting to discuss simulated emergency situations. Stakeholders review and consider the necessary actions taken in multiple emergency scenarios, testing their incident response plan in an informal, low-stress environment. Central to the exercise is creating an understanding of the information needed to handle an incident, the sources of information, the decision-making roles & responsibilities and the sequence of hand-offs.

Specifically, a table top exercise goes beyond the need to purely understand an incident.  While not all scenarios will cause business impact or outage to the availability of service, the role of a table top will include how the company makes provisions to ensure continuity of business.  Equally, a table top may determine recommendations for how to ensure an incident does not recur, or specify how to preserve information for subsequent external inspection and audit.

Ahead of time, stakeholders will agree on a set of potential incident scenarios varying in severity.  Each will be simulated to test the possible response with the essential steps captured and documented.


Finally, a Table Top exercise might be useful as part of contractual discussions between a client and cloud service provider in the creation of a Cloud Services Agreement.  The table top output might be a RACI that unambiguously specifies roles and responsibilities.




Reference Examples:

The FDIC has good planning materials for evaluating cyber incidents.
https://www.fdic.gov/regulations/resources/director/technical/cyber/cyber.html

https://www.youtube.com/results?search_query=table+top+incident+managment contains multiple video examples of table top exercises that focus on public emergency situations or business continuity situations.

Consider These Themes Before Selecting a Cloud Provider for Your Regulated Workloads

Might I suggest you consider the following important points as you select a cloud provider...

• Are cloud computing offerings the core business of your chosen cloud providers?
• Is cloud a financially viable business for the cloud provider?
• Does the cloud provider have a strong technical vision, ability to deliver and proven expertise?
• How does the cloud service provider maintain a compliant position if using 3rd party staff? What contractual arrangements are in place that enables compliance to be asserted or validated?
• Are data centers and operational function locations appropriately secured?
• What are the plans for Continuity of Business and Disaster Recover?  Do major outages impact a client? What capacity remains available in the event of an outage and how can it be reserved and accessible?
• Track record and availability statistics for service offerings?
• References from existing clients within regulated industries
• Unambiguously documented roles and responsibilities (especially for availability, monitoring, incident management, security, and privacy)
• Reporting capabilities for availability, usage and financial metrics
• Ability to assure infrastructure, storage, and staffing location 
• Compliance with published regulatory standards
• How should consideration of the above change when buying higher value offerings such as PaaS and SaaS?
• Does the cloud provider understand how to sell and service enterprise clients?
• Does the cloud provider encourage a one-size fits all approach? - likely this does not work for regulated industries?
• Can the cloud provider support hybrid on-prem/off-prem deployment models with a supporting ecosystem of connectivity, consistency, and interoperability?
• Is pricing competitive?  
• Is the cloud provider profitable and sustainable?

*** Vic Winkler's book, "Securing the Cloud: Computer Security Techniques and Tactics", inspired me in creating this list.

Media Sanitization: Even if you Erase Your Data, is it Really Gone?

A topic that I've seen multiple times on cloud security/privacy evaluation questionnaires centers on the question of how data is disposed and erased such that it cannot be retrieved.

It goes without saying that there are many subtleties to this question is answered, and infrastructural characteristics become huge considerations.

Most people understand that commoditized operating systems do not remove files from physical media in the event of a delete request.  Delete operations merely place a deletion mark in the index entry of a file system directory structure. Having marked a file as deleted makes the physical blocks or sectors of a file available for re-use, but over-write typically only takes place after writing new data or extending an existing file.  Until that time, a deleted file is likely recoverable using simple and readily available utilities.

Within cloud services, multiple storage options are available.  These range from locally attached HDD, SSD or Flash to SAN or NAS attached storage servers.  For each of these options, the sanitization approach is likely to be different, and, depending on which cloud provider, it may be difficult to get a definitive answer about whether sanitization is even available.

Many of us have used destructive deletion utilities on our personal devices. These tools use a technique called over-writing in which a file or block is overwritten multiple times with 0s or streams of random characters. However, many people are yet to realize that the overwrite technique is uniquely applicable to magnetic media.

However, Cloud services are now making use of locally attached Flash or Solid State Drives (Flash in an HDD form factor). To preserve their performance and lifespan, flash controllers seek to distribute writes across the entire available memory capacity of a device.  Overwrite is not possible without severely degrading lifespan by over-writing every single unused sector.  To overcome this feature, flash manufacturers provide reset capabilities that destructively overwrite every device sector simultaneously.  Overwrite is not available at the file level. 

Therefore, as a cloud user, it is imperative to understand whether you can discover information about the physical infrastructure and piece parts of your server.  IBM Cloud's bare metal servers make this knowledge readily available for information and audit purposes.

Before releasing a bare metal server back to a cloud provider's inventory, assuming operating system access, a client can choose to execute appropriate data destruction techniques or contract a service provider to do the same.  It is also possible to contract that a cloud provider will physically destroy a server and storage components as needed, especially at end-of-life.

As one moves further away from physical infrastructure to virtualized network-attached and potentially multi-tenant storage, the question of assured media sanitization becomes much more challenging, if not impossible.

For this reason, other techniques such as encryption of data at rest become necessary.  We'll discuss these another time.


For more background reading on this, you might enjoy:

NIST Guidelines for Media Sanitization (PDF)


As always, your thoughts and comments will be enjoyed and appreciated.

Asserting Data Ownership in Regulated Cloud

When deploying to the cloud, regulated companies need to be able to make unambiguous assertions about their data ownership.

The primary question about data ownership centers on who can access regulated data, and how to protect data from unintentional access by an unauthorized user.

It goes without saying that we expect that data placed in a cloud environment will be encrypted.  Both data in motion and at rest is likely to be encrypted, with the encryption/decryption operations performed by the communication, application, middleware or base infrastructure tiers.

An encryption algorithm needs one or more encryption keys to determine its approach to scrambling or unscrambling stored data or streams of data in motion.  A random stream of bits is used to construct an encryption key.  The longer the stream of bits, the harder it is to crack or break the algorithm.

As the equivalent of a front-door key which controls access, the encryption key itself has value, just as the data it protects. For compliance purposes, a regulator typically needs confidence that the regulated company owns any key, and does not share them.

While there are many ways to store a key, protecting one robustly typically falls to a Hardware Security Module (HSM).  An HSM is a tamper resistant security appliance in which security parameters can be stored, exercised and retrieved using standard protocols and APIs.

https://en.wikipedia.org/wiki/Hardware_security_module

"A hardware security module (HSM) is a physical computing device that safeguards and manages digital keys for strong authentication and provides cryptoprocessing. These modules traditionally come in the form of a plug-in card or an external device that attaches directly to a computer or network server."

Cloud service providers recognized the need to provide customer-dedicated HSM devices.  With IBM Cloud's SoftLayer offering, a client can order an HSM from the management portal.  While some cloud providers can provide both logical (virtualized multi-tenant appliance) and physical HSM devices, IBM's approach favors regulated environments by exclusively offering a dedicated hardware option.

For assurance, a client ordering an HSM, IBM cloud provides a physical single-tenant device delivered in factory state accessible only to the customer's virtual LAN (VLAN).  The client receives a one-time user-id and password and must set the HSM with credentials that are never captured or shared with IBM. IBM has no access to the HSM, except to monitor that it is powered and alive.

The working principle is that the regulated client will never share HSM access or encryption parameters with the cloud provider. On that basis, a customer subject to regulation,  can confidently assert that they uniquely own both their security parameters and their data.

For more information, take a look at the link below or drop me a message.

https://www-01.ibm.com/common/ssi/cgi-bin/ssialias?htmlfid=KUS12364USEN