We live in a world where more and more of our personal information is held online. It’s often a single source of truth about us, the place where health information and financial records are stored and managed, used to make decisions about what we can and can’t do. Critical business records are stored online, finally replacing paper for contracts and for important transactions.
But how do we know that data is secure? There’s a certain trust in an encrypted hard drive sitting in a PC under your desk or even in your data center. But what about the cloud? So much of our compute and storage has migrated to services like Azure, either using cloud-native compute or lifted and shifted as virtual infrastructures. Now our data is just one tenant among many in a shared infrastructure where we have no control over how it’s stored and managed.
What’s needed is a cloud architecture that is delivered as a secure infrastructure for networking, compute, and storage, not only for the code running on it, but secure at such a low level that cloud platform operators can’t access it, even if there’s a breach that breaks isolation between tenants. It’s an approach that’s become known as “confidential computing,” relying on encryption at all levels, even application execution using the Software Guard Extensions (SGX) to the x64 instruction set, with code running in trusted execution environments.
On the compute side of the scale, Azure Confidential Computing provides a way to work with confidential data in a cryptographically secure space, using Intel’s SGX instruction set to enhance the isolation between tenants. By encrypting memory there’s no way for information to leak between users and between applications.
Things are more complicated when it comes to storage and working with stored data. What’s needed here is more than encrypted data. We need to know who did what to that data. You can think of it as an extension of the logs used by modern databases, a tool that can reconstruct every transaction made in order, replay it, and arrive at the exact same state. That’s what we mean when we talk about secure ledgers.
Running a secured confidential ledger in Azure
An encrypted log like this is basically a blockchain, a solution that Microsoft has experimented with in Azure in the past. But if you don’t need to use a blockchain to verify the actions of untrusted parties. You can implement the key ledger functions as a stand-alone application that still implements a secured log, using a blockchain-based approach without the complexities that come with the proof-of-work and proof-of-stake approaches to blockchains.
We’ve seen some of this work in the recently announced Azure SQL secure ledger tables, but now Azure Confidential Ledger takes Microsoft’s ledger technology out of the database, offering it as a simple API that can be used from any application with a simple REST call. Azure Confidential Ledger’s API-based approach goes as far as providing administrative APIs that can be used from your own management tools.
Microsoft describes its approach to ledger technology as “designing ourselves out of the solution.” Only you have access to the ledger, ensuring data integrity that’s not normally provided by cloud solutions. Microsoft’s staff, from its developers to its administrators, are blocked from access to your encrypted data.
Under the hood is a minimal Azure host running a trusted computing base that only supports the ledger and can’t be accessed by other applications, avoiding the risks that come with shared physical memory. Keeping the overall attack surface of the host to a minimum reduces risk, making it harder for a bad actor to compromise your ledger and access its data.
The service has entered public preview (currently with no charge), with a focus on providing an immutable and tamperproof record store. You can set it up from the Azure Portal, via an ARM template, or from the Azure CLI. Access is controlled through certificate-based authentication. Future releases will extend this to Azure Active Directory, adding role-based access control. For now, any code you use will need to work with the Azure identity client.
Other prerequisites include the Confidential Ledger control plane and data plane client libraries. The preview has Python, .NET, and Java libraries, with more promised. Once you’ve installed your chosen set of tools into your development environment, you can either create a new resource group for your ledger or add it to an existing one. Once you’ve opened a resource group, you can register a Confidential Ledger and verify that it’s been created.
Getting started with Azure Confidential Ledger
Once a Confidential Ledger is up and running you can start to write code to use it. One important note: Ledgers need to have globally unique names, so make sure to use one that has a low chance of collision with one from outside your organization.
The two libraries have different purposes. The control plane library manages ledgers: creating them, deleting them, listing them. All actions need to be associated with an Azure account, setting up the basic details of a ledger before a data plane application adds data to the ledger. Using the data plane library to create a client is relatively simple, as you’re going to be writing unstructured data to the ledger. A client needs to use the ledger certificate to authenticate a connection, using its endpoint URL and application credentials. Adding a record is simply a matter of appending a new entry, with the entry contents a simple string.
Each new entry gets its own unique transaction ID, which can be used to read back data. It’s all very simple, with basic REST API calls that interact with the ledger. You don’t need to worry about the underlying secure execution environment or any of the cryptographic techniques used to store data. The Azure Confidential Ledger provides a sufficiently high-level abstraction from the technology so all that matters is what you write and how you read it back.
The role of a ledger is to hold data that’s at risk of forgery or compromise, protecting it from deletion or editing. Using Azure Confidential ledger as part of a line-of-business application can reduce the risk of fraud, as insiders won’t be able to cover up their actions. It also helps avoid some of the effects of ransomware or other attacks. A well-designed ledger can help recover lost data in traditional stores. For example, it can provide an external store for any transaction logs or add an extra layer to a non-relational document store.
The future: confidential computing as a service
Currently the Azure Confidential Ledger is a single-party system, with multiple replicas for redundancy. There are plans to extend it to more than one party, using a similar consortium model as used by the now deprecated Azure Blockchain Service. However, that’s still some ways off, and in practice, much of the benefit of a confidential ledger is to provide a single source of validated truth for a line-of-business system. Ensuring that confidential data is stored securely is perhaps the most important aspect of such a system, especially in regulated industries where significant fines and other penalties can be applied if data is lost in any way.
Tools like Azure Confidential Ledger are a way to get the benefits of secure blockchain storage while avoiding the latency and other issues that can occur in large-scale distributed systems. Locking down the system to a set of trusted secure environments with only API-based access adds an additional level of security, minimizing any attack surface. The result is many of the benefits of confidential computing with none of the complexity. You can think of Azure Confidential Ledger as “confidential computing as a service,” with no need to understand working with SGX instructions, something you should expect to see more of in the future.