TL;DR - .NET Core provides a great way to define where your configuration is located. However, the provider for Azure Key Vault is a bit tricky and you should be cautious about potentially wrong credentials due to stale caches.

With the introduction of .NET Core, you can build a list of configuration sources and define which one has priority over another.

Here is an example of how I build my configuration for Promitor:

var configuration = new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddYamlFile("/config/runtime.yaml", optional: false, reloadOnChange: true)
                .AddEnvironmentVariables()
                .AddEnvironmentVariables(prefix: "PROMITOR:")
                .Build();

I tell .NET Core to use the following lookup flow:

1. Check if there's an environment variable which uses my key, prefixed by PROMITOR:
2. Check if there's an environment variable which uses my key
3. Check if it's in my runtime YAML configuration (kudos to Andrew Lock for building a YAML provider)

Very descriptive, very simple!

Best practices for working with Azure Key Vault

It is no secret that I'm a fan of Azure Key Vault and I've been using it for a long time! It's truly awesome and very easy to use!

However, Azure Key Vault has a few pitfalls and hurdles that you need to be aware of.

Authentication with Azure AD

One of the biggest hurdle(s), but also benefits, is that authentication for data & control plane are managed by Azure AD. If you want to integrate with Key Vault you need to authenticate to AD first!

I highly recommend using Azure Managed Identity which allows you to defer authentication to Azure. I wrote about it a long time ago but the concept is still the same - An Azure AD identity gets assigned to your Azure resource which will do the authentication handshake for you and we don't have to manage any secrets!

Note: Azure Managed Identity used to be called Azure Managed Service Identity.

Throttling & Caching

The service limitations of Azure Key Vault are very low allowing you to do 2000 operations per 10 seconds on secrets, managed storage account keys, and vault transactions per vault per region - That's 200 per second.

When you are building scalable applications that's a very low amount, certainly if you keep in mind that you might have other systems, like Azure DevOps, that read from the vault as well.

This is why it's important to not use the "One Key Vault to rule them all" principle and build them around your security boundaries as I've discussed in my talk at Azure Low Lands.

Another practice I recommend is using in-memory caching which allows you to reduce the load on Azure Key Vault while still being able to use the secrets. However, it's important to only cache them for a certain amount of time allowing your app to regularly check for new versions.

Applications have to be aware of key rolling so that if they are unauthorized that they can skip the cache and go directly fetch the latest version of a secret.

One approach to achieve this is by using Polly:

private static async Task QueueMessageAsync(byte[] rawOrder, string connectionString)
{
    var orderMessage = new Message(rawOrder);
    var sender = new MessageSender(connectionString, OrdersQueueName);
    await sender.SendAsync(orderMessage);
}

private async Task ProcessNewOrderAsync(byte[] rawOrder)
{
    var connectionString = await secretProvider.GetSecretAsync(SecretName);

    var retryPolicy = Policy.Handle<UnauthorizedAccessException>()
        .RetryAsync(retryCount: 5, onRetryAsync: async (exception, retryCount, context) =>
        {
            telemetryProvider.LogTrace($"Unauthorized to access Azure Service Bus. Reading latest key from Key Vault");
            connectionString = await secretProvider.GetSecretAsync(SecretName, ignoreCache: true);
        });

    await retryPolicy.ExecuteAsync(async () => await QueueMessageAsync(rawOrder, connectionString));
}

You can see all of this in action in my "In-Memory Caching with automatic Service Bus authentication key rolling" demo.

Getting secrets from Key Vault in .NET Core

.NET Core has built-in support for Azure Key Vault via the Microsoft.Extensions.Configuration.AzureKeyVault NuGet package, allowing you to use it as a configuration provider.

Here is an example of using Azure Managed Identity with Azure Key Vault in .NET Core: (docs)

// using Microsoft.Azure.KeyVault;
// using Microsoft.Azure.Services.AppAuthentication;
// using Microsoft.Extensions.Configuration.AzureKeyVault;

public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
    WebHost.CreateDefaultBuilder(args)
        .ConfigureAppConfiguration((context, config) =>
        {
            if (context.HostingEnvironment.IsProduction())
            {
                var builtConfig = config.Build();

                var azureServiceTokenProvider = new AzureServiceTokenProvider();
                var keyVaultClient = new KeyVaultClient(
                    new KeyVaultClient.AuthenticationCallback(
                        azureServiceTokenProvider.KeyVaultTokenCallback));

                config.AddAzureKeyVault(
                    $"https://{builtConfig["KeyVaultName"]}.vault.azure.net/",
                    keyVaultClient,
                    new DefaultKeyVaultSecretManager());
            }
        })
        .UseStartup<Startup>();

You can read more about it in the documentation

Pretty awesome right?! We can now pull secrets from Azure Key Vault and we're good to go!

The danger

.NET Core provides caching by default until you force a reload, and you cannot control for how long to cache it. So what's the problem?

You have to deliberately reload secrets by calling Configuration.Reload();.

This means that while your app requests secrets, it will initially load all the secrets and cache them internally.

Did somebody change a secret in the vault? Your app has to reload the configuration. One way to approach it is to write your own cron job and call Configuration.Reload to stay up to date.

Another important aspect is the unauthorized scenario that I've mentioned above, this means that instead of skipping the cache for a given secret, you now have to reload it for all of them.

.NET Core 3.0 will give you cache control

.NET Core 3.0 will ship with the capability to control how long secrets are cached thanks to HobbyProjects who contributed this!

This is more than welcome, but will not be available in .NET Core 2.x.

Key Vault made simple with Arcus Security

At Codit we started Arcus a collection of open-source libraries that make it easier to build applications on Microsoft Azure.

One of the areas we have invested in is security - We've built a SecretProvider which allows you to connect to multiple stores where we currently support Azure Key Vault.

We handle all of the authentication, allowing you to focus on your application

var vaultAuthenticator = new ManagedServiceIdentityAuthenticator();
var vaultConfiguration = new KeyVaultConfiguration(keyVaultUri);
var keyVaultSecretProvider = new KeyVaultSecretProvider(vaultAuthenticator, vaultConfiguration)

Another aspect that we provide is built-in caching where you can manage how long they should be cached:

var cacheConfiguration = new CacheConfiguration(TimeSpan.FromMinutes(10)); // Optional: Default is 5 min
var cachedSecretProvider = new CachedSecretProvider(secretProvider, cacheConfiguration);

Do we want to replace the .NET Core configuration provider? Certainly not! But we want to support scenarios where you cannot/do not want to use the .NET Core configuration.

Arcus is free to use and we are open for feedback, feature requests and/or contributions!

Are secrets actually configuration?

Consuming secrets via the configuration provider is nice, but I'm always cautious when going that route.

The beauty of the configuration is that you don't have to care where the value comes from, you just get it. The risk is that you do not know where it comes from, you just get it.

Make sure you handle secrets correctly; for example, you should not log them nor persist them somewhere, etc.

For this exact reason I tend to use a SecretProvider and a ConfigurationProider/IConfiguration next to each other, it clearly states that values coming from one are secrets while the others are configuration.

Am I exaggerating? Maybe, but better be safe than sorry.

Conclusion

When you are working with Azure Key Vault, make sure to always cache your secrets for a given amount of time to avoid hitting the service limitations.

However, do not cache them indefinitely to reduce the exposure in case somebody takes a memory dump while the secret is no longer being actively used in your app.

Using key rolling is recommended. This allows you to frequently change the secrets in your application allowing you to reduce the risk when secrets get compromised.

Use the built-in Azure Key Vault configuration provider in .NET Core v2.x as they are cached until you reload it. However .NET Core 3.0 you will be able to define how long they should be cached which is a big step forward!

If you want to use Key Vault outside of the configuration context, I recommend using Arcus Security which makes it a lot easier for you to authenticate, consume and cache secrets.

Thanks for reading,

Tom.

Collins Lesulie