Technology highlights of 2019

Tom Kerkhove -

Christmas trees started popping up, Mariah Carey was non-stop singing about Christmas and holidays had started so it was time to wind down and look back on some of the major technology highlights from 2019.

Our industry has never been as fast-paced as it was in 2019 so I won't cover everything, it's simply not possible.

In this article, I'll cover my most favorite highlights about the things I've been excited and/or passionate last year, most of them will be about Microsoft Azure and Cloud Native Computing Foundation (CNCF).

Here's an overview of what I'll discuss :

Azure is everywhere

Azure is expanding at a high pace

In 2019, Microsoft announced regions becoming GA or new ones that are being added:

  • General availability of Africa DCs - The first cloud datacenters in Africa at that time, but AWS is coming to Cape Town as well (Announcement)
  • General availability of United Arab Emirates (UAE) North/Central (Announcement)
  • Introduction of Switzerland North/West - Having worked for Swiss customers I have to say that this is a big one. Data sovereignty is a key aspect for banks, (re)insurance companies, et al so data cannot leave the country. Now they don't have to anymore! (Announcement)
  • Introduction of new Germany datacenters - Azure was already available but more as a sovereign cloud. With the new DCs they are also available in the traditional public cloud (Announcement)
  • Introduction of Norway West/East (Announcement)
  • Introduction of Sweden (Announcement)
  • Announcement of Qatar (Announcement - Coming in 2021)

With 55 regions worldwide; Microsoft Azure is available around the globe and there is not a single continent, other than Antarctica, where Azure is not available. That's even more than AWS (22 launched, 4 announced) and Google (20 launched, 5 announced) combined!

Here's a nice overview of all the Azure regions that are available today:

Azure Regions

That's crazy, right?!

But there's bad news as well, Azure US Gov Iowa region is being retired and removed on April 30, 2020 - If you're running your workloads there, it's time to move! (Announcement | Azure Deprecation Dashboard)

I'm looking forward to seeing what's next!

For example, Brazil South is still very lonely with no secondary region around, so I'm wondering - Did they just introduce an Azure region there for the FIFA World Cup back in 2016 or are they planning on expanding it further? We'll see how that evolves!

No, Azure is really everywhere with Azure Arc!

Azure Arc allows you to monitor & manage all your resources from within the Azure portal; even if they are running on-premises, on Azure Stack, edge or another cloud provider!

Azure Arc

It brings the Azure Resource Manager to where it needs to be allowing you to re-use the same tools & deployment mechanisms as you can do with Azure.

Azure Arc extends Azure Resource Manager

The Azure Arc Data Controller acts as a gateway between your resources on any infrastructure and leverage those insights in Microsoft Azure.

Local infrastructure

This feels like a good middle ground for a lot of scenarios that cannot be moved to the cloud but can still be monitored from the cloud using the same approach as your cloud-workloads.

Read the announcement here, here and here or read my take on it here.

Running Azure API Management everywhere with Azure Arc

One of the things that I've heard customers ask about a lot was to be able to run Azure API Management on-premises or anywhere they want - And I was one of them! Having worked with Kong in the past, I was mainly interested in having it as a Docker container which makes it super portable.

This year, Microsoft announced the preview of Azure Arc enabled API Management - A self-hosted version of the Azure API Management gateway which allows you to run it practically anywhere as it's a Docker container. The gateway uses a federated model where it's fully managed from the Microsoft Azure and will propagate all the changes to the gateways.

However, with great power comes great responsibility - Now that you are self-hosting the gateway it's also up to you to manage, operate, secure, scale & update the gateway to ensure it's still up. My recommendation is still to use a cloud-centric approach and use self-hosting only if you need to.

Read more about it in the announcement here, findings of my colleague during the private preview or watch my talk at Microsoft Ignite.

Compute

There has been a lot of changes in Azure Compute but here are a few that I've found interesting:

  • Introduction of Azure Spot Virtual Machines - The long-awaited Azure service which allows you to run workloads on unused VMs at a reduced price. AWS has had this for a long time and it's finally in Azure! (Announcement)

  • Introduction of Azure Functions Premium Plan - Can't handle cold start with Azure Functions? No worries, Premium plan avoids that by dedicating instances to you, but it takes the "less" out of "serverless". (Announcement)

While these are just a few announcements, there are a lot more that could interest you!

Open-Source & Cloud Native Compute Foundation (CNCF)

There's a heap of things that changed in the open-source space so I'll focus on the ones I care about the most.

Introduction of Open Application Model

What I've been lacking over the past couple of years is a clean way of describing what an application is. Typically your app consists of multiple components that are spread across multiple services in Azure or are represented by a variety of Kubernetes resources, but if you're new to the team you have to learn what these components represent.

That's why I was a fan when Azure Service Fabric Mesh was announced because everything is done from an application & component standpoint allowing new people to understand what our app landscape looks like, but it was only scoped to Azure Service Fabric Mesh.

In 2019, Microsoft & Alibaba started the Open Application Model (OAM), a team-centric standard for building cloud-native apps. I was more than happy to be part of this preview as it tackled exactly the problem that I was having - How do we define an application without making it specific to the runtime it will be deployed to?

With the Open Application Model, you describe your application in a vendor-neutral way allowing you to focus on your application and not the underlying infrastructure.

Since it's a specification, every implementation knows how everything should be deployed and will run it on their infrastructure based on what you've defined. This allows you to describe your app once, and run it anywhere.

Open Application Model is designed for collaboration to define what the application looks like. This allows everybody in the team to focus on what they know best, rather than having to learn new technology stacks every time.

OAM designed with the following personas in mind:

One of the main drivers is that each person has a different focus when it comes to the application where the consolidate all of their needs into the Open Application Model of your app:

As of today, there are already a handful of implementations but more of them are on the way:

With Microsoft being part of this initiative I think we'll see this being supported in Azure over time as well, I hope.

So don't worry - OAM is not only for Kubernetes customers, it goes beyond that! Given the sheer amount of companies involved, Kubernetes SIG App Delivery even asked OAM team to present their project and chances are that this will be the defacto application model in Kubernetes!

Once the application is defined, you can deploy it to one of the above implementations.

In this example, you see that the app is being deployed on Kubernetes by using Rudr where it maps all the app requirements in the OAM spec to Kubernetes components for us. This avoids that all developers have to be a Kubernetes expert to get the app deployed.

It's still early to move all of our customers over but the fact that companies like Riot Games, who started League Of Legends, have built an internal approach similar to OAM to tackle the same problem it's clear that this is a general problem that OAM is solving here and 2020 will be the year that more and more platforms support OAM!

There's a more detailed post on this in the works so stay tuned for that!

Read the announcement here, learn from Mark Russinovich why it's so great or Alibaba’s perspective on the Open Application Model.

CloudEvents reaches 1.0

CloudEvents is a specification for describing event data in common formats to provide interoperability across services, platforms, and systems and has been around since 2017.

Before CloudEvents, every vendor had its format such as Azure Event Grid:

{
    "topic":"/subscriptions/{subscription-id}",
    "subject":"/subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.EventGrid/eventSubscriptions/LogicAppdd584bdf-8347-49c9-b9a9-d1f980783501",
    "eventType":"Microsoft.Resources.ResourceWriteSuccess",
    "eventTime":"2017-08-16T03:54:38.2696833Z",
    "id":"25b3b0d0-d79b-44d5-9963-440d4e6a9bba",
    "data": {
        "authorization":"{azure_resource_manager_authorizations}",
        "claims":"{azure_resource_manager_claims}",
        "correlationId":"54ef1e39-6a82-44b3-abc1-bdeb6ce4d3c6",
        "httpRequest":"",
        "resourceProvider":"Microsoft.EventGrid",
        "resourceUri":"/subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.EventGrid/eventSubscriptions/LogicAppdd584bdf-8347-49c9-b9a9-d1f980783501",
        "operationName":"Microsoft.EventGrid/eventSubscriptions/write",
        "status":"Succeeded",
        "subscriptionId":"{subscription-id}",
        "tenantId":"72f988bf-86f1-41af-91ab-2d7cd011db47"
    }
}

With CloudEvents, you can choose to use a vendor-neutral way of eventing.

It allows customers to work with a platform-agnostic way that not only reduces vendor lock-in but also allows you to re-use tooling and products from other parties to build your application.

A typical CloudEvents looks like the following:

{
    "specversion" : "1.0",
    "type" : "com.github.pull.create",
    "source" : "/cloudevents/spec/pull",
    "subject" : "123",
    "id" : "A234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "comexampleextension1" : "value",
    "comexampleothervalue" : 5,
    "datacontenttype" : "text/xml",
    "data" : "<much wow=\"xml\"/>"
}

This year, CloudEvents has reached 1.0 which is a fantastic milestone which allows us to now adopt and use it everywhere as it's considered to be stable.

As part of that milestone, they provide a variety of SDKs (C#, Go, Java, Javascript, Python & Ruby) and other than the core spec, you can find more detailed specs for protocol bindings such as AMQP, NATS, HTTP, and more; for event formats such as JSON & AVRO or how it works with webhooks.

One of the great things of CloudEvents is that it supports legacy platforms without breaking existing clients via the binary CloudEvents rather than structured CloudEvents:

POST /someresource HTTP/1.1
Host: webhook.example.com
ce-specversion: 1.0
ce-type: com.github.pull.create
ce-time: 2018-04-05T17:31:00Z
ce-id: A234-1234-1234
ce-source: /cloudevents/spec/pull
    .... further attributes ...
Content-Type: text/xml; charset=utf-8
Content-Length: nnnn

<much wow="xml"/>

Existing products can just add the additional headers to describe the existing payload and that's it.

Kubernetes, which has its eventing engine & format, is a good example of this. We are currently looking at making it CloudEvents compliant by using this approach. Do you want this as well? Head over to the GitHub issue and give a thumbs up!

One aspect that I've found to be lacking was the discovery aspect - If I'm new to the company, how do I know what events are being emitted, who owns them, what their contract & subject is and what they represent.

As of today; your best bet, at least with Azure Event Grid, is to subscribe to all of them and learn from the events you're receiving and taking it from there.

Serverless WG has decided to start working on Discovery/Subscription API to tackle these issues such as event discovery, how subscriptions should behave and event domains. The latter should sound familiar to Azure Event Grid customers. Interested in learning more? You can have a look at their rough idea.

Read the annoucement here.

Autoscaling made simple with Kubernetes Event-driven Autoscaling (KEDA)

Microsoft collaborated with Red Hat to make application autoscaling on Kubernetes simple with Kubernetes Event-driven Autoscaling (KEDA).

KEDA builds on top of Kubernetes Horizontal Pod Autoscalers (HPA) to leverage 0-to-n application scaling based on a variety of scalers which you can use.

Application operators can deploy ScaledObjects to define how and when their application should scale and KEDA will then monitor the depending source for the required metrics and scale it accordingly. This means that application operators no longer have to deploy metric adapters themselves nor do they have to learn how application scaling works on Kubernetes, they just leave it up to KEDA to do the magic!

Scaling on Kubernetes has been one of my focuses in the past couple of years, heck that's how Promitor started, and it's not been super easy to scale apps because of the knowledge, complexity, and infrastructure it brings but KEDA changes all of this.

I've been super excited about it and am looking forward to seeing where it goes over time, you should keep an eye on this as we're working on donating this to the CNCF.

Read more about it in:

Virtual Nodes is GA on Azure Kubernetes Service

Virtual Nodes on Azure Kubernetes Service is now generally available and allows you to overflow your workloads to Azure Container Instances (ACI)!

This is another step that reduced the operational overhead of AKS and allowing us to focus on our applications. As of today, you still need at least one AKS node but in theory, you could have no nodes and run your workloads on ACI while using the Kubernetes management experience.

Read the annoucement here.

Cluster Autoscaler is now GA on Azure Kubernetes Service

When running Kubernetes, it's up to you to ensure that the cluster has enough resources available to share across your workloads.

Since this year, Kubernetes' Cluster Autoscaler is now generally available in Azure Kubernetes Service allowing you to do that automatically!

This is a huge step forward! Over time I hope the integration with the Azure ecosystem will get improved as it's now running in the Kubernetes cluster, without any awareness that it's scaling the cluster. It would be good to have Azure Monitor Autoscale integration which goes and manages the CA inside our cluster, but use the same scaling approach as we have with other Azure services.

Oh, and Event Grid events for scaling actions - Thank you!

Read the annoucement here.

Understanding your Kubernetes clusters with Octant

This year I've discovered Octant, a very nice open-source tool created by VMware that allows you to better understand what is going on in your Kubernetes cluster and learn how their applications are running.

My favorite feature is Resource Viewer - It visually shows how the resources are linked to each other and help you diagnose issues.

Here's a nice demo of the tool :

Event Grid is on fire!

Azure Event Grid is one of these powerful services that is often forgotten about, one could even say it's the heart of Microsoft Azure.

The team has been on fire and released a ton of great features:

  • Support for CloudEvent 1.0 to improve interoperability and integration with oter services (Announcement)
  • Natively push events to Azure AD protected endpoints (Announcement)
  • Batch delivery in Azure Event Grid for high-throughput scenarios (Announcement)
  • Enhanced observability for event-driven applications with Azure Event Grid (Announcement)
  • Automatic server-side geo-disaster recovery (Announcement)
  • New event destinations such as Azure Service Bus, Azure Storage Queues & Azure Hybrid Connections
  • Manual validation handshake (Announcement)
  • Event Domains is now GA (Announcement)

Next to that, Azure services are starting to see the value of Azure Event Grid and are emitting events - Finally! New services include Azure Key Vault, Azure Data Lake Store (Gen2), Azure Maps, Azure App Configuration and many more!

Luckily, Azure Key Vault is a textbook example of why it's so powerful! You can now subscribe to events when secret/keys are near expire, have expired, are updated, etc. This allows you to fully automate processes like certificate management without the depending applications noticing it - This is crazy!

Learn more about it here or use this tutorial to get it up & running.

Getting new Azure services onboarding Azure Event Grid has been a passion for a while now but it's been very hard, but I'm happy to see that it's starting to help!

A good friend of mine, Sean Feldman, started Azure Event Wishlist that acts as a dashboard for all event requests in Microsoft Azure. Are you missing events? Make sure to list them on Azure Event Wishlist, ask for them in feature requests or drop a comment below.

Introduction of Azure Lighthouse

At Codit, we work with a variety of customers to build the platforms that increase their business but also operate them. One thing that has been a bit of a pain is getting everybody access to get the job done, keeping the list up to date and having a birds-eye overview of all of our customers.

I'm more than happy to see Azure Lighthouse join the Azure family which is built for partners, like Codit, that are providing managed services! One of its core features is Azure delegated resource management which allows you to manage customer resources securely from within your tenant. This is a big improvement as we can now have a single pane of glass with all resources for all our customers, without having to use multiple tabs/browsers and context switching.

The Azure portal is enhanced by providing My Customers & Service Providers blades that provide you with an overview of all the parties with whom you are working, from both sides.

You can now offer your services to customers through private or public offers in the Azure Marketplace, and have them automatically onboard to Azure delegated resource management. As a logical step; Azure managed applications, which allows you to publish your app in the Azure portal so that customers can deploy it, have been moved under the Azure Lighthouse umbrella as well as it is aimed at tackling similar hurdles.

Read the annoucement here and here.

Collaboration done right with Jetbrains' Space

I've been mostly working remotely for the past couple of years and it's nice, but you have to have the right tools!

Azure DevOps is a really good tool that makes it nice to work as a team - You can plan, code, build, test & ship features with one suite of tools which I've begun to be a fan of. It's nice - And then came Jetbrains' Space!

Space is an integrated team environment that allows you to practically do everything in your day to day job. You can see it as a unification of Azure DevOps, Microsoft Teams Office 365 and more but in one tool.

Here's a nice overview:

One of the features that I'm excited about is how easy it is to keep track of what everybody in your team is doing and who I should not be disturbed by using the team calendar:

It's super easy to share progress with other people in your company or team via the internal blog:

Very easy to schedule meetings:

Space gives me a clear overview of what's on my plate, what I'm working on and what is going on in our company.

I've saved the best one for last, Space allows me to learn more about my colleagues!

For example, in what office is Sebastiano working? What does his agenda & travel planning look like? What teams is he part of an what is he working on? What languages does he speak and how can I contact him? But most importantly, who is he and what does he do in his spare time because that can be fun to know as well!

Does it mean I don't like tools such as Azure DevOps anymore? Certainly not, I'm still using it daily and like it a lot, but hopefully, we'll see better integration with other Microsoft products in the future! And does anybody still remember the Team Rooms? Maybe it should be kept around anyway, heh.

You can watch the full announcement here:

A big thumbs up for my friend and long-time mentor Maarten Balliauw for the awesome demo. It gives a nice overview of what the product delivers.

Conclusion

2019 was a very busy year and if you have a look at what was added in one year, it's crazy to think how we can all keep up!

Unfortunately, it's not all good news and things are getting removed - You can stay up to date on deprecations via the Azure Deprecation Dashboard, but more on that later on!

2019 was also the year that shipped Promitor 1.0 which was a personal milestone to me, certainly given companies like Walmart are using it.

I'm looking forward to seeing what 2020 will bring!

Thanks for reading,

Tom.

unsplash-logoJeremy Thomas