Over the past year or so, I've been spending a lot of time in the world of machine learning and data science. Not as a data scientist, but as a DevOps practitioner*.

I wanted to know -

• How do machine learning projects work?
• Who is involved?
• What is the process like from idea through to running in production?
• Most importantly - can DevOps help?

*As an aside, I tried about 4 different words before "practitioner". It feels so businessey. But here we are.

Learning Machine Learning

When I first encountered any kind of machine learning, it was at university in the form of "deep neural networks". It wasn't my project, but another student's.

He had great success with it, so I spoke to him about what was involved. After 5min, I was lost. It felt so inaccessible.

Non-linear activation functions, multi-dimensional calculus, stochastic gradient decent... I would have spent as much time with a dictionary as with a computer.

So, as any good student would do... I dropped it and started my career writing Visual Basic for the government.

Fast forward a few years, and I'm working at Microsoft as a Cloud Advocate. I'm here because I've apparently convinced the company that I know about DevOps and can communicate those ideas well. Sure. I'll take it.

The great thing about being in this role is I have access to fantastic experts in other fields, who are great at communicating. Enter Seth Juarez.

If I had to put my knowledge of AI and ML on a scale against Seth's, it would look a bit like this:

It's taken a while, and a lot of experimentation and conversations, but I feel like I've got a handle on what's involved in a machine learning project.

I mean, you probably shouldn't hire me to build a predictive model for you just yet, but I'm no longer completely clueless.

About that process

One of the things that all these conversations taught me was that the majority of ML projects don't have a great story when it comes to DevOps.

But let's back up a bit - isn't DevOps traditionally used to help deliver software?

Yes. And machine learning is software.

"MLOps", or "Machine Learning Operations" seems to be the term that much of the industry has landed on. I don't love it - it implies a narrow focus on operationalizing the results of a machine learning project and not the broader idea of creating value - but if that's the term used, I'll play along.

I really prefer "DevOps for Machine Learning". The reason? "DevOps" can absolutely apply to machine learning projects - without even stretching the definition.

DevOps

Let's quickly look at Microsoft's DevOps definition:

DevOps is the union of people, process, and products to enable continuous delivery of value to our end users

The important part of this definition is the word "value". DevOps isn't about features or bug fixes or infrastructure as code, it's about value. ML is software that provides value.

For example, if you're working on producing a predictive model for evaluating lending risk for a bank, what's more valuable than improving the accuracy of that model?

Back to the process

DevOps is full of great practices and patterns, and most of these can be applied to ML projects.

For me, the main differences between a machine learning project and a software development project come down to the following two distinctions:

There's a lot more experimentation

When writing a new feature in a piece of software, you generally know what you need to do up front.

For example, for a new "customer orders" report, you know there's probably a database query, some data processing logic, and a few screens or pages of some kind of reporting UI. There's a direction right away, even if you may have to try a few things.

ML Projects often involve a lot more experimentation. While the end goal is generally known, the path to that goal is often far less clear.

There are countless algorithms you could use and limitless ways to transform the data and select features. Undiscovered outliers, biases in the data, or poorly-tuned hyperparameters could push you off course for weeks.

Experience is important, and great data scientists have a feel for what will work, what to look for, and how to solve these tricky problems.

Data science is a specialist art form.

All that said, the actual execution of these experiments is ripe for automation.

The training, scoring, evaluation and testing are all things that require scripts and processing. And DevOps helps us with this.

Training runs are like really expensive builds

When we build our software on a build server, it's usually a process measured in minutes. Maybe hours if we're unlucky. That means CI is definitely something we should do

Producing a predictive model on an ML project might take weeks.

Each training run may require many thousands of compute hours on expensive GPU-laden machines and cost a senior developer's salary each time.

So you may not want to do a full training run for every small change (like you would with continuous integration). You may want to do a run with a subset of data to prove the idea first. Or maybe you just want to run tests or linting (it is code after all)?

Even so, the idea of a "training run" as a parallel to a "build" makes sense to me.

A build is essentially a compute-intensive process that produces an artifact that can be deployed or used by other software.

A training run is the same - except it's also very data-intensive.

The same DevOps techniques around CI, pull request builds, and even security, performance, and load testing can apply.

Same same, but different

The skills required to produce a good result may be very different. The cadence of work may be slower at first, and less cleanly broken down.
The tools used day to day might be very different.

But ultimately it's all writing code, running automated tasks, and deploying the artifacts safely.

If it's not all in code and in source control, it should be!

If model training requires manual work, it should be automated!

If deployment is manual and scary, it should be automated and gradual!

Finally, if the project is "done" when the model is in production, it can get stale and stop delivering value. There's a great opportunity to continue delivering value by using telemetry, monitoring, and ideally automated retraining strategies.

Apply DevOps practices to your ML projects. Work on delivering value more effectively on a continuous basis.

Find out more

There will be more blog posts drilling into some of these ideas in much more detail.

I've been speaking about MLOps or DevOps for Machine Learning at a few events recently, and there are more to come.

You can find out what Microsoft has been doing in the MLOps space on our Azure Machine Learning docs pages. We have some amazing tooling to help solve a lot of MLOps problems, and there's so much more to come!

Related events

Microsoft Ignite

One of the big events on the Microsoft calendar is Microsoft Ignite. It's less than a week away - November 4-8.

I'll be delivering a session on exactly this topic on Thursday afternoon. I'd love you to come along!

Also...

If you're going to Ignite and have an interest in MLOps, I'd love to speak to you!

I'll even have some of those "friends don't let friends" stickers in the image above. Just ask! 😀

Send me a message on Twitter, or just come by the Debug Bar in the Hub. I'll be there 11am-1pm Monday and Tuesday, from 1pm-4pm Wednesday, and from 12pm-2pm Thursday.

Microsoft Ignite The Tour

Microsoft Ignite The Tour is an event that Microsoft is running in 30 cities across the world, starting just after Ignite and running through to May next year.

The AIML50 session is focused completely on MLOps. I won't be delivering any of the tour sessions, but the presenters are amazing. Even though it won't be me saying the words and running the demos, the session is my work and I'm really proud of it.

Don't miss the (free) event when it comes to your city!

In my last post - Azure Pipelines YAML - Uh, what do I type? - I gave you two suggestions for how to find out what to type - both links to the docs.

Shortly after publishing, a few people from the awesome DevOps community recommended a few other ways, so I thought I'd share them. I totally already knew about all of them of course. Absolutely definitely.

So, more answers!

Answer #3 - Use the Visual Designer

But wait, Damian, isn't the whole point that we're not using the visual designer?

Yes. Yes, it is. However it can be very useful to use the visual designer just so you can then view the equivalent YAML.

To create a build with the visual designer, there's a not terribly obvious link in step 1 of the New Pipeline flow.

The YAML required for tasks is usually very straightforward, but by using the visual designer, you get a UI showing you every possible parameter.

It's especially useful for tasks you're unfamiliar with, or marketplace tasks where the YAML isn't documented or obvious. The name of those tasks is usually very long as well.

Just add the task, set the parameters, then click "View YAML". You can also "View YAML" for the whole pipeline if you want to build it with the UI but use YAML.

It's a bit like using the Visual Studio ASPX design view in old-school Web Forms before switching back to source view to see what OH NO WHERE DID ALL THESE BR TAGS COME FROM WHY WOULD YOU DO THIS!?

Ok, it's not completely like that.

Answer #4 - look at the task's git repo

Every out-of-the-box task in Azure Pipelines is open source. If that surprises you, you haven't been paying attention to this new Microsoft!

You can find all these tasks on GitHub at https://github.com/Microsoft/azure-pipelines-tasks in the Tasks folder. At time of writing, there are 135 of them.

To help you work out the YAML, the file you're looking for is task.json. Here's an example for the Kubernetes task (version 1): https://github.com/Microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/task.json

The name of the task is the name in the json file plus the version - e.g. Kubernetes@1.

The json file also contains all the possible input parameters you can use.

It can take a little bit of practice to understand these json files, but as a bonus - once you understand them, you can write your own tasks! That's right, just copy ours to create your own!

Answer #5 - More comprehensive docs

I kind of covered this in the first post, but it deserves another mention because I didn't link to it specifically.

There's a reference section in the Azure Pipelines docs that lists every out of the box task along with the YAML required to use it.

Unlike the quickstarts or language-specific guidance, the reference section lists every parameter each task can use. It's far more comprehensive and is extremely useful if you need to drill down a bit deeper.

Check it out. Again.

It's incredibly easy to get started with Azure Pipelines, especially if you have your code in GitHub already. Just add the Marketplace app and follow the prompts.

Does your code live somewhere else? No problem, just go to https://dev.azure.com and create a new organization and project.

And once again, Azure Pipelines is free. Not add-a-credit-card-but-make-sure-you-cancel-3-months-before-the-trial free, but totally free.

A few months ago, a preview feature was launched in Azure DevOps (then VSTS), letting you define your build in a YAML file that lived with your code.

This was awesome. It meant you could use the same branching and code review practices for your build definitions as you did for your code.

Now with the launch of Azure DevOps (which, by the way, is free), the service really encourages using YAML. In fact, if you start a new project, those feature flags are turned on by default, and the flow through the GitHub Marketplace app also pushes you to a YAML definition.

However, if you're like me and were used to the UI, or you're just completely new to Azure DevOps, it could be a bit confusing.

Namely - uh, what do I actually put in the YAML file?

Note: If you do want to keep using the UI, you can turn the feature off by following the instructions here.

Answer #1 - the templates

There are a whole lot of templates available for you to choose from and they contain some nice starting points.

When you create a new build definition, Azure Pipelines analyzes your code (at a pretty high level) and will suggest a template to start with. Or you can click "All templates", and pick one. There are quite a few!

In most cases, this will be a super simple definition that just builds the app. For example, the ASP.NET Core template (see the screenshot above) has a single step that runs dotnet build on the commandline.

Even if you plan to build out a task-rich build, the templates are a good place to start for a couple of reasons:

• First, they show you the syntax you'll need, and will already have the recommended build agent pool, variables, and basic tasks set.

• Second, there's a link at the top of the template that points you at the documentation for that specific project type. Which is exactly what you need right now.

So choose a template that makes sense, then [ctrl/⌘]-click on that link. It'll tell you the YAML you'll need for pretty much everything you'll want to do. If it doesn't, let me know and I'll get it fixed!

For example, here's what my ASP.NET Core build looks like. Every task I added is listed in the docs page linked to in the header - https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

# ASP.NET Core
# Build and test ASP.NET Core projects targeting .NET Core.
# Add steps that run tests, create a NuGet package, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

pool:
  vmImage: 'Ubuntu 16.04'

variables:
  buildConfiguration: 'Release'

steps:
- script: dotnet restore
  displayName: 'dotnet restore'

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'

- task: DotNetCoreCLI@2
  displayName: 'dotnet test $(buildConfiguration)'
  inputs:
    command: test
    projects: '**/*Tests/*.csproj'
    arguments: '--configuration $(buildConfiguration) --collect "Code coverage"'

- task: DotNetCoreCLI@2
  displayName: 'dotnet publish $(buildConfiguration)'
  inputs:
    command: publish
    publishWebProjects: True
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
    zipAfterPublish: True

- task: PublishBuildArtifacts@1
  displayName: 'publish artifacts'

That definition restores, builds, runs tests, pushes test results and code coverage back to Azure DevOps, packages the site for deployment, and pushes the packaged artifact back to Azure DevOps for safekeeping.

Answer #2 - the "Docs" button

You may have noticed the "Docs" button at the top of the screenshot as well. To save you scrolling, I'll paste it again:

That link takes you to an awesome docs page with links to quickstarts for (at time of writing) 15 different app types. There are also some links to pages showing how to deploy to 11 different deployment targets.

I firmly believe you should deploy with a separate Release pipeline (build once, deploy many!), but you can do it in the build if you want. It's yours, you can do whatever you like!

Check it out

It's incredibly easy to get started with Azure Pipelines, especially if you have your code in GitHub already. Just add the Marketplace app and follow the prompts.

Does your code live somewhere else? No problem, just go to https://dev.azure.com and create a new organization and project.

No credit card required - it's free. Did I mention that? Free!

Every year, DORA (DevOps Research and Assessment) releases an awesome "State of DevOps" report based on tens of thousands of surveys responses. For loud, talky DevOps people like me, it's an awesome opportunity to a) validate, and b) justify all the stuff we say is important.

The report is put together by Nicole Forsgren, Jez Humble, and Gene Kim. You should follow them (and me) on Twitter.

The 2018 report came out a few days ago, so I sat down to read it and summarise my highlights!

After 25 pages, I realised I was just writing bullet points of the whole report.

Yeah, highlights aren't highlights if they just summarize everything.

So that leads to my first recommendation:

Read the whole thing.
Then a couple of days later, read it again.
And then later that week, you've probably forgotten it, so read it again.

My highlights

Ok, I'll give you some highlights anyway. I managed to find highlights in my summary. Then highlights in those highlights.

The research itself used cluster analysis to group the respondents into three "software delivery performance" groups - high performers, medium performers, and low performers. There are also a couple of subgroups. Elite performers is a subset of high performers, and a really interesting "misguided performers" is a subset of low performers - more on that later.

So how are these groups measured?

Four measures you should care about

There are four key measures you should care about when it comes to software delivery performance:

• Deployment frequency (how often you can push into production)
• Lead time for changes (how long it takes to get a change from commit to production)
• Time to restore service (When stuff goes wrong, how long does it take you to fix it)
• Change fail rate (how often do your changes break stuff)

Here's how the best and worst groups compare:

That's the page you want to print out and hand to your manager. Convert that human-resource-time-dollars (or whatever managers use) and it can be pretty compelling. 💰💰💰💰💰

Misguided performers

One of the more interesting parts of this report was the discussion on misguided performers.

There is a tendency, especially in highly-regulated industries, to slow down. Be careful and cautious and make sure nothing can go wrong. You ask what happens when something does go wrong with a deployment and it's often deflected with, "Oh, it won't. We've made sure."

So then it does go wrong and because there's no process for dealing with this, the time to recover is horrible.

These are "misguided performers". Low performers, but with reasons*.

The data shows that these teams actually do tend to make fewer mistakes. Their change failure rate is better than other low performers. Not as good as medium performers, but better than the lowest.

Where this all falls down is the recovery. When it comes to recovering from an issue, they're worse than low performers. Like way worse. Up to 6 months worse.

Manual work

Measuring how much time is spent on manual work is a great way of measuring quality. There was a really interesting set of measures here too.

The medium group in that table is interesting and seems wrong at first glance - why would they be doing more manual work?!

But as the report explains, it makes sense. As teams start improving and automating things, there's suddenly a lot more testing that needs to happen. Technical debt builds up and can block progress. It's only after continually improving things that things start looking better.

So, like... "don't give up!" or something equally inspiring.

The rich get richer

One interesting thing in this report was that the best performers are getting better, and the worst are getting worse.

Compared with previous years, low performers have more failures and take longer to restore service. The gap is widening.

As DevOps matures in a lot of organisations, it's understandable that the highest performers continue improving, but why are the lowest performers getting worse?

Maybe it's the complexity of applications these days (microservices anyone?). Maybe it's doubling down on proven bad ideas?
Maybe it's viral chaos engineering meets artificial intelligence and the start of Skynet?

Do Cloud, but do it properly

If you're on the cloud, it's important that you are actually doing cloud stuff.

Ok, that doesn't make much sense. Let me try again.

The report references the NIST definition of cloud computing, which defines five essential characteristics. Things like on-demand self-service, and rapid elasticity. You can think of these characteristics as representing "doing cloud properly".

If you're just running a VM in the cloud, but doing everything else the same, you're probably not checking those 5 boxes. But if you actually meet all these cloud characteristics, you're 23 times more likely to be in the elite group when it comes to software delivery. That's a lot of times.

Outsourcing is bad m'kay?

This was really interesting. As a dev who has worked in quite a few companies across quite a few industries, it absolutely rings true. But that was just my anecdata. This puts real data behind that idea.

In short, if you outsource by function (e.g. all your testing goes to this third party company, or all the dev work is done by another org), you're far more likely to be in the low performing group.

And remember those Misguided performers? Yeah, turns out they're the worst for this. They use more functional outsourcing than any other group.

The reason? Contracts and agreements and change requests and batched work and all that stuff. Can you tell I did an MBA?

Look, all features are not equal. And if there's an agreement to deliver 10 features as part of some work, then it's possible that really important features are waiting around for nice-to-have's just to fulfil that agreement. I've seen it many, many times. It's nice to see my feelings backed up it in a real report.

A few other quick snippets

I realise I'm in danger of just summarizing the whole report again, so here are a few other highlights:

• Continuous Delivery contributes to reductions in burnout!
• If software relies on anything that's handled in a different department (e.g. infrastructure, security, databases), a) involve those teams super early (shift left!) not at the end, and b) include them in the software delivery process

One more that was amazing (from Google research):

• The most important factor in predicting a high-performing team is psychological safety.

Yep. If you feel safe taking a risk or admitting a mistake, your team will perform better.

Final note

Remember, all of this ties back to organisational performance. That's super important. If all these recommendations just made you better at delivering software, but that didn't have any impact on your bottom line, then why bother?

But it does. Regardless of industry.

Teams that are elite performers when it comes to software delivery and operational performance are 1.53 times more likely to meet or exceed their own goals.

So this stuff works.

Again, check out the whole report, and let me know your thoughts!

Awww Yeeeaah. That's right.

If you're just here for the absolutely-totally-not-supported agent,
you can download it here.

Us VSTS Microsofties like to talk about "Any Language, Any Platform". We do demos with .NET and Java and Node on Macs and Ubuntu machines, and deploy code to Azure and AWS and on-prem.

So when I saw this tweet from Edward Thomson, I was intrigued 🤔:

I ported the @VSTS build agent to Raspberry Pi - if you’re at #OSCON and interested in talking about cross-platform CI/CD drop by the Microsoft booth.

— Edward Thomson (@ethomson) July 19, 2018

Existing agent flavours

VSTS gives you a really easy way to install agents for Windows, macOS, Red Hat, and Ubuntu 14.04 and 16.04. Unfortunately, the Linux builds don't work on a Raspberry Pi. Probably not a surprise.

BUT, in typical recent-Microsoft style, the agents are open source! And they're written in C# for .NET Core!

So it should be totally possible to compile it for the Pi, right?

Well, yes, but there were a few changes needed.

Building a Raspberry Pi agent

Thankfully (for everyone), I didn't have to write any code, because Edward already did that for me. He had a pull request already in the works (don't be surprised if that link breaks soon), so I just grabbed that PR and compiled it on my Pi.

To compile, I needed to run two commands:

> ./dev.sh build
> ./dev.sh package

In fact, that second command is probably totally useless because I just unzipped it again.

But lo and behold, I had a zipped up agent, just like the ones VSTS gave me!

Well, kind of.

Missing bit #1

The agents you download from VSTS include a few things that this compiled agent doesn't have.

The first was the configuration and run scripts. I'd just compiled the pure, unadulterated agent. I imagine the scripts aren't totally necessary to run the agent, but I'm lazy. So...

I downloaded a Linux agent using the "Download agent" link in the Admin settings > Agent pools section of VSTS (https://your-account.visualstudio.com/_settings/agentpools) to pilfer those files and copy them to my new agent folder.

Now, I could run config.sh to configure and connect to VSTS, then run.sh to execute... and OMG the agent works!

Deployment agents

A small aside.

Did you know the exact same agent is used for builds, releases, and even targets in deployment groups?

Yeah, we're big into reuse.

I really just wanted to use the Pi as a machine to deploy to, so I reconfigured it as a target in a deployment group.

What's the difference between those three agent types? Here's my 2 second summary:

• Build agent - What you build your application with
• Release agent - What you release your application with (actually the same agents)
• Deployment agent - What you deploy your application to (different)

The separation isn't that black and white, but I'm getting off track, so ANYWAY...

I configured the agent as a deployment group target, ran it using run.sh again, and hooray, it connected!

All that's left now is to create a basic release definition using the Deployment Group (the Bash script is just a hello world at this point), and run it!

Missing bit #2

And it didn't work!

Why?

Well it turns out all* the tasks run via node. Yep, even if I have a single bash script in my build or release, it runs that baby through node.

There's probably a good reason. I should look into that.

* I don't actually know if it's all of them, but bash and python definitely do

The second missing bit is a packaged version of node. Remember that externals folder? Turns out there's a known version of node in there that the tasks use to run.

Which poses a problem - the versions that come with the supported agents won't work (I tried). They're for other OSs...

So what did I do? I just found similar folders elsewhere on the Pi and naively copied them over!

Hey, if it's stupid and it works, it's not stupid. 👉👉

For the record, here's what I copied:

• externals/node/bin - copy node from /usr/bin, and the npm shortcut from the linux agent (it won't resolve yet, but that's ok)
• externals/node/lib/node_modules/npm - copy the whole npm folder from /usr/lib/node_modules

Sure enough, success!

Should I do this?

Probably not.

I mean, don't get me wrong, it's pretty cool being able to deploy directly to the Pi using an agent like this, but what if you have hundreds of devices, or you're actually deploying to 25 light globes or a refrigerator?

This works great for a Pi, but otherwise you're better served using Azure IoT Hub. That's what it's for after all.

So... off to research that!

Someone sent me a message the other day asking for some help. They'd recently joined a new company that was thinking of setting up a "DevOps Department".

My immediate thought:

🚨🚨🚨 That's a recipe for failure before they've even begun. 🚨🚨🚨

K but why tho?

Usually when I see a "DevOps Department" or a "DevOps Team" or similar, it's implemented as a silo in the organisation. That silo holds all the knowledge regarding how to deploy and manage applications, and they're the only ones who can create new pipelines.

When that happens, development and operations don't work together any more than they did, they just have another group of people with their own priorities to contend with.

A new silo between Dev and Ops (or sometimes just replacing Ops) is completely against the point of DevOps.

Can it ever work?

Yes. I think so.

But only if that "department" is considered a knowledge center, and concentrates on the services and tools the developers and operations teams need to do things properly.

Tools like VSTS and Application Insights.

Or maybe GitHub, Team City, Octopus Deploy, and Raygun.

There are a ton of tools out there. The important thing is that the DevOps department not be the sole operators of those tools. They can guide, train, maintain, and manage those tools, but the developers and operations folks should be using them day to day. Not just clicking the buttons, but configuring builds, defining releases, pushing code that defines the infrastructure, watching production metrics, etc.

Cue happy teamwork shot:

The organisers of the Global DevOps Bootcamp put it well. The DevOps tooling should be self-service - it should be easy to configure a new pipeline, manage work, define new infrastructure, and monitor production.

✨ Easy does not mean raising a ticket with the DevOps department. ✨

Ok, so is my DevOps department destined to fail?

Is DevOps only practiced within that department?

There's your probable answer.

Microsoft Build 2018 is coming up next week and I'll be there with the rest of my team, as well as a whole lot of Cloud Developer Advocates and awesome Microsoft folk!

Here's where I'll be during the week (subject to change!).

FYI In the gaps between these times, I'll most likely be at the Lounge, or the VSTS booth - both in the conference hall.

Day 1 (Monday):

### Day 2 (Tuesday):

### Day 3 (Wednesday):

If you want to meet up any time around these commitments, either come find me in the Lounge, or hit me up on Twitter

I hope to see you there!

I've been throwing out this little one-liner for years and it seems to have resonated with some people. So much so that I made a sticker!

If you want one of these stickers, you have three options:

• Download the photoshop file here: https://brdy.in/fdlfrcp
• Tell me the email you use for StickerMule and I'll share the design
• Come and see me or one of the #LoECDA at one of the events we're going to and we'll probably just give you one!

Cool, cool, but what does it mean?

Great question. Glad you asked.

"Right-click publish" refers to the really, really, really, super common practice of publishing your web application to Azure directly from Visual Studio. It's a really popular technique used by developers everywhere... and far too frequently by Microsoft presenters on stage.

This is an awesome way to deploy your web application to Azure.
To try it out.
Like, once or twice, by yourself.
Then never do it again.

The moment you need to work with anyone else on the application, and/or start caring about what you're pushing to production, you should stop doing this.

But it's so easy, Damian, why are you making my life hard?

Well first, doing it properly is really not that hard. More on that in a bit.

But second, and more important, right-click publish is a fantastic way to accidentally break production. Which is bad. Obviously.

There's so much that's dependent on one person and one machine in the right-click publish scenario.

Maybe you have something installed on your machine that's not in production.
Maybe you didn't merge all the changes people made.
Maybe you did, but you resolved that tricky bit slightly wrong.
Maybe you merged successfully, but forgot to run tests.
Maybe you did run the tests, and they work fine on your machine, but not in production.
Maybe there are untested bugs that don't show up until the app is on a server.
Maybe your OS just got an upgrade and the code no longer compiles.

The list goes on.

What you really want is a repeatable, automatic pipeline that takes your code, merges it with everyone else's, runs some tests, deploys it to a test server, runs some more testing and validation, then automatically promotes it to production once you know it works.

Yeah, that sentence was long and scary, but it's honestly not hard. Donovan builds 4 of these pipelines in an hour on stage. All in different languages. From different operating systems. Even I've done it a bunch of times - you can build these things in a few minutes.

But if you're really attached to right-clicking, you can still do that. Just install the Continuous Delivery Tools for Visual Studio and now you can right-click Configure Continuous Delivery! Much better.

But it's just a small app, there's no point building a whole pipeline!

I see you're not letting go of this. Ok:

As Donovan frequently says, "you play how you practice". If you don't bother doing things properly for small apps (that sometimes grow into big ones), then you're less likely to do it for "real" projects.

When I worked in consulting, it became standard practice to build a pipeline all the way to production on day 1 of a project (yes, that's all the time it took). Sure, at that point you're only deploying "Hello World", but it makes every subsequent change so much easier to deploy safely. And if the process needs adjusting, you're only tweaking a pipeline rather than debating whether to build one from scratch.

Let's do the math(s🇦🇺), if it takes you 4 hours to set up a basic pipeline with Continuous Integration and Continuous Deployment you're happy with, how long until you've recovered that time?

Think about how long it takes to merge, compile, run tests, deploy to test, then deploy to production. If you're committing and deploying to (at least) test servers regularly, you'll get that time back really quickly.

Can you use right-click publish to go to production? Sure, you can, but you shouldn't.

Friends don't let friends right-click publish.

Visual Studio Team Services (which I'll call VSTS from now on to save the extra time I've just wasted typing this) has long been one of my favourite products. Now that I work for Microsoft, I can be a bit more effusive and open about my fanboyism.

via GIPHY

One of the great things about VSTS is the fast release cycle. Every 3 weeks there's a decent-sized release. Occasionally the changes are less visible (along the lines of "bug-fixes and performance improvements"), but often it's a massive slab of super useful new features.

With that cracking pace, it's often hard to keep up. Even if you work for Microsoft and have a direct line to the product teams.

Luckily, the Visual Studio Team Services Release Notes blog is really useful. Whenever I get a bit behind, it's easy to go through the recent sprints to see what fancy new features I missed.

Even so, it's easy to miss something, so I thought I'd put together a post with my (maybe) lesser-known VSTS highlights from 2017.

Feature flags for preview features

In the very first release of the year, a nice UI for preview feature flags was added. The VSTS team sometimes experiments with new features, and practices "testing in production" (the good kind, not the I-don't-have-a-test-environment kind). When a new feature is about to be released, it may be available to you early as an opt-in preview. It's worth checking out the available flags from time to time - you might be surprised!

Continuous Delivery in the Azure Portal with any git repo!

Way back in March, the team added the ability to do continuous delivery to an Azure App Service from any git repo. If you're using GitHub or VSTS, it's easy to set up a hook, but for other git repos Azure will periodically check for changes. It's CI/CD for everyone!

Commit history graph for git

A very welcome addition to the commit history for files or repositories. In April, we got a beautiful commit history graph right in the browser. It's beautifully done and uses canvas which is one of those HTML things I don't really know how to do so it's black magic.

Deployment Groups for deploying to VMs

VSTS has been great at deploying to Azure, but it was previously a little tricky to deploy to a bunch of VMs. With Deployment Groups, introduced in Preview in May, you can install agents on target machines and run release tasks directly on those machines! Yes. I know. Just like Octopus.

Export and Import Build Definitions

This was a top-ten request on UserVoice, so it's awesome it was made available in June. You've managed to get your build definition exactly right and want to use it in a different project or account? Easy - export the JSON, and import it elsewhere!

A new Release Definition Editor

In July, the team introduced this UI update. I love this one because it's JUST SO PRETTY. It's a beautiful way to visualize how you move between environments. Again, there's that svg and canvas black magic going on. It was taken out of preview and made generally available in August.

New Burndown and Burnup widgets

A small thing maybe, but burndown and burnup charts were awesome additions to the Analytics Extension in September. Especially useful for managers who are riding you about deadlines and estimates and stuff. Just give 'em a dashboard. Managers love dashboards.

Deployment Outputs as Variables

Often you want to use some information from the output of one deployment task in another task. For example, if you're deploying to an Azure Web App, you might need to use the URL in a testing task later on. It was always possible (albeit a little tricky), but this October addition made it much easier for some tasks - just give the variable a name and it's now available to you later.

YAML build definitions

My final pick, and this is a big one for a lot of reasons. For me, there are two reasons worth mentioning.

First, whenever a code update required changes to the build definition, you were left with a choice - commit the changes and watch the CI build fail (then update the build), or update the build first, then commit your changes and hope everything lined up. The story was much worse if you had multiple branches - you'd need multiple builds, managed separately, and a merge into master would require changing to what was the branched build... it got complicated.

Second, the build definition (i.e. the way to turn your code into software) was managed completely independently from the code itself. It makes much more sense for them to live together. Many companies and teams used build scripts checked into source control (build.ps1 or Cake) which worked fine, but really didn't make use of the awesome VSTS Build engine.

This is the first step to the best of both worlds! Have a YAML build definition that lives alongside your code. It uses the build engine, but also solves the two problems above. Ideally, it'll be able to be run completely independently of VSTS as well. Even better, in December we got the ability to view the YAML of an existing build definition!

Ok, that's it for me!

There are a TON of features released all the time with VSTS, and many more coming soon. To keep up to date, check out the Release Notes blog, as well as the VSTS Features Timeline which has future features as well.

There's a common pattern in IT. In many cases, it ultimately leads to a new buzzword. It's not complicated. In fact it's fairly obvious. But understanding what you can do with it can help immensely when it comes to pushing change in your organisation.

Agile, DevOps, Microservices, Containerization, whatever the next buzzword is - all of these "revolutions" are really a result of these three steps.

1. Awareness

2. Acknowledgement

3. Adjustment

Examples:

Let's look at a couple of examples to solidify what I'm talking about. I'm simplifying for the purposes of brevity, but consider Agile and Microservices:

Agile:

1. Awareness of slipping timelines, incorrect estimates, project failures.
2. Acknowledgement that big, upfront plans never seem to go smoothly, no matter how long you spend on them or how many people are involved. Some things you just can't know until you start.
3. Adjustment of how projects are run - you can't plan for the long term, so let's just plan a short distance ahead and iterate.

Microservices:

1. Awareness that even small changes can be expensive in a monolithic architecture, and side-effects can be hard to predict.
2. Acknowledgement that isolated and decoupled services can make change easier and more predictable.
3. Adjustment of how software is put together - small, self-contained components that can be iterated on more quickly.

The same pattern can apply for DevOps:

DevOps:

1. Awareness that software in production is the aim, and that there are some obvious bottlenecks that slow things down.
2. Acknowledgement that software should be able to help with this - most of what we do is able to be automated!
3. Adjustment by removing the bottlenecks using software and scripts, adding surety through automated testing, and reducing the cycle time from idea to production.

So what can I do with this?

I've given a few talks lately on the challenges of implementing DevOps in an organisation where you have little to no control over how the business operates.

In particular, my "Doing DevOps as a Politically Powerless Developer" talk has resonated with a lot of people lately.

As a "politically powerless developer" in an organisation, it can be difficult to instigate change, even when you know it's the right thing to do. In short, you probably don't have the power to implement the Adjustment part of this pattern.

But you can definitely affect Awareness. It's easy - start measuring.

Measure how long it takes for a change you commit to make it to production. What happens in that time? Is there a lot of waiting?

Measure how long it takes for a user-reported issue to get to your backlog. Add some production monitoring, and measure how much time is saved by learning about issues first-hand. How much time did you save?

Measure the time it takes to write tests, and compare that to the time you take fixing bugs for code that doesn't have tests around it. Can you prove that writing tests is beneficial overall?

Once you have information, you can give it to your boss or pass it up the chain. You've already acknowledged there's a better way, so you'll be ready with solutions once management acknowledges there's a problem!

Summary

To really instigate change and adjust how your organisation delivers software, there needs to be an acknowledgement that there's a problem. That won't happen unless there's awareness.

If you're having trouble pushing change, focus on awareness first.

Merging can suck.

My first real-world exposure to version control was in the good old Visual SourceSafe days. Back then (and continuing into server-side Team Foundation Version Control), it was common to have exclusive locks on files - i.e. only one person could work on a file at a time. If someone changed a file, you had to "Get Latest" before you could start working on it.

In practice, big changes required many files being locked by one person, and everyone was blocked until it was done.

This was unproductive.

So we worked on branches. We knew that the merge at the end was going to be horrible, but that was a problem for the future. Not worth worrying about now.

Then exclusive locks went away, and we started working on the same files, merging as we went. We could even work offline with distributed solutions like mercurial and git, and make changes in tiny branches (rather than branching the whole codebase), which meant smaller merges and less pain.

But big changes and refactorization were still a problem. You still had to merge your changes with everyone else's, and with everyone working on the same files, it became easier to work on your own branch (or a team branch) and worry about merging later. You could even push broken code to the server without bothering everyone else.

So we deferred merging.

Again, we knew the merge at the end was going to be horrible, so for big changes, our situation hadn't improved much.

Continuous Integration

Continuous Integration means merging all of your changes into everyone else's changes on a continuous basis in order to make sure problems are caught early. You can ensure that the code compiles, all the tests pass, and even run analysis tools to check for code quality or security vulnerabilities.

There are obvious advantages to continuous integration. It's common knowledge that problems are cheaper to fix the earlier they're identified. Whether that's a bug or a merge conflict, if you can resolve it immediately, it will save time in the long run. You want to know straight away whether you've "broken the build". Not a week later when another team member pulls your changes and can't compile, and certainly not a few months later when it's time to push to production¹.

If everyone on the team is working on the same code, and they're all pushing their changes regularly, continuous integration will identify problems quickly.

If a team is working with long-lived branches, they're not doing continuous integration. Instead, they're doing continuous isolation.

The term "continuous isolation" came from a tweet by benjiwebber and was expanded upon by Paul Hammant. Thoughtworks later identified the practice as "CI Theatre".

By working on a long-running branch, changes become more and more isolated from the other branches, including trunk (or master). CI builds can still exist against that branch, but they're only integrating changes within that branch, not the code that the other teams are committing. They might make the team feel good, but they're just deferring the pain.

Trunk-based development

The solution seems obvious - don't work on branches! Or if you do, use short-lived branches and merge back into the trunk as soon as you can.

I promote this way of working as much as I possibly can, but in practice, it's not always that simple. Not every change can be knocked over in a few days.

trunkbaseddevelopment.com is a fantastic website that goes through this idea in detail, including the practicalities of doing it in real teams. I'd highly recommend reading through it.

The other technique that can help is feature toggles (or any of a bunch of names that mean the same thing). The idea is that even unfinished code can be merged into the trunk, as long as it's hidden behind a flag. As long as it a) compiles, and b) doesn't break anything, you can deploy it all the way to production with the flag turned off. The code won't run, but you can still continuously integrate your code.

Feature toggles deserve their own blog post, so I won't expand further here.

Summary

While long-running branches may feel more productive day to day, realise that you're just deferring the merge pain. That conflict you introduced is going to need attention at some point. The sooner you address it, the easier it will be to resolve.

Integrate continuously, and don't let your branches last long.

Incidentally, there's a recording of an excellent talk on this very topic from GOTO Chicago earlier this year by the always-impressive Sam Newman. Do yourself a favour and watch it!

¹ Of course you should be deploying to production more regularly than once a month. Why wait if you've got something meaningful to ship!?

When you start something new, first on your agenda should be a website! Right?

One of the first things Donovan set up for the new "League of Extraordinary Cloud DevOps Advocates" team was a new LoeCDA.com website. The "Tour Dates" section will be extremely useful if you'd like to see any of us at an event near you!

He used the extremely useful yo team yeoman generator and had a new ASP.NET Core website, complete with a CI/CD pipeline, running on docker in Azure in just a few minutes. This still blows my mind, but I digress - that's not the point of this post.

Our dev process

Even though this is a very simple website, we agreed to make all changes via pull requests (PRs). This helps us in two ways:

1. We can exercise the PR process in Team Services and Visual Studio
2. It helps us learn from each other

Exercising features that real dev teams will be using is really important. Half of the Cloud Dev Advocate role is bringing feedback back to the engineering teams. What better way to do that than using the features?

Learning from each other is an often-overlooked benefit of working with pull requests. It forces code reviews, which is a great way to learn about new techniques or better ways to implement features.

So every change will occur on a feature branch (or topic branch), we'll submit a PR, another team member reviews and approves it, and merges in those changes. Of course once that happens, CI/CD ensures it gets to production with almost no more effort!

Finding out about PRs

We use Microsoft Teams for day-to-day communication, so what better way to learn about new pull requests than through a chat client that's always open?

Teams has a great feature called Connectors, which allows you to integrate with third-party applications. Thankfully, there's a connector for VSTS.

Configuring the VSTS connector

The first (optional) step is to set up a new channel in Teams. You can easily add a connector to an existing channel, but having a specific channel for PRs reduces the noise in your other channels.

To do this, just click on the ellipsis next to your team, and choose Add Channel.

Now click on the ellipsis next to the new channel, and choose Connectors.

Find the Visual Studio Team Services connector. If you already have some VSTS integration, you'll see a Configure button. If not, it will be Add.

If you're adding a VSTS connector for the first time, you'll need to walk through the steps to connect to VSTS. Once you've done so, you'll be able to choose your connection (or "Profile") for subsequent VSTS connector configurations.

The next part is easy - just go through the fields and choose the appropriate entries from the dropdowns. You'll need to choose a VSTS Account and Project, and you can choose to be alerted by events for individual VSTS teams, or for all teams.

You'll then be asked for an event type. There are a lot of VSTS events you can choose from here.

In our case, we want to hear about both "Pull request created" events, and "Pull request merge commit created" events - i.e. when there's a new PR, and when one has been merged. Unfortunately you can't choose both at the same time, so we had to go through this configuration twice. Not a big deal - it's a once-off configuration and only takes a couple of minutes.

Once done, going back to Connectors (from the channel ellipsis) allows us to see the connectors we've configured.

The end result

When a new PR is submitted by anyone in the team, there's a new message in the channel - complete with a direct link to view the PR!

I'm excited to announce that I'm joining Microsoft as a Sr. Cloud Developer Advocate in the Cloud and Enterprise group!

In a seriously ego-boosting Twitter DM, Donovan Brown asked whether I'd be interested in the role and encouraged me to apply. I went through the process, and I'll be starting on his team next week, on the 7th of June.

What about Octopus?

I joined Octopus Deploy early in 2015 as a Solution Architect. When I joined, I was one of only six in the company, but since then the company has grown to about 25 and my role started revolving more around the community. I also wrote our extension for Visual Studio Team Services.

I love Octopus, and leaving wasn't in the plan at all, but this new opportunity was far too good to pass up. I consider myself extremely lucky to have worked with such a dedicated and smart group of people for over 2 years. I have no doubt that Octopus will continue pushing out awesome software.

While Octopus occupies a commanding position in the world of deployment automation, Microsoft (through VSTS and Azure) has a set of tools and services for the whole DevOps lifecycle. I'm looking forward to talking to a broader audience about everything from requirements gathering to monitoring in production.

However... you can still expect me to recommend Octopus as part of that process. :)

What will I be doing?

It's been made clear to me that my goal at Microsoft is to help people succeed. Not to help them succeed with a Microsoft-only stack. Not even to help them succeed with a mainly Microsoft stack. If Microsoft tech can be used anywhere in the pipeline, I'll try to help out.

This is really important to me, and is another example of Microsoft putting their money where their mouth is. Gone are the days of Microsoft lock-in - today it's all about competing on merit, enabling any platform, and open-source. The Microsoft of ten years ago didn't excite me, but that's changed dramatically.

As for my day to day, you can expect to see me at more conferences and events. I'll be creating useful content to help people succeed, and hopefully visiting a few customers directly. It'll take me a little while to settle in, but feel free to reach out on Twitter if there's something I can help with.

Microsoft is hiring, so if you have a passion for developer advocacy, maybe you'd be a good fit as well!

It's easy to decide you want to get away from long, painful, weekend deployments, but it takes much more than just a decision. For legacy code (whatever your definition of legacy might be), there are likely some things that prevent you from deploying fast and frequently.

In my previous Brownfields posts I talked about DevOps as a Culture and changing the dev process. In this post, I'll talk about some practical changes you can make to your codebase and how you work with it. These changes will make it easier to deploy regularly and without fuss.

One codebase

An application should only have a single codebase, and only one branch that you deploy to production from. There are two common practices that get in the way of this:

1. Separate copies of code for different scenarios
2. Long-lived branches

Code for different scenarios

This is a surprisingly common practice. I've worked with organisations in the past that had different, almost-the-same codebases for different flavours of the application. They could be separated based on country, set of features, or individual customers.

The common thread is that at some point there was a diversion based on two seemly unreconcilable differences. Rather than a larger effort to refactor, two versions emerged. The result is that it's incredibly difficult to apply changes and bug fixes to all versions consistently and safely.

Long-lived branches

It's also common to separate versions of code into branches. This can be a way of implementing the previous practice (code for different scenarios) that gives the feeling that it's the same codebase. It can make changes a little easier to propagate across versions, provided the versions are similar.

It's also common to have separate long-lived branches for different environments. The team writes code on a dev branch which then gets merged into staging for a test deployment, then to production for production deployment.

In my opinion, this is a mistake. Each time you need to deploy, you're merging and ending up with a brand new codebase. That means when you deploy, it's the first time you've ever deployed that code.

If a successful deployment relies on a clean merge, that should scare you!

So how do you get to a single codebase? Well, it's likely to require (sometimes significant) refactoring.

Identify and isolate variations

There will invariably be variations between deployments of your application. Whether they're environmental (test vs production) or otherwise (e.g. customer configuration).

It's important to find these differences and isolate them. Rather than if statements scattered throughout your code, properly independent classes or assemblies make it much easier and safer to make changes to your software without unexpected side-effects.

Dependency injection is great for this, but of course you don't want to have to change the code that resolves your dependencies each time you compile for a different scenario. You need to be able to change your configuration without recompiling.

Externalise configuration

To avoid having to recompile for each different deployment scenario, it's important that your configuration is outside the compiled application.

That means you can deploy the same binaries and compiled code, but change the configuration based on the environment you're in, or the customer you're deploying to.

It's really up to you where you keep your configuration. .NET developers tend to keep config settings in .config files, but environment variables or even external databases are reasonable alternatives. As long as you don't have to recompile to deploy to a new location!

Get your versioning right

Finally, it's extremely important that you can easily distinguish between different compiled versions of your software. The usual way to do this is with versioning.

Because there's only a single codebase, and we only deploy from the master branch, we only need one version number. At least for production candidates.

Following Semantic Versioning is an awesome way to manage your versions, but if you can't do that, you just need to follow two rules:

1. Each build has a new version number
2. New code has a higher version number

These two rules mean that given any two builds, you can easily determine whether they're the same, and if not, which is newer. Remember, we only have one branch, so we're comparing like with like!

Summary

There are changes you can make to your code and the way you work with it that will make deployments easier.

1. Get to one codebase
2. Identify and isolate variations
3. Externalise your configuration
4. Get your versioning right

Once deployments are easier, you can feel confident doing them more often, which will ultimately lead to a better DevOps story!

There are a lot of options for connecting with people online these days, and I use many of them. To differing degrees, the main ones I use are Twitter, LinkedIn, Facebook, Email, Skype, and Slack.

So which do I use for what purpose, and why did I only supply links for two of them?!

Over the past few years, I've started being more careful about the contacts I have through these channels. If you've asked me to "add you" on a particular channel, I might have ignored you or refused. This post explains why.

Note: There's a chance I'll come across like an arsehole in this post. My intent is to explain, not offend. But if you think I'm doing it wrong... uh, I'm sorry you feel that way.

Twitter

I use Twitter a lot. My profile and tweets are public, and I tweet a mix of personal and work-related stuff. I'm really happy for anyone to follow me. Seriously, go do it.

I respond to most @ tweets, but fair warning - if an account is abusive or a bit too spammy, I won't hesitate to block and report.

Twitter is almost certainly the best way to connect with me.

The accounts I follow are extremely varied as well. There are a lot of tech people, but also family, sports people, companies, parody accounts - basically anything I want to read. If I don't follow you and you think I should, let me know! I might! But I also might not. Please don't be offended.

LinkedIn

I rarely use LinkedIn. I'll keep my profile more or less up to date, but blog posts happen on my blog, and I'll post interesting links to Twitter.

I do have a lot of LinkedIn contacts, but I'm pretty strict on who I "connect with".

If I've annoyed you by "not connecting", there's a good chance it was on LinkedIn.

As a rule, I don't connect with recruiters. No, I'm not interested in the job you think I'm perfect for because my profile matched the keywords in your search.

I also don't connect with people I haven't met. It's great that you have a similar job as me, or you like the same things, but I treat LinkedIn as a place to keep track of people I actually know.

Facebook

I use Facebook frequently, but only for friends and family. Luckily Facebook friend requests are few and far between. I'll let you read into that what you will.

The overlap between business connections and Facebook friends is small,
and I intend to keep it that way.

Occasionally, work-related stuff will creep onto Facebook, but I'm conscious that most of my Facebook friends, well, couldn't care less. So it doesn't happen a lot.

Email

I welcome emails! So why didn't I provide a link to my email above?

It's not hard to find an email address for me, and I'd prefer if you did a little bit of work. It indicates that you actually wanted to contact me (and not that you spammed an address you found somewhere). It's a coarse-grained filter. Of course, if I meet you in person and you ask, there's a good chance I'll give it to you.

If I haven't given you my email address,
it's not hard to find if you actually want to contact me.

I've set up a lot of email addresses. Ultimately they filter into the same couple of accounts, but into different folders. Some I care about deeply, some not so much. Some only exist so I can see who sold my details.

For that reason, I'll check the publicly-available ones a lot less than the others.

Of course, just because I welcome emails, doesn't mean I'm great at returning them. By its very nature, email is asynchronous, so I might not reply to you today. Or tomorrow. Or this month.

Skype

Skype is great for voice and video conversations. That means if we're connected on Skype, it's almost certainly because we needed to see and/or hear each other at some point in the past.

My Skype contact list consists almost exclusively of people
I previously needed to have a conversation with.

Unless we need to speak remotely, I won't add anyone on Skype. For me, it has one purpose, and making new friends isn't it.

Slack

At time of writing, I'm up to eleven Slack Teams. Yes, it feels like a lot, and yes, they're sometimes a pain to manage. They're all tech-related, and I'm very unlikely to join a Slack Team unless it has relevance to me right now.

Each Slack Team has a purpose,
but if we're on the same Slack Team, let's chat!

I find them extremely useful. For example, the Octopus Slack Team is how I communicate with the rest of the company about 99% of the time.

Interestingly, I'm not in any public teams. All are invitation-only, and I only manage one of them myself.

Summary

So that's how I treat my social media accounts, and why I might have ignored your "friend request" or "network connection". Don't take it personally!

I like to meet new people in person, not online.

I'm friendly, I promise. So come say hi to me at a conference, or ping me on Twitter!