Reducing ASP.NET Deployment Interruptions With Azure DevOps

Increasing developer effectiveness gives you a competitive advantage and helps you meet SLA. In this blog post Janis Veinbergs will discuss how to improve the deployment process from the traditional file copy to something more effective, automated, and low friction while causing no downtime to end-users.

Janis Veinbergs is a developer at Data Center Operator DEAC with over 13 years of IT experience. As a joyful father of four, he says he enjoys learning, debugging, and delving into the inner workings of computers.

---

Our customer, a nationwide school management platform, was facing problems deploying a rather large classic .NET Framework application to 22 Windows IIS servers.

The webroot (and IIS Shared Configuration) are actually served from a single Windows share, which enabled a low-effort solution to simply copy files to a single location and keep them in sync with the IIS configuration.

That's how we'd been doing it for years, and it was obvious that something had to change.

Issue With ASP.NET Deployment

The main issue is that whenever .NET resources (.aspx pages, razor views) change, they are compiled to assemblies on the initial request. When IIS is constantly flooded with requests to all potential resources, they must all be assembled at the same time on all IIS servers. When deployed at night, however, the delay was minimal because there were few requests, and those that did, compiling was completed rapidly.

Main Pain Points

• Deployments would be taken at night
• If a hotfix had to be issued at day with high load - it could take up to 30 minutes of complete system halt as IIS server farm compiles resources while pile of requests waits in a queue.
• The deployment is not atomic - that is, while files are being copied between v1 and v2, we would get v1.314 - while some files are updated, the others not.
• If we ever wanted a rollback - occasionally files would be locked, exacerbating already bad situation.
• The website itself consists of ~220MB of data across ~2500 number of files. And small file copy is SLOW.

---

PRO TIP: ASP.NET Requests Queued is a good performance counter for monitoring to identify when user browsers are just waiting for response and how quickly situation resolves after an update. A healthy value is 0 or something close to that.

---

We considered utilizing robocopy to copy updated files depending on timestamp, but that wouldn't work for us because the code would be deployed by several people. When Visual Studio publishes the project, the timestamps of files on different workstations diverge. Rsync would most likely be beneficial, but it would be difficult to implement on developers' Windows computers and the Windows share server. Furthermore, if different hosts generate files, minor changes in build techniques may result in checksums that differ.

Deployment Strategy

We decided that we no longer needed to rely on a file share.

We've already been utilizing Azure DevOps for various purposes. So we came up with a new deployment concept:

• Keep files on IIS local disk.
• Azure DevOps Agent would be the one compiling code.
• We would precompile views in advance. Locking wouldn't be an issue as we will copy within a new folder.
• Orchestrate deployment to all IIS servers via Azure DevOps Release Pipelines.
• Run all .dll files through ngen. That would eliminate additional delay when assembly is first loaded. As C# code compiled is actually MSIL - an intermediate language that CPU doesn't understand any of that. JIT compiler is the one that translates that to bytecode for the processor architecture at hand. And it does so usually Just-In-Time or On-The-Fly. Or, in our case, ngen (Native Image Generator).
• A deployment task would have to copy files within a new folder and then we would change IIS Physical path from where to serve site files.

Implementation

Build Publishing

Let's get one thing straight before we go into build and deployment: we use the Azure DevOps online service. This might be an on-premises installation or any other CI/CD platform of your choosing. Nevertheless, we use self-hosted agents to complete the build processes. We are talking about solution, which consists of multiple .NET projects.

When writing Azure DevOps pipeline, Janis had to figure out what kind of MSBuild properties he has to use to invoke the "Publish" process. So, after searching the web, reading MSBuild diagnostic output logs, reading MSBuild .target files, he'd come up with properties he needs.

The relevant "Publish" command within Azure Devops build pipeline:

- task: VSBuild@1
  displayName: Publish Web.Application  
  inputs:
    solution: 'Web.Application'
    msbuildArgs: >
      /t:Build,GatherAllFilesToPublish
      /p:PublishProfileName="$(publishProfileName)"
      /p:WebPublishMethod=FileSystem
      /p:DeleteExistingFiles=true
      /p:DeployOnBuild=true
      /p:MvcBuildViews="${{ parameters.Precompile }}"
      /p:PrecompileBeforePublish="${{ parameters.Precompile }}"
      /p:WDPMergeOption="MergeAllOutputsToASingleAssembly"
      /p:SingleAssemblyName="Web.Application.Precompiled"
      /p:UseMerge="true"
      /p:DebugSymbols="True"
      /p:EnableUpdateable="False"
      /p:PublishUrl="$(Build.BinariesDirectory)my"
      /p:WPPAllFilesInSingleFolder="$(Build.BinariesDirectory)my"
      /p:RunNpmScripts="$(runNpmScripts)"
      /p:AutoParameterizationWebConfigConnectionStrings="false"
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    msbuildArchitecture: x64
    logFileVerbosity: detailed

---

The full build publishing can be seen under DEAC European Data Center Operator technical blog: Bringing down ASP.NET deployment interruptions from minutes to blip of a time with Azure DevOps

---

Release Deployment

Once we have the artifacts ready (archive of website files), we are ready to implement Release.

The release must copy given files to other directory and instruct IIS to change base path from where it will server files. We had 2 options to choose from:

1. Make Build agent connect to IIS servers and perform the deployment on each IIS server. In this case, we have to write some script that will copy files onto each server and issue some commands to IIS. Moreover, we must control/see whether deployment is successful or not. And the build agent could actually be in another domain, with no direct access to production.
2. Install Deployment agent on all target IIS servers. Every host then receives deployment job and does whatever it is instructed to do. We don't have to bother about any other remote communication channel other than Deployment Agent with Azure DevOps server over 443/TCP. Plus we get nice UI of seeing whether deployment succeeded, partially succeeded (if partially, which hosts failed) and on which step it failed.

We went for the second option and split some operations into multiple steps:

1. Root App Pool - ensures appropriate application pool is created on IIS. It is actually a one-time step and may have been created beforehand.
2. Copy my files - PowerShell script to extract 7z files.
3. Ngen my - runs ngen.exe on all .dll files found within deployment folder. Also using custom condition so this step which may take a little more than a minute, could be turned off.
4. Deploy my - Switchers virtual directory, issues some IIS configuration commands. When this stage is completed, IIS servers new code.

The great thing is that if any of steps fail for ANY IIS server before Deploy step, Deploy won't run for ANY IIS server and they will all be consistent.
In case of a rollback, we can open appropriate release, and for "Deploy my" stage - press Redeploy. IIS will immediately switch back to the appropriate folder.

Solution Result - Automation

Everything is set up so that when the production branch receives new code the build runs automatically.
After build publishing, deployment occurs automatically but is halted pending permission.
When we click the Approve button, the code is deployed, and we simply receive a notification within the corresponding channel using the Azure Pipelines Slack app.

Note: This does not include database updates, which still must be performed manually, but DACPAC deployment can eventually be put into a pipeline too. Fortunately, database updates are less often than backend code/frontend updates.

---

Did this information inspire you to improve your developer effectiveness in order to gain a competitive advantage?

Please let us know.