Auto Build/Deploy Server using NodeJS and Git Hooks for Web

Introduction

In an agile and rapid software development environment, it is hard to keep control of Quality Assurance(QA) and staging builds when you have too many projects running and several stakeholders (other developers, QA team, Account Managers, Senior Management, Project Managers, Designers ...etc) who need to check, verify and validate builds before moving to client environments. With  manual deployment, someone who has access to the staging servers has to create a build package and perform the deployment procedure -  this can be different depending on the web project type. But what if (s)he were not available? What if there were too many projects that would consume a lot of unnecessary time?

You can give more people access to the staging servers, developers for instance. But what if developers made an urgent hotfix directly on those servers for other stakeholders to check and then forgot to update the code repository with that code?

We built a NodeJS application that acts as a build/deploy composer that theoretically can handle any type of web solutions and can run on any type of Operating System: Microsoft Windows, Mac OS or Linux. We called it a composer because it is not a native build server for a specific technology, it is rather hybrid and combines tools from different technologies to build the solutions and deploy them.

Breakdown

The diagram below shows, on a high level,  the main components of the approach and how they interact together.

To put a little context to the diagram, we highlighted each step in this sequence:

• The main element is the Git Hook, which sends a signal to our NodeJS application whenever a repository is updated.
• The NodeJS application validates the signal by doing a couple of security checks, then it parses the signal and compares it against the application's configuration.
• Configuration primarily contains information about the repositories you want to keep track of to automatically be built and deployed.
• Depending on the type of the repository and, hence the technology, the NodeJS application will run a specific command that contains instructions to build and deploy the files.

The NodeJS application will be running on your staging servers, wherever they are, whether they are local or hosted on an online service. The only requirement is that the NodeJS application should have a publicly accessible URL.

Git Hooks

Git hooks are a facility that Git repositories provide as a way of communicating updates for integration with external parties. Some hooks are preconfigured by the hosts to communicate and integrate with specific systems such as Campfire, Grove or Twitter. A very famous  hook is email, which comes without any configuration. If you choose to watch (or follow) a repository for instance, you get email notifications to the email address registered with your Git account.

In addition to the preconfigured Git hooks, there is a POST hook that is generic and could be consumed and used as suited for your business(this is the one we used). POST hook does an HTTP POST request that contains detailed information about a Git update in HTTP form encoding that could be received by any application.

For instance, if you host your code repositories on BitBucket, you can configure a POST hook by following these steps:

• Go to your repository's settings (your account must have administrative privileges in order to access settings);
• Under the Integrations section choose hooks;
• In the hooks view you will find a list of hooks, search for POST and choose Add;
• In the prompt box enter the public URL of the NodeJS application in this format: http://username:password@IP_or_hostname:port;
• That is it, you are all set.

In the link above, username and password are optional. If you wish to use basic HTTP authentication to have more security,  you can enter the credentials there which you will validate in the NodeJS server. The port field is also optional depending on your URL.

NodeJS Server

The NodeJS server acts as the middleware that connects your Git actions to the staging environment. It consumes the Git Hook then receives, decodes and parses the posted data. After processing, it determines which command to execute depending on the type and technology of the code repository.

The first thing NodeJS server does is the security checks:Check the incoming HTTP request verb. It must be always POST, otherwise it throws an HTTP verb not allowed, 405 HTTP codeCheck the source IP addresses against the whitelisted IP's that are allowed to send data to the NodeJS server. Those IP address must be preconfigured in both your staging server's firewall and the NodeJS application. Note that those IP could be changed so you might need to keep an eye on them. If the source IP was not recognized on the NodeJS application level, it will return an unauthorized HTTP error code, 401 HTTP codeCheck the HTTP basic authentication credentials that come within the HTTP POST request, which you enter when you setup the Git Hook in your repository settings. If provided credentials are incorrect the NodeJS application throws an unauthorized HTTP error code, 401 HTTP code

Once all security checks are validated and passed, the NodeJS server proceeds by parsing the POST data using an application/x-www-form-urlencoded parser.

You can use NodeJS modules to simplify your application logic like express. body-parser, basic-auth ...etc.

Configuration

Configuration data is the backbone that controls the flow of the server. It is a JSON object that is stored in the server and given no public access. It contains mainly two types of data (in addition to anything you want to keep configured).

Server-specific configuration such as flags - e.g. Great if you want to check for HTTP basic authentication or not (with allowed credentials) or check for whitelisting or not (with whitelisted IP addresses), ...etc. It also contains the local paths of build/deploy commands specific to each supported technology.

Repository-specific configurations -  these contain an array of repositories that you need to automate their build and deployment. For each repository object you need:

• Repository name, which needs to be similar to the repository's name on the Git host. The NodeJS server will check the name provided in the POST data (name if slug field) against the name of the repositories in the configuration, and if the name was not present the server should ignore the request. This field is the key of each object in the repositories array.
• Repository path, which is the local path of the repository's folder on the staging server. The NodeJS server will perform a Git pull in the specified path.
• Solution application path, which is the local path of the web folder where the application runs and needs to be deployed.
• Staging branch name, which is the name of the branch that you want to track, build and deploy. It is a best practice to always have a development branch, staging branch and production branch as a minimum, but it is different depending on your business needs. The NodeJS server will pull the specified branch commits from the repository, and if the request was originated from a commit on another branch, the NodeJS server will ignore that request.
• Repository type, which is the technology used in the repository; e.g. .NET, PHP, JavaScript, HTML5 ...etc. Depending in this field, the NodeJS server will decide which command needs to be executed in order to build and deploy the solution.
• Additional fields could be added depending on the technology. For instance, in .NET web solutions you need to provide the full local path of the solution or project file.

Commands

Commands are script files that contain instructions related to a specific technology that build and deploy a web solution. Those instructions are written in an Operating System specific language - in Microsoft Windows you can use command line or PowerShell, for MacOS and Linux you can use bash scripts.

For instance, you can build and deploy .NET web solutions using Microsoft Build Tool with command line script using this instruction (executed after the Git pull request succeeds): msbuid_path project_file_path_in_repository "/p:Configuration=staging_configuration" /p:DeployOnBuild=true "/p:PublishProfile=staging_profile". You can use additional parameters as well to specify how you want the build log to be handled. The configuration and publish profiles has to be present in the repository (they can be created in development machines using Microsoft Visual Studio) and they help determine the publish (i.e. deploy) path, the build configuration that would publish the configuration files (i.e. web.config) in the proper transformation.

The command script's instructions should be written in a way to read parameters like the project file path from input arguments, and those arguments will be read by the NodeJS server from the configuration file and passed to the execution command as arguments. You can use the child_process NodeJS module to handle executing the command scripts.