The following article was written specifically for GitHub, but some of these techniques could be applied to other Git hosting platforms as well.

How AppVeyor is cloning private repos

AppVeyor uses SSH to clone private Git repositories. When you add a project in AppVeyor a new RSA key-pair is generated which consists of private and public keys. Public key is deployed to a remote Git repository using GitHub (or BitBucket) API and private key is pushed to build worker during the build. For SSH protocol to work on Windows private key should be located in %USERPROFILE%\.ssh\id_rsa file.

The problem with private sub-modules

Git has submodules support and this is a wonderful tool for organizing large projects or reusing some code. While building your solution on AppVeyor you need to checkout sub-modules as part of your build. Well, you can use the following command during install phase which occurs between clone and build:

git submodule update --init --recursive

The problem arises when sub-modules refer private Git repositories which cannot be cloned without authentication and as a result you get stalled build. This is because sub-module repository does not contain SSH public key used to authenticate main repo, so Git is asking for credentials:

The solution

A custom SSH key could be used to checkout repository private sub-modules. The rest of this article explains how to generate SSH key and setup AppVeyor project to use it.

Check sub-modules path

First of all you have to check sub-modules URLs in .gitmodules to make sure they are in SSH format. For GitHub it should be something like:

url = git@github.com:{owner}/{repo}.git

Generate SSH key

Now, let’s generate a new SSH key that will be used to fetch sub-modules.

In command prompt type the following command:

ssh-keygen -t rsa

When prompted enter key file name, say submodules and empty passphrase.

ssh-keygen.exe utility is part of Git installation for Windows and is typically located in C:\Program Files (x86)\Git\bin directory.

In the current directory you’ll find two files: submodules which contains private key and submodules.pub with public key.

Add SSH public key to GitHub

If you have only one sub-module in your main repository you can add public key directly to sub-module repo, however if there are multiple dependencies GitHub won’t allow you to add the same key again.

Open submodules.pub file and copy its contents to clipboard.

Navigate to SSH Keys under your GitHub profile and add a new SSH Key with contents from clipboard and any title.

Configure AppVeyor project to use SSH key

Next, during the build on the worker machine we have to put private key contents into %USERPROFILE%\.ssh\id_rsa before running git submodule update --init --recursive command.

We’ll store contents of private key in environment variable.

UI

Open “Environment” tab of project settings in AppVeyor and add a new environment variable called priv_key. Open submodules file with private key and copy base-64 body of the key between -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- into clipboard:

Paste contents of clipboard into value field of environment variable. New lines will be changed to spaces - that’s OK - we’ll turn them back to new lines with PowerShell script shown below.

Mark variable as “secure” by clicking “lock” icon next to it - this will prevent it from being decoded during pull requests (see explanation below).

In Install script field paste the following code:

$fileContent = "-----BEGIN RSA PRIVATE KEY-----`n"
$fileContent += $env:priv_key.Replace(' ', "`n")
$fileContent += "`n-----END RSA PRIVATE KEY-----`n"
Set-Content c:\users\appveyor\.ssh\id_rsa $fileContent
git submodule -q update --init --recursive

appveyor.yml

Copy the contents of private key to clipboard as shown above and open Encrypt data tool in AppVeyor. Encrypt the value of clipboard using that page.

Add this to your appveyor.yml:

environment:
  priv_key:
    secure: <encryped-value>

install:
  - ps: $fileContent = "-----BEGIN RSA PRIVATE KEY-----`n"
  - ps: $fileContent += $env:priv_key.Replace(' ', "`n")
  - ps: $fileContent += "`n-----END RSA PRIVATE KEY-----`n"
  - ps: Set-Content c:\users\appveyor\.ssh\id_rsa $fileContent
  - git submodule update --init --recursive

Security considerations

“Secure” variables means you can safely put them into appveyor.yml that is visible to others. Other than that they are just regular environment variables in a build session that could be easily displayed in a build log by simple Get-ChildItem env:.

However, secure variables are not decoded during Pull Request builds which prevents someone from submitting PR with malicious build script displaying those variables. In more controlled environment through with a trusted team and private GitHub repositories there is an option on General tab of project settings to allow secure variables for PRs.

If you accidentally submitted any sensitive information into public repo or displayed it in a public build log don’t wait - invalidate/change/re-generate that data immediately!

We are continuously improving AppVeyor platform and doing a couple of changes/deployments during the week. To give you more information about ongoing/upcoming changes and status updates we introduced a new “Technical updates” mailing list. It will be sent approximately two-three times a week.

All existing customers can subscribe to this mailing list on Profile page. If you decide not to subscribe to technical updates you’ll still be receiving this monthly newsletter.

Now, back to deployment improvements. Deployment has always been a strong part of AppVeyor and we are committed to make AppVeyor a single shop for your entire continuous delivery. Also, YAML configs worked amazingly well for AppVeyor customers and we continue to invest into this area with a new features based on your feedback.

New SQL Database deployment provider

Your AppVeyor builds may produce SSDT packages (.dacpac files) describing application database changes. Publishing SSDT project from Visual Studio is a trivial task, but it’s always been a challenge of doing that on a build server. Most common tools for synchronizing DACPAC packages were SqlPackage.exe and MSDeploy.exe with built-in DacFx provider.

Now AppVeyor offers a new SQL Database deployment provider for incremental publishing of SSDT packages to a local SQL Server instance, remote SQL Server or Azure SQL databases.

SQL Database provider uses SQL Server Data-tier Application Framework (DacFx) and as most of AppVeyor deployment providers it can be used during the build for staging deployment as well as a new “environment” for production deployments. Read more

SFTP support

We added SFTP (SSH File Transfer Protocol) support into FTP deployment provider. Don’t mess it with FTPS which is also supported - it’s a completely different thing though it organically complements a new “unified” FTP provider. Read more

Install MSI packages with Deployment Agent

With the improved AppVeyor Deployment Agent it’s now possible to install MSI packages on staging and production environments behind the firewall. With MSI added you can use Agent to deploy various types of workloads: web applications, windows services, console apps, SQL Databases and MSI packages. Read more

New GitHub Releases provider

This is definitely a great news for open-source projects hosted on GitHub and using AppVeyor for their CI! GitHub deployment provider allows to publish build artifacts as assets to your repository release. Read more

YAML configuration validation

We re-factored appveyor.yml configuration parser to make it work in “strict” mode, so you get immediate feedback if there is something wrong with project config and as a bonus there is a new page for validating appveyor.yml instead of try-and-fail process:

Export project configuration in YAML

You can easily switch your projects to YAML and benefit from portable and versioned configuration. There is a new tab on AppVeyor project settings which allows you to see how project changes made through UI would look in appveyor.yml:

New REST API for configuring project with YAML

It’s been a challenge to configure project settings through REST API as their request/response JSON format was, well, derived from UI and not suitable for processing by humans. With all these parsing and exporting improvements in YAML config we also added two new API calls: get project settings in YAML and update project setting in YAML.

Holidays are coming and we would like to wish all our customers more green builds and less bugs! Merry Christmas and Happy New Year!

SVG is great! We’ve just added a small, but very neat feature that allows you customizing badge text.

This looks really great for batch-specific status badges put in one line (these are statuses for Grunt.js project):

To customize SVG badge titles for pending, failing and passing states add pendingText, failingText and passingText query parameters respectively.

For example:

https://ci.appveyor.com/api/projects/status/32r7s2skrgm9ubva?svg=true&passingText=master%20-%20OK&failingText=master%20-%20Fails

Read more about status badges in AppVeyor documentation.

Enjoy!

Based on feedback from our customers we’ve been working on improving AppVeyor performance and got amazing results!

New super-fast environment

For the last couple of months we’ve been experimenting with running builds on new “Premium” environment. It’s based on Hyper-V and hosted on a dedicated hardware with SSD drives and faster CPUs.

We moved most of our existing customers to this new environment and they were very satisfied with the results. Builds start almost instantly, run 2-3 times faster with greater stability!

We still have Azure environment for open-source projects and “Basic” plan.

New Pricing

With the introduction of the new environment we decided to review our plans once again to make them more flexible for companies with different business needs.

There is a new entry-level plan for individual developers and small teams with 1 private project and 1 concurrent job building on Azure. There is an upgraded “Pro” plan now with unlimited number of projects and super-fast builds. For those teams actively using AppVeyor new “Premium” plan now offers 3 concurrent jobs on fast environment.

Also, we introduce yearly pricing for “Pro” and “Premium” plans giving you 2 months free!

All existing plans are honored.

If you are a student, educational organization or open-source project looking for more calculation power or concurrent jobs we provide 50% discount on all plans.

AppVeyor on-premise

You may have noticed that we don’t have “Enterprise” plan anymore. This is because “Enterprise” is reserved for AppVeyor on-premise edition that will be available in January 2015!

If you are interested to be a beta tester just reply to this message and we’ll add you to the “AppVeyor Enterprise early bird” mailing list. We’ll be publishing more information and roadmap for on-premise in the coming weeks.

Updated website

AppVeyor has gathered many great open-source projects, such as Mono, Julia, Grunt, Redis, nodegit, Chocolatey, JSON.net just to mention a few. People contribute their priceless knowledge and experience on AppVeyor forums.

To more actively engage the community in shaping AppVeyor we decided to host our entire website with documentation on GitHub where everyone could contribute by sending a pull request.

See https://github.com/appveyor/website.

The website runs on a new Jekyll-like engine (we called it NJekyll), so you can grab it and use for your own website :)

Chocolatey is a wonderful tool that allows you installing your favourite programs with a single command. Unlike regular process of installing software with interactive setup package where you keep clicking “Next”, “Accept”, “Finish”, etc. Chocolatey does the job without questions asked. By analogy from Linux world Chocolatey is a package manager for Windows.

Chocolatey is great when you setup your development environment and it’s especially great for installing custom software during the build process on AppVeyor! As you know AppVeyor offers fully-customizable build environment where you have admin rights on build machines. For example, if you need to install MongoDB for your integration tests you can do that with the following command (the latest version of Chocolatey is already installed on AppVeyor build workers):

choco install mongodb

The list of software that can be installing with Chocolatey is huge and it’s growing fast. The catalog is community-driven and you can contribute your own packages or update existing ones. Behind the scene Chocolatey uses feed of NuGet packages. Each package contains install.ps1 and uninstall.ps1 PowerShell scripts. If it’s so-called “portable” package application files are stored along with scripts or if it’s “native” package application MSI is downloaded from the Internet and silently installed.

So, Chocolatey is well-established thing, but Chocolatey team want moving it to the next level and making Chocolatey an alternative Windows Store! They created a KickStarter campaign and are asking for our support! Let’s help Chocolatey to do an open, community-driven Application Store for Windows.

Let’s get Chocolatey!