Introduction

What is Rusty-CI?

Rusty-CI is just a user interface for buildbot. Instead of having to write the Python for your CI project to get the versatility you want, just fill out Rusty-CI's template YAML file and it'll do the rest for you.

Suggestions

I highly recommend running this in some sort of container. All features of Rusty-CI are tested using fresh Ubuntu 18.04 linux containers.

Usage

Here is the template YAML file that Rusty-CI will output for you to fill out with your own data.

It explains itself for the most part, but I'll be covering it in more detail in a later chapter.

# The required of Rusty-CI to build this CI
requires: x.x.x

# This section holds data specific to the master of the workers
master:
  # The title subsection of the master holds the title of your web gui
  title: "Rusty-CI"
  title-url: "https://github.com/adam-mcdaniel/rusty-ci"

  # This is the ip of the web-gui
  webserver-ip: localhost

  # This is the port of the web-gui
  webserver-port: 8010

  # The address of your repository
  repo: "https://github.com/adam-mcdaniel/rusty-ci"

  # The number of seconds to wait before checking for updates on your repository
  # Two minutes is a good poll interval
  poll-interval: 120

# This section holds data specific to the handler that will look for
# pull requests / merge requests on your repository
merge-request-handler:
  # This is basically the website you're using for version control
  # Right now, github and gitlab are the only supported sites
  # If you're using an unsupported version control system, no worries,
  # rusty-ci just wont run on pull requests.
  version-control-system: github
  # The username of the owner of the repository
  owner: adam-mcdaniel

  # The name of the repository
  repo-name: rusty-ci

  # You dont want to run arbitrary code on your machine when anyone
  # makes a pull request. Rusty-CI will not test anyone's pull request
  # if their username is not in this list.
  # Note that this has no effect on GitLab merge request building!
  # Rusty-CI will only build merge requests from a branch
  # thats already inside the repository.
  whitelist:
    - adam-mcdaniel


# This section holds each worker
# You can have as many workers as youd like, just be sure to fill out
# each of their fields out properly.
workers:
  # The name of this worker is `test-worker`
  test-worker:
    # The ip of the master
    master-ip: localhost
    # The worker's files will be installed in this directory.
    # This can also be an absolute path
    working-dir: 'test-worker'


# This section holds each scheduler.
# Like the workers section, you may have as many schedulers as youd like.
schedulers:
  # Create a scheduler named `ci-change`
  # This scheduler will trigger the `rusty-ci-test` builder whenever it
  # detects a change in a yaml file for any branch.
  ci-change:
    # This scheduler triggers the `rusty-ci-test` builder.
    # You can put as many builders as youd like here, and the scheduler will start them all.
    builders:
      - rusty-ci-test

    # This will make the current scheduler run if the "your-scheduler-name-here"
    # has run successfully. You can only put one scheduler name in this section.
    # depends: "your-scheduler-name-here"
    # IF YOU USE THE `depends` SECTION, YOU SHOULD REMOVE OR COMMENT THE FOLLOWING SECTIONS
    # Using the depends section will ignore the `branch`, `triggers`, and `password` sections

    # This is a regular expression that matches a branch.
    # If there is a change in a branch whos name matches this regex,
    # it will be checked by the following triggers section.
    # THIS WILL ONLY USE THE FIRST REGULAR EXPRESSION IN THIS SECTION TO MATCH THE BRANCH
    branch: ".*"
    # If a change has occurred in a branch that matches the regex in the branch section,
    # Then the files that were changed are matched against the regular expressions in the
    # triggers section. You can have any number of regular expressions in the triggers section.
    # If any one of them matches the name of a file that was changed in a matched branch,
    # then the builders in this scheduler's `builders` section are executed.
    triggers:
      - '.*\.yaml'
      - '.*\.sh'
      - ".*Makefile"
    # The password a whitelisted user can comment on a merge / pull request
    # to mark it for testing; that is if the pull request was made by a non-whitelisted
    # user. If the pull request was made by a whitelisted user, it is automatically run.
    password: "ok to test"

# These are the builders that are executed by the schedulers
# Each has its own specific task that is delegated to one or more workers
# When a builder is run, its script is run on the command line.
# You can have as many builders as youd like as well.
builders:
  # The name of the builder is `rusty-ci-test`
  rusty-ci-test:
    # This is the shell script that the workers will run when this builder is executed
    # You can have as many instructions as youd like
    # Mind you, you cannot use the |, >, <, >>, <<, etc. operators. Sadly, buildbot
    # passes each item separated by whitespace as another parameter to function.
    script:
      - echo Hello world!
      - echo Im an instruction in a script!
    # These are the workers to delegate this build job to
    workers:
      - test-worker
    # The repo to refresh from before running
    repo: "https://github.com/adam-mcdaniel/rusty-ci"

Installation

Install Python3

You're gonna need python3 to use Rusty-CI.

To install on a *nix system, run the following commands.

apt update -y && apt upgrade -y
apt install -y build-essential python3-dev python3-pip python3-venv

You will need to be able to use python3 -m pip and python3 -m venv.

Install Rust

To install Rusty-CI, you must install Rust. You if you're on a Unix like platform, can do so by running this shell command.

curl https://sh.rustup.rs -sSf | sh

If you're on Windows, go to the rust website. You'll need to download rust-init.exe and follow its instructions.

Install Rusty-CI from Crates.io Package Registry (recommended)

Now that you have Rust, you should be able to install Rusty-CI by running the following command.

cargo install -f rusty-ci

This will automatically add Rusty-CI to your path, so we should be all done!

Build from source (not recommended)

If you don't want to install Rusty-CI from the package registry, you can always build Rusty-CI from source. Here's how you would do so.

git clone https://github.com/adam-mcdaniel/rusty-ci

cd rusty-ci
# Cargo will output the executable to ./target/release/rusty-ci
cargo install -f --path .

This will automatically add Rusty-CI to your path, so we should be all done!

Problems Installing?

If you run into some problems installing Rusty-CI, or if Rust can't find cc, you need to upgrade some of the packages on your system.

Here's the commands I run to solve these errors.

apt update -y && apt upgrade -y
sudo apt install build-essential

After running these commands, try running cargo install rusty-ci again.

Can't find Cargo or Rusty-CI?

Try running the following to add Cargo and your installed crates to your environment's path.

source $HOME/.cargo/env # Add `cargo` to your path

Still having problems?

If you're still having some serious problems, post an issue on the repository.

Usage

This chapter covers

• The features and usage of the rusty-ci executable
• How to properly use the input YAML files for rusty-ci

Help Subcommand

Rusty-CI has several subcommands, the first of which is the help subcommand. To run the help subcommand, run rusty-ci help or alternatively rusty-ci. If you dont provide any arguments to rusty-ci, it will print the help message by default.

rusty_ci x.x.x
Adam McDaniel <adam.mcdaniel17@gmail.com>
A continuous integration tool written in Rust

USAGE:
    rusty-ci [SUBCOMMAND]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

SUBCOMMANDS:
    build      Build rusty-ci from YAML file(s)
    help       Prints this message or the help of the given subcommand(s)
    install    Install buildbot
    rebuild    Build and restart rusty-ci from input YAML file(s)
    setup      Output a template YAML files for you to change to customize
    start      Launch rusty-ci from an input YAML file
    stop       Stop rusty-ci

To start a project, run the `setup` subcommand.
Be sure to follow the instructions after each subcommand very carefully!

This tells you more about the program, but it doesn't go into any depth. Let me explain each subcommand individually.

Setup Subcommand

The setup command is very simple. All it does is write template YAML files for building your CI and controlling email notifications to any paths you'd like.

$ ./rusty-ci setup template.yaml mail.yaml
==[INFO]===> Writing template master yaml file to template.yaml...
==[INFO]===> Writing template mail yaml file to mail.yaml...
==[INFO]===> All done!
==[INFO]===> Next, run the `install` subcommand command using either the `bash` or `make` flag

$ more template.yaml

# The required of Rusty-CI to build this CI
requires: x.x.x

# This section holds data specific to the master of the workers
master:
  # The title subsection of the master holds the title of your web gui
  title: "Rusty-CI"
  title-url: "https://github.com/adam-mcdaniel/rusty-ci"

  # This is the ip of the web-gui
  webserver-ip: localhost

  # This is the port of the web-gui
  webserver-port: 8010

  # The address of your repository
  repo: "https://github.com/adam-mcdaniel/rusty-ci"

  # The number of seconds to wait before checking for updates on your repository
  # Two minutes is a good poll interval
  poll-interval: 120

# This section holds data specific to the handler that will look for
# pull requests / merge requests on your repository
merge-request-handler:
  # This is basically the website you're using for version control
...

$ more mail.yaml

# Rusty-CI will automatically email "interested users" about
# all tests that run. The list of "interested users" is the
# list of people who have a commit in the branch or pull request.

# The extra recipients to email
extra-recipients:
  # Emails under the failure section will be emailed
  # info about every failed build
  failure:
    - failure@gmail.com
  # Emails under the success section will be emailed
  # info about every successful build
  success:
    - success@gmail.com
  # Emails under the all section will be emailed
  # info about every build
  all:
    - all_tests@gmail.com


# The "from" email address used to send email updates to recipients
from-address: your-email-here@gmail.com

# The suffix to add to the interested users' usernames
# to get an email we can send updates to.
lookup: gmail.com

# The smtp relay hostname (self explanatory)
# gmail's smtp relay hostname is `smtp.gmail.com`
smtp-relay-host: smtp.gmail.com

# The smtp relay port (self explanatory)
# 587 is the smtp port that `smtp.gmail.com` uses
smtp-port: 587

# The password used to login to the "from" email address account
smtp-password: "p@$$w0rd"

When the setup command is finished, run the install subcommand.

Install Subcommand

The install subcommand is responsible for installing Rusty-CI's dependencies. When you run the install subcommand, it will output a bash script that uses your existing python3 installation to install buildbot and its dependencies.

You can also choose to output a makefile instead of a bash script, but it's not really necessary.

Usage

To output a Makefile for installation, run one of the following commands.

# Output an install makefile
rusty-ci install --make

# Identical
rusty-ci install -m

# Run the makefile to install
make

To output a bash script for installation, run one of the following commands.

# Output an install shell script
rusty-ci install

# Identical
rusty-ci install --bash

# Make the shell script executable
chmod +x ./install.sh

# Run the install script
./install.sh

If you do decide to either the bash or make buildsystems, be sure to read rusty-ci's output and follow any instructions given.

$ rusty-ci install

==[INFO]===> Installing dependencies for rusty-ci...
Do you already have python3-dev, python3-pip, and python3-venv installed? (y/n) y
==[INFO]===> Writing install file to `./install.sh`
==[INFO]===> Successfully wrote install file
==[WARN]===> To install dependencies run `install.sh`
==[WARN]===> Before building from a YAML file, be sure to run `. venv/bin/activate`
==[INFO]===> Next, write your VCS's api token to 'auth.token', and then run the `build` subcommand
Successfully finished install

Now, get an access token from your version control system, and write it to a file named auth.token. Next, run the move on to the build subcommand.

Build Subcommand

The build subcommand is responsible for constructing the buildbot master, the buildbot workers, and their respective configuration files from a YAML file.

rusty-ci-build x.x.x
Adam McDaniel <adam.mcdaniel17@gmail.com>
Build rusty-ci from YAML file(s)

USAGE:
    rusty-ci build [FLAGS] [OPTIONS] <MASTER_YAML>

FLAGS:
    -h, --help       Prints help information
    -q, --quiet      Don't ask user anything
    -V, --version    Prints version information

OPTIONS:
    -m, --mail <MAIL_YAML>    The path to the YAML file dedicated to SMTP authentication info for sending email
                              notifications

ARGS:
    <MASTER_YAML>    The path to the master YAML file

Usage

First, confirm that you're inside your python virtual environment.

. venv/bin/activate

To build from a YAML file, simply run this command.

rusty-ci build template.yaml

If you want to build your CI with support for email notifications, run it like so.

rusty-ci build template.yaml --mail mail.yaml
# is identical to the following
rusty-ci build template.yaml -m mail.yaml

Now, run the start subcommand.

Start Subcommand

Usage

After running all the other rusty-ci subcommands, run

rusty-ci start template.yaml

This will kill the master and workers that were previously running, and start new instances of them.

Now, to view your web gui, go to http://localhost:8010.

If you want to RESTART your CI without killing it, use the following subcommand instead.

rusty-ci rebuild template.yaml mail.yaml

This will not start your CI if it hasn't already been started!

Not working?

Confirm you're in your python virtual environment!

If your web gui isn't loading, buildbot probably failed to start the master, or you put the wrong IP in the master section of your YAML file.

You probably just forgot to change the path to one of your worker's working directories, though.

To see the log for the master, run tail -f master/twistd.log.

If you see an exception, then the master ran into an error. This error will most likely be self explanatory and easy to debug, but in the case that it isn't, go to the buildbot website.

Stop Subcommand

Usage

First, confirm you're in your python virtual environment.

If you want to kill your CI and all of its processes, run the following.

rusty-ci stop

This will kill Rusty-CI and all other python(3) processes!

Input YAML

The input YAML file is pretty simple, but here's an explanation of each section individually.

The requires section

The requires section declares the minimum required Rusty-CI version to build this CI.

requires: x.x.x

The master section

The master section contains the data that controls the master, the bot that controls the workers.

# This section holds data specific to the master of the workers
master:
  # The title subsection of the master holds the title of your web gui
  title: "Rusty-CI"
  title-url: "https://github.com/adam-mcdaniel/rusty-ci"

  # This is the ip of the web-gui
  webserver-ip: localhost

  # This is the port of the web-gui
  webserver-port: 8010

  # The address of your repository
  repo: "https://github.com/adam-mcdaniel/rusty-ci"

  # The number of seconds to wait before checking for updates on your repository
  # Two minutes is a good poll interval
  poll-interval: 120

The merge-request-handler section

The merge-request-handler section holds information that determines how buildbot will handle merge and pull requests on your repository.

Right now, this is only supported for github.com.

# This section holds data specific to the handler that will look for
# pull requests / merge requests on your repository
merge-request-handler:
  # This is basically the website you're using for version control
  # Right now, github and gitlab are the only supported sites
  # If you're using an unsupported version control system, no worries,
  # rusty-ci just wont run on pull requests.
  version-control-system: github
  # The username of the owner of the repository
  owner: adam-mcdaniel

  # The name of the repository
  repo-name: rusty-ci

  # You dont want to run arbitrary code on your machine when anyone
  # makes a pull request. Rusty-CI will not test anyone's pull request
  # if their username is not in this list.
  # Note that this has no effect on GitLab merge request building!
  # Rusty-CI will only build merge requests from a branch
  # thats already inside the repository.
  whitelist:
    - adam-mcdaniel

The workers section

The workers section lists each worker and the information required to construct them and connect them to the master bot.

# This section holds each worker
# You can have as many workers as youd like, just be sure to fill out
# each of their fields out properly.
workers:
  # The name of this worker is `test-worker`
  test-worker:
    # The ip of the master
    master-ip: localhost
    # The worker's files will be installed in this directory.
    # This can also be an absolute path
    working-dir: 'test-worker'

The schedulers section

This section lists each scheduler. Each scheduler has a regular expression that matches a branch to track, and a list of regular expressions that match file changes. A scheduler can also depend on another scheduler using the depends section INSTEAD of the triggers, branch, and password sections.

Read the comments in the template YAML for more information.

# This section holds each scheduler.
# Like the workers section, you may have as many schedulers as youd like.
schedulers:
  # Create a scheduler named `ci-change`
  # This scheduler will trigger the `rusty-ci-test` builder whenever it
  # detects a change in a yaml file for any branch.
  ci-change:
    # This scheduler triggers the `rusty-ci-test` builder.
    # You can put as many builders as youd like here, and the scheduler will start them all.
    builders:
      - rusty-ci-test

    # This will make the current scheduler run if the "your-scheduler-name-here"
    # has run successfully. You can only put one scheduler name in this section.
    # depends: "your-scheduler-name-here"
    # IF YOU USE THE `depends` SECTION, YOU SHOULD REMOVE OR COMMENT THE FOLLOWING SECTIONS
    # Using the depends section will ignore the `branch`, `triggers`, and `password` sections

    # This is a regular expression that matches a branch.
    # If there is a change in a branch whos name matches this regex,
    # it will be checked by the following triggers section.
    # THIS WILL ONLY USE THE FIRST REGULAR EXPRESSION IN THIS SECTION TO MATCH THE BRANCH
    branch: ".*"
    # If a change has occurred in a branch that matches the regex in the branch section,
    # Then the files that were changed are matched against the regular expressions in the
    # triggers section. You can have any number of regular expressions in the triggers section.
    # If any one of them matches the name of a file that was changed in a matched branch,
    # then the builders in this scheduler's `builders` section are executed.
    triggers:
      - '.*\.yaml'
      - '.*\.sh'
      - ".*Makefile"
    # The password a whitelisted user can comment on a merge / pull request
    # to mark it for testing; that is if the pull request was made by a non-whitelisted
    # user. If the pull request was made by a whitelisted user, it is automatically run.
    password: "ok to test"

The builders section

The builders describe the tasks given to workers.

# These are the builders that are executed by the schedulers
# Each has its own specific task that is delegated to one or more workers
# When a builder is run, its script is run on the command line.
# You can have as many builders as youd like as well.
builders:
  # The name of the builder is `rusty-ci-test`
  rusty-ci-test:
    # This is the shell script that the workers will run when this builder is executed
    # You can have as many instructions as youd like
    # Mind you, you cannot use the |, >, <, >>, <<, etc. operators. Sadly, buildbot
    # passes each item separated by whitespace as another parameter to function.
    script:
      - echo Hello world!
      - echo Im an instruction in a script!
    # These are the workers to delegate this build job to
    workers:
      - test-worker
    # The repo to refresh from before running
    repo: "https://github.com/adam-mcdaniel/rusty-ci"

TL;DR

If you were too bored to read the whole thing, just paste this stuff into your terminal (I'm assuming you're using a Debian based OS).

# Update && Upgrade
apt update -y && apt upgrade -y
apt install -y build-essential python3-dev python3-pip python3-venv

# Install rust
curl https://sh.rustup.rs -sSf | sh # Run the rust installer
source $HOME/.cargo/env             # Add `cargo` to your path
cargo install -f rusty-ci           # Install the latest rusty-ci release


# Write template yaml files
rusty-ci setup template.yaml mail.yaml

# Uncomment to modify your CI's settings to fit your project
# nano template.yaml # Controls how your CI tests your code
# nano mail.yaml     # Defines email update / notification settings

# Install rusty-ci dependencies
rusty-ci install -q   # Build install.sh
chmod +x ./install.sh # Make install.sh executable
./install.sh          # Install!

# Enter venv
. venv/bin/activate   # Enter the venv created by rusty-ci
                      # to avoid poisoning your environment

# Add an authentication token from your VCS (github)
echo "YOUR AUTH TOKEN HERE" > auth.token

# Construct your ci bot
rusty-ci build -q template.yaml --mail mail.yaml

# Spin up the workers!
rusty-ci start template.yaml -q

# All done!