The ROS build farm

build.ros.org

Central Service vs. Distributed

• You could
  • create Debian packages yourself
  • run the test using e.g. Travis CI
  • generate API documentation yourself
• 
  But
  • setting up all these processes is quite some effort
    • having someone else do it is great for the users
  • the artifacts should be hosted in a central place
  • if you download "official" binaries you know what you are getting

Central Service But Distributed Repositories

• A ROS distribution consists of numerous packages
  • various vcs types
  • various hosting services
• 
  github.com/ros/rosdistro
  • for each ROS distribution
    • target platforms
      e.g. Kinetic: Ubuntu Xenial and Wily,
      Debian Jessie, (Fedora 23 and 24)
    • packages and versions
    • referencing all repositories
  • format of yaml files specified in REP 143
• 
  To use any of these services
  you need to register your repositories

rosdistro/index.yaml

distributions:
  kinetic:
    distribution:
    - kinetic/distribution.yaml
    distribution_cache: <url .yaml.gz>
            

./kinetic/distribution.yaml

repositories:
  catkin:
    doc: <...>
    release: <...>
    source: <...>

release_platforms:
  ubuntu:
  - wily
  - xenial
            

The High-Level View

• Get the source code:
  
  • e.g. git clone
• Install the required dependencies:
  
  • using rosdep
• Build a package:
  
  • cmake, make, make install
• Run the tests:
  
  • make test
• Create a Debian package:
  
  • apt-src build
• Generate documentation
  
  • rosdoc_lite

Some More Details

• Need to setup the environment for building
  • Install all dependencies (using binary packages)
  • Since each job has different dependencies
    each job needs to start with a "clean" state
  • Inside a Docker container
• Build the code from one repository
  • Might contain more than one ROS package
  • The repository is cloned into a catkin workspace
  • Invoke catkin_make_isolated
• Install the code
  • To check that the install step "works"
• Build and run the tests
  • Each test produces a xUnit-like result file

One container for multiple packages

Build vs. test dependencies

catkin_make vs. catkin_make_isolated

• While the Docker container contains the dependencies for all packages
• Each package is still built "in isolation":

The build farm builds with -j 1 for easily readable output

Devel Jobs

rosdistro/kinetic/distribution.yaml

genmsg:
  source:
    type: git
    url: https://github.com/ros/genmsg.git
    version: <valid branch name>
            

pkg/package.xml

<maintainer
  email="dthomas@osrfoundation.org"
>Dirk Thomas</maintainer>
            

+ committers since the job was stable

Pull Request Jobs

rosdistro/kinetic/distribution.yaml

genmsg:
  source:
    test_pull_request: true
    type: git
    url: https://github.com/ros/genmsg.git
    version: <valid branch name>
              

Pull Request Jobs

• Limitations
  • build.ros.org only supports GitHub at the moment
  • Jenkins GitHub user ros-pull-request-builder needs
    access to the org unit
    • wiki.ros.org/buildfarm/Pull request testing
•  
• Alternatives
  • Run the same process using any other CI provider
  • E.g. Travis CI, see example .travis.yml file:
    • docs in github.com/ros-infrastructure/ros_buildfarm

Doc Jobs

rosdistro/kinetic/distribution.yaml

genmsg:
  doc:
    type: git
    url: https://github.com/ros/genmsg.git
    version: <valid branch or tag name>
            

kinetic/api/genmsg/manifest.yaml

api_documentation: \
  http://docs.ros.org/kinetic/api/genmsg/html

description: Generate ROS message and service data.
license: BSD
maintainer_status: maintained
maintainers: Dirk Thomas <email>
authors: <names>

has_changelog_rst: true
depends: <list of package dependencies>
depends_on: <list of packages depending on this>
timestamp: <last generation>

url: http://wiki.ros.org/genmsg
repo_url: https://github.com/ros/genmsg.git
bugtracker: https://github.com/ros/genmsg/issues

devel_jobs: <list of urls>
doc_job: <url>
release_jobs: <list of urls>

actions: <list of action names>
msgs: <list of msg names>
srvs: <list of service names>
            

Wiki Integration

wiki markup

<<PackageHeader(roscpp)>>
            

Release Process

gbp-repo/tracks.yaml

kinetic:
  devel_branch: master
            

rosdistro/kinetic/distribution.yaml

foo:
  release:
    packages:
    - bar
    - baz
    tags:
      release: release/kinetic/{package}/{version}
    url: https://github.com/ros-gbp/foo-release.git
    version: 1.2.3-4
            

Bloom

AUTOMATION

- Makes releasing as easy as possible
- But allows customization anywhere in the process
- Creates the rosdistro pull request for you
            

Sourcedeb Jobs

Binarydeb Jobs

Apt Repositories

Status Pages

"Decipher" Jenkins Job Names

• Some example job names
  • Isrc_uS__roscpp__ubuntu_saucy__source
  • Jdev__ros_comm__ubuntu_trusty_amd64
  • Kbin_uX64__rospack__ubuntu_xenial_amd64__binary
    
    
  • Until first __: prefix for group of jobs
  • 
  • Until next __: the package or repository name
  • Then os name, os code name, arch (except for source jobs),
    source / binary (for release jobs)

Jenkins Statistics - Indigo

Jenkins Statistics - Kinetic

Resources for Jenkins

• Hosted at Amazon
  • 1 x Jenkins master (beefy machine)
  • 1 x Webserver
  • 2 x Nodes (always available)
  • N x Nodes (spot instances when necessary)
  •   → $1,000 - $2,000 per month
• 
  E.g. in September 2016
  • Jenkins ran ~80k builds
  • Which equates to ~10k hours of work

Run "Jobs" Locally

• The build farm is powered by the Python package ros_buildfarm
• After installing it:
  • Debian package python-ros-buildfarm
  • PyPI ros_buildfarm
• You can run:
  • generate_devel_script.py <many args>
  • generate_doc_script.py <many args>
  • generate_release_script.py <many args>
• 
  
  • to easily reproduce results of the build farm
  • run "jobs" for any target platform (within Docker) locally

Test Across Multiple Repos

• ①  You make related changes in different repos, e.g.
  • repo A: add a new API
  • repo B: use the new API
  •   → The devel / PR jobs won't work for this case ☹
    • repo A: passes but doesn't really check the new API
    • repo B: fails because it's using the last released version of A
• 
  ②  You want to check if your changes break any downstream packages
  • repo A: make some changes
  • repo B: check that the tests for the last released version still pass
  •   → The devel / PR jobs don't cover for this case either ☹

Prerelease "Job"

Devel job with multiple repos
            

Supported Targets

• The build farm can produce binary packages for
  • Ubuntu, Debian
  • amd64, i386, arm64, armhf
• 
  We can't cover all combinations ( and ), for Kinetic we cover
  • Ubuntu: Xenial & Wily amd64 & i386, Xenial arm64 & armhf
  • Debian: Jessie amd64 & arm64
• 
  Bloom also generates meta information for rpm (Fedora)
  • But we don't build binary packages (yet?)
• 
  If you want to create binaries for a not supported target
  • Talk to us (<hint> donations as well as time to support efforts helps 😉)
  • Contribute code to automate building binaries for new targets
  • or [see next slide]

Your Own Build Farm

• Use cases
  • Build a different OS name / OS code name / architecture combination
  • Process private repositories
    • You will need your own rosdistro database
  • Try modified version of the build farm
• 
  Two step process [wiki.ros.org/buildfarm]
  1. Provision machines:
     • Jenkins master, Apache webserver, Build nodes
  2. Generate the Jenkins jobs
     • Many configuration options
• 
  Last years presentation from Bosch: [pdf]

   Expect continuous effort

What Can You Do

• Register your repositories
  • add doc entry to generate documentation
    • create a wiki page for your packages
  • add source entry for devel jobs and easy cloning for users
    • if the devel job fails, don't remove the entry, only disable the job:
      

      test_commits: false

    • e.g. your repo depends on other packages which are not released
  • consider enabling pull request testing
  • release your packages, also into newer ROS distros
• 
  Watch out for notification emails from Jenkins
  • read it, search for the problem, try to fix it, ask for help
  • as a last resort: disable the job, blacklist target, remove the release

Troubleshooting

• "It works locally but the job fails" → something must be different
  • Have you tried using `catkin_make_isolated`?
  • Most commonly you have a dependency installed but the build farm doesn't. Check the declared dependencies.

    CMake Error at /opt/ros/kinetic/share/catkin/cmake/catkinConfig.cmake:83 (find_package):
      Could not find a package configuration file provided by "<pkg_name>"

• "The devel job passed but the release failed"
  • In a devel job all packages share the same container. One package might declare a dependency another package uses without declaring it itself.
• Expect (very rarely) all kinds of "hiccups"
  • apt update failing since the repository is being modified concurrently
  • network problems, GitHub resetting the connection, an http request not being handled, ...
  • If your problem persists it's probably not such a hiccup