TL;DR
If it ain't broke, don't fix it — that's usually where we stand with infra and process changes. To change your entire CI is a brave thing to do — it's a huge project, involving the entire R&D team. So what triggered Mitiga to do so? The shift from many repositories to a single monorepo and one source of truth to all our code.
This Mitiga Blog covers our organizational monorepo shift and the reason it triggered a CI adjustment, as well. It also addresses why Mitiga chose CircleCI, how we support the build of a single project from the monorepo each time and save cycles on redundant builds, why auto-generated CI is necessary in this architecture, and examples on how to get started with it on your own.
Why Monorepo?
With the size of the Mitiga DevOps Engineering effort, the logical thing to do would be to separate our code into several repositories for less complexity and order.
But in our case, things got out of hand with almost 300 repositories, most of them containing duplicate code.
That is when the concept of our using a monorepo solution surfaced. Using monorepo, we could minimize our ~300 projects to ~ 6 central monorepos containing code based on certain similarities, including product goal and programming language.
Using monorepos creates a faster development cycle on common packages without the need to update and build the projects dependencies tree whenever a certain base package is changed. It allows you to commit a cross-package feature in one PR.
While monorepos are great, the CI remains a great challenge. You would expect that adjusting the CI to work with the monorepo would be easy. It wasn’t.
At first, we tried changing a few jobs to build the code, Dockers, and deploy from the monorepo. However, as we got more and more projects in the monorepo, things got more complex, and the build process took forever. At the time, we were fully invested in the GitLab CI — the main issue sidetracking developers occurred when the CI was triggered by a push to the repo. Whenever a certain project was pushed, the entire monorepo was built (sometimes 20-to-30 projects). You can imagine how long that took and the versioning mess it caused.
This is when we realized our CI served us well thus far, but it was time for a change.
After inspecting GitLabCI features for monorepos, we decided we were going to have to migrate to a different CI tool. GitLab did not have native options for running certain parts of the CI according to changed code paths on the repository. That meant we couldn’t run the CI exclusively for changed projects — only the entire repository. We needed a tool to allow us to do path filtering.
Why path filtering? if a monorepo looks like this...
...and you’re pushing code to secret project, you want only the secret project’s code to have a CI flow triggered. And not all the projects are built.
In addition to path filtering, we needed something else in our new CI flow — dynamically generated CI code.
What do we mean by dynamically generated?
Let’s say a developer wants to add a new package to the monorepo. The new package needs a CI flow according to the artifacts it needs to generate.
We wanted this CI generating flow to happen automatically without DevOps interference.
So, in conclusion, there were two primary requirements in our new CI — path filtering and dynamically generated CI flow.
How did we do it?
After examining several tools, CircleCI seemed like the best fit. We’ll elaborate on our rationale in the next few paragraphs.
Our new CI is divided into two parts:
Circle CI keeps its CI configuration in a folder under the main repo — .circleci
Part 1 — config.yaml:
The first file is the config.yaml — This file is a pre-step of the CI. It runs and modifies the parameters that will be used in the main CI configuration file.
orbs are pieces of code, templates, usually in charge of some actions. Most vendors create CircleCI orbs of their own to implement main actions. As users, it was easier for us to use these orbs, rather than writing the code ourselves.
One of the orbs CircleCI offers is the path-filtering orb that runs a flow based on changed paths. On this file, we filtered the changed paths on the current commit. If a certain path was changed, it also changed a parameter that would later be used to decide whether a certain workflow would run.
config.yaml
- If packages/common/logging/.* changed → put the value true in mitigacloud-logging-modified parameter.
- Another important thing in this file is the setup: true in the top level of our parent configuration file, this label will point that this file is a setup phase, and there is another CI configuration file to follow.
- Based on the path filtering, the CI config file that would now run will be the .circleci/workflow.yml as mentioned at the end of the config file, with the label config-path.
Note: in order to use dynamic configuration and the setup, true flag must be enabled in the project’s circleCI settings on the advanced tab.
Part 2 — workflow.yaml:
Our config.yaml calls the workflow.yaml file, which is the main part of our CI.
This file will assume a simple case, where we have several packages, and we require that changing a package won’t trigger a build for its dependents.
The yaml is segmented into four main parts:
- orbs — As mentioned previously, give us tools templates. A full list of circleCI orbs can be found here. You can also create your own orb, containing your own functions.
- parameters — CI parameters, some passed from previous config.yaml phase. The default value of modified parameters is false — after the config file runs, it passes the new values for these parameters (some have changed to true after the path-filtering took action).
- jobs — new, defined jobs for the CI. These jobs can be used when creating a workflow for a project. Notice we use orbs' steps in the jobs — for example, the docker orb’s check, build, and push steps are used in this job.
- workflows — flows running jobs based on conditions (such as modified parameters being true). The flows call jobs defined in this yaml, with parameters, in a certain order.
Combining these four parts creates a simple workflow.yaml that builds a typescript package and a docker image for triggered projects.
workflow.yaml
Using the config.yaml and the workflow.yaml as the CI configuration will allow you to run your CI worry-free — whenever a certain project has pushed changes, only that project will be built.
This leaves us with a bigger question — how do we modify these CI files to include every new project added to the monorepo without having to edit them manually, add parameters, workflows, and path filtering mappings for each new component?
We chose to generate our CI using code!
The next part of this article will describe our self-generating CI tool.
