Tutorials
How to Automatically Update Plugins Using Renovate
In this guide we will discuss how to leverage Renovate to keep all your source and destination plugins on the latest version.
Introduction to Renovate #
Renovate is a tool for automated dependency management for your code repositories to help automate away the tasks of manual dependency versioning. For more information on the features and why you should use Renovate, please read this.
This guide covers how to leverage Renovate to keep your CloudQuery source and destination plugins up-to-date but is not meant to be a comprehensive tutorial of Renovate. For instructions on how to setup Renovate, please reference their getting started documentation. This how-to guide assumes you have Renovate configured and running already on your git based repository.
Walkthrough #
Step 1: Identify Regex Capture Groups #
While Renovate provides tons of built-in functionality to make most dependency management simple, it also provides a powerful functionality called 'RegexManager' which allows you to leverage custom regular expression statements to manually extract dependencies, their version, and updating mechanism. Click here for more details on the specific configurations of RegexManager.
For our purposes, we're concerned about the following items:
datasourceTemplate
: data source is where the package's versions are managed.packageNameTemplate
: this is the lookup name that is passed to the data source.depNameTemplate
: this is the 'friendly' name that is shown in renovate PRs.extractVersionTemplate
: regular expression for discovering plugin version in the data source.versioningTemplate
: what versioning 'format' the data source followsfileMatch
: path to thecloudquery.yaml
file in your repository.matchStrings
: regular expression to find the plugins in yourcloudquery.yaml
file.
Using a simplistic configuration file that looks like:
---
kind: source
spec:
name: gcp
path: cloudquery/gcp
version: "v17.4.1"
tables: ["*"]
destinations: ["postgresql"]
---
kind: source
spec:
name: aws
path: cloudquery/aws
version: "v30.0.0"
destinations: ["postgresql"]
tables: ["*"]
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
version: "v8.7.4"
write_mode: overwrite-delete-stale
You can see that we have three 'keys', meaning fields that are static across all plugins, we can leverage these to construct our custom regex statement for each plugin. These 'keys' will be used as regular expression named capture group. Those fields being:
- kind: what type of plugin this is (source or destination).
- path: the 'name' of the plugin (such as GCP or AWS).
- version: the current version of the plugin.
Step 2: Create Custom RegexManager #
Now that we've identified the fields we need to extract from the configuration file to create the custom regex, next we need to construct the regex statement to extract those 'keys' into named capture groups.
For this it's recommended to use a regex testing tool to ensure your regex is actually working. Regex101 is a popular one but feel free to use any others as long as they support the PCRE2 regex engine. So taking our sample configuration file from above, we begin to play around with various regex until we meet the following criteria:
- a named capture group for the plugin type (source or destination) which we will reference as 'kind' in this tutorial.
- a named capture group for the plugin name (AWS, GCP, etc..) which we will reference as 'plugin' in this tutorial.
- a named capture group for the plugin current version which must be named
currentValue
.
For our sample configuration file this regex will look like:
kind:\s(?<kind>.*)\nspec:\n\s{2}name:.*?\w\n\s{2}path:\scloudquery\/(?<plugin>.*)\n\s{2}version:\s\"?v(?<currentValue>.*\d)\"?\n
This regular expression does the following:
- captures the
kind: (source|destination)
into a named capture group called 'kind` - skips the
spec:
andname:
lines of the file - captures the
path: cloudqery/(gcp, aws, etc...)
into a named capture group calledplugin
while escaping thecloudquery/
prefix. - skips to the next line and captures the
version: "vX.X.X"
into the required named capture groupcurrentValue
while optionally checking if the string is quoted or not.
On the Regex101 site we can verify the regex is working by viewing the explanation section on the left side of the screen:
Important Note: This regex is specific to the example shown in this how-to guide, if your configuration file is formatted different (such as lines in different orders or indented differently like in a helm
values.yaml
file) then you'll need to adjust the regex accordingly.Step 3: Add Custom RegexManager to Renovate #
The last thing we need to get this working is to simply add a block to the
regexManagers: []
in your renovate.json
file in your code repository such as:{
"regexManagers": [
{
"description": "Update Cloudquery Plugins",
"fileMatch": ["^cloudquery.yaml$"],
"matchStrings": [
"kind:\\s(?<kind>.*)\\nspec:\\n\\s{2}name:.*?\\w\\n\\s{2}path:\\scloudquery\/(?<plugin>.*)\\n\\s{2}version:\\s\"?v(?<currentValue>.*\\d)\"?\\n"
],
"datasourceTemplate": "github-releases",
"packageNameTemplate": "cloudquery/cloudquery",
"depNameTemplate": "{{kind}}-{{plugin}}",
"extractVersionTemplate": "^plugins-{{kind}}-{{plugin}}-v(?<version>.*)$",
"versioningTemplate": "semver"
}
]
}
Things to be aware of when adding this regex to your
renovate.json
file:- Any regular expression token (such as
\s
or\n
) needs to be escaped in the configuration file as shown above. This does not apply to regular escapes such as escaping the"
with a singular\
. - the
depNameTemplate
is the friendly name we'll see in PRs for the plugins. Our example simply uses the kind and plugin names from the capture group but feel free to use your own. - the
extractVersionTemplate
must match the releases in the CloudQuery GitHub repository which at the time of this guide being written follows this format:plugins-source-aws-v9.1.0
. We simply use the named capture groups to substitute into the string, simply think of them as 'variables' in this context. - the
packageNameTemplate
is simply the name of the GitHub repository for CloudQuery, this is just what thegithub-releases
data source expects as input.
Once you've done this and renovate successfully runs, you should see the following the next time a plugin is released on GitHub (with different labels based on your git workflow and CI):
Summary #
In this post we've discussed how you can leverage Renovate for automatic plugin updates to ensure you're always running the latest versions of your source and destination plugins without having to manually update.
Ready to get started with CloudQuery? You can try out CloudQuery locally with our quick start guide or explore the CloudQuery Platform (currently in beta) for a more scalable solution.
Want help getting started? Join the CloudQuery community to connect with other users and experts, or message our team directly here if you have any questions.