Civis Platform workflows are extremely powerful ways to chain and execute jobs on a preset schedule, connecting related jobs and automating them as a single unit. This suite of jobs is treated as any Platform object, allowing the sharing and permissions Platform users have come to expect.
Let’s look at different ways Civis Analytics uses parameters to extend workflows further and test code changes in a pseudo-development environment.
Parameters are form fields that enable Platform users to customize scripts or workflows before they are run — for example, passing a series of zip codes of interest for a report before the workflow runs the query job on the data (and the resulting job that builds the report).
There are many ways to use parameters to increase the power, efficacy, and collaborative possibilities of your organization’s workflows. Sometimes you want to add new steps to an automated workflow, make it more efficient, or experiment and test changes without causing an interruption. Whenever Civis implements and tests Platform changes — when we want to work with a smaller dataset, for example, or wish to pipe results to a different location to compare them with what we’ve done before — we use parameters to run our workflows in “dev” mode.
When we set up a workflow, we almost always define a task in the flow that expects a parameter to be given, such as a database, schema, or table: This allows the workflow to be more flexible, and gets us in the mindset of abstracting our work as much as we can.
We define parameters in our workflow and reference them in our task using YAQL syntax. As a best practice, and for ease of use, we provide default values for all parameters, so that when the workflow runs on its standard schedule, it references real data from the defaults. We also can override the defaults when testing changes, so the workflow runs against a test dataset. See below for an example:
Trying out a new version of a script is another common Civis testing scenario. We typically develop our more complicated code in containers, then run them on Civis Platform.
A container is a type of job that runs on Platform and bundles the required code and resources into a single resource. This container job runs in a virtualized environment using Docker, and allows you to write code outside of Platform but still run it alongside any other Platform jobs, workflows, or scripts.
When we create a task in a workflow that runs a container job, we simply specify the git repo where the code lives, along with the container and a tag to identify which branch of code to execute. Just as we can change the database/schema/table we use in our workflow, we also can change the version of the code we’re using by creating a parameter for the container tag we want to use, reference it in our workflow, and then override it when we want to test it. See below:
We often include steps in our workflows that we wouldn’t want to run during development testing. For example, because no one on my team checks email, I’ve set up workflows to notify them via Slack when my workflow fails — precisely the kind of step I want to skip when testing a change.
In this case, I used YAQL syntax to define a parameter called “workflow_mode,” which I reference in a conditional trigger on all my tasks, ensuring that my team is notified only when I’m not running my workflow in dev mode:
Setting up parameters, leveraging tags, and skipping specified steps all play key roles in Civis’s robust development and iteration process. Hopefully these examples of how we use parameters to customize workflows, scripts, and containers in Platform have given you ideas for bringing abstracted logic, debugging, or customization to your own code.
