When you deploy new versions of your services, you always hope for the best but expect the worse. Likely, part of your workflow is to check performance after each deploy. But that means everyone has to remember to do that and then use additional tools to do the monitoring and if there’s an issue, a regression investigation. Switching context like that can be hard. And even then, you may not have enough information to understand what code actually caused that change.
Lightstep can help you by automatically determining if there are changes after a deploy and reporting those changes directly to GitHub so you don’t need to leave your CI/CD workflow. Our Services Change Report GitHub action automatically checks the system health of the current environment once a deploy in GitHub is successful by taking a Snapshot of performance at the time of the deploy and comparing it to the most recent snapshot (or any snapshot you choose) in Lightstep. It let’s you know if there are any latency, error rate, or operation rate issues for each service you’ve configured it to check. It also includes links to downstream dependencies for each service, operations on the service, and traces from the Snapshots, so you can immediately start remediation, right from GitHub.
But this action doesn’t only work in a post-deployment scenario. You can use it anytime you need span data to provide observability information into your committed code. For example, you can configure the action to run whenever a particular label, like
bug is added to a GitHub issue. Or, you can configure it to check for violations of your coding best practices!
GitHub Actions allow you to create a workflow for your repo. Triggers (like a push to a repo or the approval of a pull request) start the workflow and its steps (actions) are then run for you on a virtual machine. If any actions fail, the workflow stops and you’re notified. You can read all about GitHub Actions here.
If you choose to integrate Rollbar and PagerDuty, you also get links to Rollbar error notifications and to the person on-call for each service in the report.
More about Snapshots
You can query the span data on any combination of a service, an operation, and any number of span attributes. Every time you run a query, the results are saved as a Snapshot so you can go back to data at that point in time and analyze it in the Explorer view.
The Lightstep Services Change Report action does the following:
- Uses the Lightstep API to take a Snapshot based on an event in GitHub, like a successful deployment or a particular label is used on an issue, or a condition is violated.
- Stores the Snapshot locally.
- Optionally pulls the most recent Snapshot from Lightstep and compare it to the local version.
- Analyzes the Snapshot(s) and adds the report to your pull request or issue. The report provides links into Lightstep where you can start your investigation into any changes.
If your system also uses Rollbar, you can add an integration to the action and see error information from Rollbar along with a link into Rollbar for more information.
If your system uses PagerDuty, you can integrate it into the action to include “who’s on-call” information for the service associated with the GitHub repository.
You’ll learn how to:
- Create a deployment marker (you can skip to Step 2 if you already have a deployment marker configured for your project).
- Add the Services Change Report action to an existing workflow in GitHub to compare performance before and after a successful deploy.
- Add the action to report service changes whenever the
buglabel is applied to an issue.
- Add configuration to have the action report on any violated best practices.
- Configure the action to integrate Rollbar and PagerDuty.
Before You Begin
You’ll need following to complete this Learning Path:
- A Lightstep account and project. If you don’t have one, sign up for a free account.
- A GitHub account and a GitHub token set as a secret.
An existing GitHub workflow that you’ll add this action to.
If you don’t have one, you can use our post deploy workflow to start.
- A Lightstep API key for your account set up as a GitHub secret.
If you want to integrate Rollbar and PagerDuty, you’ll need: