Open SourceApache 2.0Azure FunctionsManaged Identity

Vigil

Name: Vigil
Author: AtLongLast Analytics

Synapse Pipeline Monitoring and Reporting

An open-source Azure Functions agent that monitors Azure Synapse pipeline runs, delivers scheduled health summaries by email, and archives CSV snapshots for audit-ready operations.

View on GitHub Talk to us about customisation

The Problem

A pipeline that fails quietly is the expensive kind.

Data platforms tend to fail in silence. A Synapse pipeline errors overnight and nothing downstream notices until a dashboard is wrong or a report lands late — by which point you're explaining the gap instead of catching it. In regulated environments you also have to show you were watching, not just say so.

Vigil makes that watching automatic. Deploy it once and it sends a structured health summary every morning, then archives a timestamped CSV snapshot as an evidence trail — fast failure visibility for the team, and a durable audit record for everyone who asks.

Daily Health Report

Deploy once. Receive a pipeline summary every morning.

Vigil sends a structured HTML summary via Azure Communication Services and archives timestamped CSV snapshots. Teams get fast failure visibility while preserving a durable operational evidence trail.

pipeline_health_report.html

Vigil Pipeline Health Summary

2026-03-27 08:00 EST · 13 total runs

Last 24 hours

SUCCEEDED · 12FAILED · 1IN PROGRESS · 0

FAILEDnightly_ingest_pipeline

Run ID: 48f2-91ac

SUCCEEDEDmodel_refresh_pipeline

Run ID: f93d-11ce

Illustrative preview · drop a real screenshot at /public/software/vigil/report-preview.png

How It Works

From deploy to a daily evidence trail — in four steps.

Deploy once into your tenant

Vigil runs as an Azure Function inside your own Azure environment. It authenticates with managed identity (DefaultAzureCredential), so there are no secrets to store or rotate.

It checks Synapse on a schedule

On a timer trigger, Vigil pulls the last day's pipeline runs from your Synapse workspace — statuses, run IDs, and durations — with no manual polling.

You get a summary every morning

Vigil emails a structured HTML health report via Azure Communication Services to your chosen recipients: what succeeded, what failed, and the detail on-call needs to act.

Every run is archived for audit

Each run writes a timestamped CSV snapshot, building a durable, immutable evidence trail that supports SOC 2 and internal audit workflows.

Built managed-identity-first on Azure Functions and Azure Communication Services — the same platform services your team already trusts, with nothing new to secure.

Who It's For

Built for teams that must prove pipeline reliability daily.

Data Platform Operations Teams

Get a daily, timestamped summary of pipeline health without building custom monitoring infrastructure from scratch.

Synapse Workspace Owners

Track failed runs quickly with run IDs and durations so on-call engineers can move directly from alert to action.

Compliance-Driven Data Teams

Produce immutable daily CSV snapshots and structured logs that support SOC 2 and internal audit workflows.

Quick Start

First report in under 20 minutes.

Provision required Azure resources, configure local settings, authenticate with Azure CLI, and start the Azure Functions host.

Python 3.12+
uv, Azure CLI, and Azure Functions Core Tools
SYNAPSE_ENDPOINT, ACS_ENDPOINT, SENDER_ADDRESS, and RECIPIENT_ADDRESSES configured

terminal

# 1. Clone and install dependencies
git clone https://github.com/AtLongLastAnalytics/vigil.git
cd vigil
uv sync --group dev
cp local.settings.json.example local.settings.json
# Fill SYNAPSE_ENDPOINT, ACS_ENDPOINT, SENDER_ADDRESS, RECIPIENT_ADDRESSES

# 2. Authenticate and run locally
az login
func start

# 3. Trigger the function (PowerShell)
curl.exe -X POST http://localhost:7071/admin/functions/vigil_monitor ^
  -H "Content-Type: application/json" -d "{}"

# 4. Tests and lint
uv run pytest tests/ -v
uv run ruff check .

Common Questions

Questions teams ask before production rollout.

Can we use Vigil commercially?

Yes. Vigil is released under the Apache 2.0 licence, which explicitly permits commercial use, modification, and redistribution. You can run it inside your Azure tenant and incorporate its outputs into commercial compliance workflows. The licence stays with the OSS project; any custom features we build for you are negotiated separately.

How does authentication work?

Vigil uses DefaultAzureCredential. In Azure it authenticates via managed identity by default — no secrets to rotate. Locally you authenticate with az login. This means the same code runs in dev and production without credential changes.

What does a customisation engagement look like?

Typical engagements start with a discovery call to scope what stock Vigil doesn't cover — common asks are integration with incident tooling (PagerDuty, Opsgenie), bespoke alerting rules, custom report templates, multi-workspace fan-out, and deployment hardening for regulated environments. We then agree a fixed-scope build, deliver, document, and hand over. Most engagements are 2–4 weeks.

Do we own the custom features, or are they upstreamed?

Both options are on the table and we agree upfront. The default is: anything that benefits the wider community gets upstreamed under Apache 2.0; anything that's organisation-specific (your alerting integrations, your audit schema mappings) stays in your private fork. You always get the source, runbook, and the ability to maintain it yourselves — no lock-in.

Do you offer support contracts for the stock OSS version?

Yes — light-touch support engagements for teams running stock Vigil in production. Covers monthly check-ins, alert triage help, upgrade guidance, and operational reviews. If you're depending on Vigil for an audit-critical workflow and want a named engineer behind it, get in touch.

Consulting

Need a private deployment or custom monitoring features?

Off-the-shelf monitoring rarely fits enterprise estates. We deploy and extend Vigil in your environment with custom alerting, reporting, and compliance workflows.

Private / in-tenant deploymentIncident tooling (PagerDuty, Opsgenie)Custom alerting rulesBespoke report templatesMulti-workspace fan-out

Discuss a private deployment