01 / Distributed systems & data correctness

Usage is only useful
if you can bill it correctly.

Delivering a multi-region Microsoft Purview consumption billing pipeline that turns usage from 6+ cloud services into invoice-ready data for Microsoft commerce.

Context: Microsoft · Microsoft Purview
My role: Pipeline delivery and maintenance
Core technologies: C# / .NET, Scala / Spark, Service Fabric, Azure

Cloud service integrations. Invoice accuracy at the center.

A multi-region pipeline with three layers of deduplication and an extension to US government clouds for public-sector billing.

A billing pipeline is a trust boundary.

Cloud consumption begins as usage, but eventually becomes a line on an invoice. Between those two points, data from multiple services and regions has to be aggregated and delivered consistently.

Repeated records are not merely a data quality issue here: they can distort what a customer is charged. The challenge was to deliver invoice-ready consumption data while treating correctness as a property of the whole pipeline, not just its aggregation step.

My contribution

I owned delivery and maintenance of the multi-region consumption billing pipeline. This work combined Scala and Apache Spark aggregation with C# / .NET services on Service Fabric to deliver invoice-ready usage to Microsoft commerce.

I designed a three-layer deduplication approach, created reusable regional deployment templates, and worked on network, firewall, and access security. I also extended the pipeline to US government clouds with isolated identity, access, and network controls.

Scope of ownership: this case study covers my billing pipeline work and its integrations. It does not imply ownership of every upstream service or component in the wider platform.

Correctness needs more than one checkpoint.

Duplicates can appear at different stages. I designed defenses at three distinct points so that the final delivery did not rely on a single check earlier in the pipeline.

Within-batch deduplication
Remove repeated records within the current aggregation batch, before they can inflate the resulting usage.
Historical lookback
Check against prior processing history to identify duplicates that arrive in a later batch.
Atomic cache check at delivery
Add a final deduplication boundary when delivering usage, protecting invoice accuracy at the point where it matters.

Make regional delivery repeatable.

Reusable deployment templates made the regional setup repeatable. Network, firewall, and access controls were part of delivery rather than a separate concern left until the end.

Extend capability without assuming the same environment.

US government cloud expansion required isolated identity, access, and networking controls. Extending the pipeline under those constraints enabled public-sector consumption billing.

Invoice-ready, across services and regions.

The delivered pipeline integrated 6+ cloud services, aggregated usage into invoice-ready data, and supported multi-region delivery. Its three-layer deduplication design protected invoice accuracy, while the government cloud extension enabled billing for public-sector environments.

The impact is in the foundation: dependable consumption data for commerce, reusable regional deployment, and explicit attention to data correctness and secure delivery.

In billing, correctness is not a final validation step. It is a design decision at every boundary.