This is the kind of project that doesn't make it onto most engineers' resumes, because when it works, nobody notices it. When it breaks, everyone notices immediately. I spent about eight months building it, and it's been invisible ever since. Which means it's working.
The problem with ad-hoc schema management
At SecurityScorecard, multiple teams consume Kafka events: the Flink pipelines are Java, the observations backend is TypeScript/Node, and there are downstream Scala analytics consumers. All of them need to deserialize the same Protobuf messages.
Before I built this infrastructure, the workflow was manual. A schema change would go into the .proto file, and then each team would run their own code generation locally, copy the output into their repo, fix whatever import paths broke, and commit it. This worked until it didn't. Schema changes would land in Kafka before consumers had updated their generated code. Fields would be silently ignored. Services would fail in staging with cryptic deserialization errors because the proto version in the service was two changes behind.
The real problem isn't discipline. Individual engineers doing the right thing doesn't scale across teams. The fix is automation: make the right thing happen automatically on merge.
What I built
The build systems
I set up three separate build systems: Gradle for Java (publishing to Maven), SBT for Scala (publishing to the SBT package index), and an NPM publishing workflow for TypeScript. Each one generates code from the canonical .proto files and publishes a versioned package to the appropriate registry.
Versioning is semantic: a backwards-compatible schema addition bumps the minor version, a breaking change bumps the major version and triggers a Buf compatibility check that fails CI.
Buf for schema governance
Buf is a tool that enforces Protobuf schema compatibility rules. I integrated it into the CI pipeline so that a PR that makes a breaking change (removing a field, changing a field number, changing a field type) fails before it can be merged. The team gets a clear error explaining what the breaking change is and what the options are (add a new field instead, bump the major version, etc.).
This is the step that moves breaking change detection from 'runtime failure in staging' to 'CI failure at code review time.' That shift in where you catch the problem is enormous.
CI/CD on merge
The publishing workflows trigger on merge to main. A schema change goes in, CI generates updated code for all three languages, and within minutes the new packages are available in Maven, SBT, and NPM. Teams add a version bump to their dependency and get the new generated code. No manual steps.
The scope
By the time I wrapped this up, we had 50+ Protobuf definitions covering: CVE vulnerability details, breach evidence, breach observation types, multi-region replication events, enriched observation messages, score messages, observation name filters, and legal entity scorecard IDs. Every Kafka message at SSC is defined once in the schema repo and automatically distributed.
Proto integration time went from hours of manual work to automatic on merge. The less visible a piece of infrastructure is, the better it's working.
I'd add schema registry integration earlier. Confluent Schema Registry stores the compiled schema alongside the message in Kafka, so consumers can always retrieve the exact schema version that was used to serialize a message, even if the .proto file has since changed. We added this later; it should have been part of the initial design.